Conversation
gitronald
commented
Jun 6, 2026
Owner
- add dev-story commit history report
- expand dev-story with commit-window excerpts and changelog mining
- add component-growth analysis and expand dev-story architecture section
- update: annotate component-growth plot with add/remove labels
- update: shade commit-activity bars by count
- version [prerelease]: 0.8.7a0
- Add plan 026: explore replacing bs4/lxml with selectolax
- plan 026: record maintainer decision to break the BeautifulSoup contract
- plan 026: add parser-parity harness; tree-shape risk measured as small
- Migrate parse pipeline from BeautifulSoup+lxml to selectolax
- Route find/find_all through lexbor CSS engine (fast path)
- Scan text nodes directly for find(string=...) (fixes quadratic has_captcha)
- plan 026: record selectolax migration results (~2x faster, suite green)
- plan 026: note latent parser fragilities exposed by the native-text experiment
- Preserve component heading and harden slug derivation across 4 parsers
- plan 026: record slug-and-heading cleanup outcome
- Native rewrite v2: add public function helpers + migrate promo parser
- Native rewrite: top_stories delegators + most_read_articles + short_videos + flights
- Native rewrite: banner + available_on + buying_guide + map_results + general_questions
- Native rewrite: scholarly_articles + view_more_news + twitter_cards/result + searches_related
- Native rewrite: discussions/top_image_carousel/recipes/images/videos/people_also_ask
- Native rewrite: top_stories + perspectives + jobs + locations + footer
- Native rewrite: shopping_ads + products
- Native rewrite: general (largest parser; self-exclusion, regex class, next_sibling)
- Native rewrite: knowledge (panel with string=True + walk_descendants + sep fixes)
- Native rewrite: knowledge_rhs + local_results
- Native rewrite: news_quotes + notices
- Native rewrite: ai_overview + ads (last parsers)
- Native rewrite v2 complete: classifier + extractors + utils + components
- Drop SoupNode adapter + bs4/lxml runtime deps (plan 026 final)
- DRY: collapse helpers + drop vestigial aliases
- Perf: drop two Python-recursive walkers, cheap feature path
- Perf: switch to selectolax lexbor backend + classifier hot-path tightens
- DRY: drop dead find_text, collapse classifier marker tables, fold captcha pre-skip
- Drop the bs4-facade wrappers from utils.py
- Perf: scope _get_dom_positions + skip ai_overview re-serialization
- Perf: cheaper attribute access + cache AI overview payloads
- docs: add selectolax-parsers guide
- update: select local-results directions anchor by Q7PwXb class
- add reparse_demo script to diff parser output over demo corpus
- docs: log head-to-head parse bench vs pre-branch baseline
- update: guard nullable selectolax tag and root access for pyrefly
- update: bump orjson, protobuf, pre-commit, pyrefly, ruff via dependabot
- Add plan: standardize component parsers on functions over classes
- Refine plan: keep footer.py as section module, drop only the class
- Restructure plan: fold cmpt->elem rename into Phase 0 (establish + apply contract)
- Correct Phase 3: parse_alink defs are not hard duplicates
- Defer parse_alink + knowledge rethink to plan 028
- Phase 0a: document the component parser contract
- Phase 0b: rename misnamed entry-parser param cmpt -> elem
- Mark Phase 0 done in plan 027
- Make parse_unknown match the parser contract; drop parse_not_implemented
- Plan 027: record Phase 0c (exception-free contract, drop parse_not_implemented)
- Phase 0b: rename sub-node helper params cmpt -> sub
- Update stale comment: 'AI overview cmpt' -> 'component' after param rename
- Phase 1: Footer class -> module functions; realign img_card naming
- Phase 2: NoticeParser class -> module functions
- Plan 027: link PR Standardize component parsers on module-level functions (plan 027) #139
- Apply ruff format to pre-existing drift on feature/v0.9.0
- ci: run Tests workflow on feature/ branches**
- Docs: correct parser-contract docstring per PR review
- Plan 027: mark completed (PR Standardize component parsers on module-level functions (plan 027) #139 merged)
- Plan 028 Phase 1: unify parse_alink into _common (lenient href)
- Plan 028 Phase 2: table-driven knowledge sub_type dispatch
- Plan 028 Phase 3a: close knowledge sub_type registry gaps
- Plan 028: mark completed; split details-schema work into plan 029
- Plan 028 Phase 2: fix pyrefly h2_text type (str|None -> str)
- Plan 028: address PR review (doc status + test panel node)
- Plan 028: note accepted carousel url=None decision from PR Plan 028: knowledge dispatch + parse_alink reconciliation #141 review
- Plan 029: flesh out knowledge details-schema alignment
- Simplification pass: dead code, reuse, and efficiency cleanups
- Remove dead Component.get_metadata method
- Fix add_component to honor an explicit cmpt_rank of 0
- Add main_layout field to parse_serp features output
- Pin ExtractorMain layout branches; fix no-rso sec2 duplication
- Refactor standard- layout ladder into a data-driven table*
- Rename standard- layout labels to observable names; add plan 030*
- Guard _extract_from_standard_sub_type against unknown sub_type
- docs: mark plan 030 done with PR reference
- docs: update README/CHANGELOG for 0.9.0 line; fix example table alignment
- docs: add plan 031 for automating the locations CSV download
- docs: backfill plan 026 and normalize frontmatter timestamps for plans 026-031
- version [prerelease]: 0.9.0a0
- docs: add fixture-corpus guide, plan 032, and corpus profiling scripts
- docs: record pass B drop verification; add verify_drops script
- update: consolidate serp fixtures into serps.json.bz2 with notes; drop 8 redundant records
- update: repoint corpus scripts at serps.json.bz2; make them report-only
- update: clarify specialized-components label in signature cluster report
- close plan 032: annotate and prune the bulk serp fixture corpus
- test: add kp-wholepage embedded-organics fixtures and baseline snapshots
- update: recover organics from kp-wholepage panels via standard-kp-wholepage layout
- docs: add guide for finding silently-dropped extraction gaps
- docs: add plan 033 (kp-wholepage sub-column extraction) and update extraction-gap guide
- docs: flag interim kp-wholepage snapshots as suspect in plan 033
- parse kp-wholepage tabs as sub-columns; add election component types
- document kp-wholepage sub-column model in changelog, guide, and plan 033
- guide: add the sub-column pattern (gate, anchors, flatten, keep-working-paths)
- close plan 033: mark done and record merge into feature/v0.9.0
- version [prerelease]: 0.9.0a1
- update complementary kp-wholepage panels: recover video organics, classify as knowledge
- document complementary kp-wholepage fix in changelog and gate assumption
- version [prerelease]: 0.9.0a2
- add plan 034: local_results sub_type category cleanup
- bench_parse: record Python version and platform in output
- optimize get_text with a native selectolax text() fast path
- add plan 036: _ComponentSignals lever + extractor hot-path review
- clarify get_text fast-path comment: walker skips script/style/template only as descendants
- extract kp-wholepage visual digest sub-results and clean panel header
- extract kp-wholepage songs section items into details
- extract kp-wholepage events section items into details
- extract kp-wholepage albums section items into details
- split related_searches accordion dropdown from grid suggestions
- extract complementary rhs panel boxes as sub_rank rows
- document kp-wholepage panel body extraction in changelog
- version [prerelease]: 0.9.0a3
- rename rhs rows to side_bar; drop things-to-know qa box noise
- version [prerelease]: 0.9.0a4
- recover dropped organic column on kp-wholepage non-recipe active tabs
- version [prerelease]: 0.9.0a5
- update: add snakeviz to dev group for viewing bench profiles
- update: log bench results to tests/benchmarks and fix stale fixture glob
- update: add scripts/_common.py salvaged from ads-no-subtype probe
- update: add plan 033 scripts audit and reorg
- update: drop ads-no-subtype ignore rule; folder retired
- update: ship demos in WebSearcher.demo so they run after pip install
- update: activate plan 033 and log demo absorption
- update: expose demos as unified ws-demo console command
- update: log ws-demo entry point decision in plan 033
- docs: note demo absorption and ws-demo in changelog and readme
- update: consolidate corpus loading into _common; generalize verify_drops to verify_corpus
- update: retire finished one-off scripts; refresh README fixtures section
- update: log buckets C, D, B in plan 033
- update: fold demo_searches into the package as ws-demo searches
- update: split demo.py into a WebSearcher/demos package
- fix: port survey_ai_overviews and demo_screenshot scripts to selectolax
- update: move reparse_demo into the parser-regression skill
- update: add ws-demo show subcommand and repoint readme off scripts/show_parsed
- update: absorb corpus-curate scripts into the skill, add tracked corpus integrity test
- update: absorb bench_parse into the parse-bench skill, repoint docs
- update: absorb show_serp + demo_screenshot into serp-inspect skill, retire show_parsed for ws-demo show
- update: add plan 035 (skills 8 to 4, scripts absorbed) and log in 033
- update: keep bench_parse tracked in scripts/, parse-bench skill wraps it
- update: fold bench_parse into WebSearcher.bench, move survey to explore-ai-overview skill, empty scripts/
- update: changelog reflects scripts/ removal, ws-demo show, WebSearcher.bench
- update: record pr 152 url in plan 035
- update: renumber plans 033/035 to 037/038 (collision with feature/v0.9.0 plans)
- update: drop double blank line after imports in knowledge_rhs (ruff)
- update: finalize plans 037 and 038 as done (merged via pr 152)
- update: add plan 039 browser-automation alternatives (preserved from unpushed draft 034)
- plan 036: add Lever 3 (available_on classifier full-component get_text)
- plan 036: record corpus evidence for available_on fallback (fires 0x)
- update: standardize plan frontmatter from git+gh, normalize plan 002 to lf
- update: use explicit null for empty plan frontmatter fields
- update: limit explicit null to closed plans, drafts keep empty pending fields
- update: add plan index table to docs readme and refresh guides list
- update: sort plan index drafts-first then by completed and pr
- version [prerelease]: 0.9.0a6
- update: sha-pin actions, group dependabot, gate publish on PUBLISH_ENABLED
- update: bump polars 1.41.2, syrupy 5.3.1, lxml 6.1.1 (hold typer, ruff for cooldown)
- update: fix selectolax migration regressions in component parsers
- update: widen election precondition to include the eer-rc-i branch
- update: save captcha pages and persist the recovered parse in ws-demo searches
- update: repoint knowledge dispatch tests and give corpus-integrity unique ids
- update: mark side_bar and make_soup changes breaking in changelog and readme
- update: trim readme recent changes to 0.9.0 only
- update: add plans 040-041, mark 042-043 inactive, refresh plan index
- update: finalize changelog and readme for 0.9.0 release
- version [patch]: 0.9.0
A scoping plan that maps where selectolax (lexbor) could replace BeautifulSoup+lxml in the parse path, gated on a parser-parity harness and a measured pilot rather than a speculative rewrite. Cross-references plan 023's profile (bs4 find traversal ~60%, lxml parse ~16-18%) and inventories the bs4 API surface and byte-identical-output risks a migration must clear. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
Resolves open question #1: the bs4 input/return-type API is fair game to change, collapsing the migration to a clean one-way rewrite (option B) with byte-identical output as the only remaining gate. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
Add scripts/diff_parsers.py, the prerequisite deliverable for the selectolax exploration: it parses every fixture SERP with both lxml+bs4 and selectolax and reports structural divergence plus signal parity for the exact class/id/tag/attr targets the classifier and extractor query. Result over 88 SERPs: zero divergence on every queried signal; the only structural diffs are inert (SVG element-name casing and <tbody> auto-insertion, neither of which the pipeline queries). The byte-identical-output risk from a parser-backend swap is much smaller than feared on real Google markup. selectolax added to the dev group only (diagnostic), not runtime dependencies. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
Replace the bs4/lxml parse and query layer with a selectolax (lexbor) backend via a bs4-compatible adapter (WebSearcher/_slx.py). make_soup now returns a SoupNode wrapping a selectolax tree; the ~50 parsers, classifier, and extractor run against the adapter's bs4-subset API (find/find_all/select/get_text/attrs/ navigation/extract), so the migration lands without rewriting every call site. Key faithfulness work to keep parse_serp output byte-identical (snapshot suite): - bs4 class-match semantics (single token / multi-token exact / list-OR), name=True any-tag, string=True via bs4 .string, find_parent, recursive=False. - get_text skips script/style/template text to match bs4+lxml. - extract() detaches via remove(recursive=False), preserving subtree and stable node identity (mem_id) so DOM-position reordering still resolves; id()-keyed ordering switched to mem_id. - _ComponentSignals walks the selectolax tree natively (hot classify path). Snapshots: 10 SERPs updated for benign whitespace-only differences in concatenated text fields (lexbor emits more whitespace-only text nodes than lxml; content is byte-identical otherwise). All other output unchanged. selectolax moved to runtime dependencies. Full suite green (299 passed, 66 snapshots) on Python 3.13. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
The initial adapter matched find/find_all with a pure-Python descendant walk per call, which was ~4x slower than bs4. Translate the common (name, attrs) queries to a CSS selector and run them on selectolax's C engine via node.css()/css_first(), falling back to the Python matcher only for queries CSS can't express exactly (regex/string= filters, attr/class lists, multi-token class exact-match, callable filters, recursive=False, name lists). Correctness: node.css() matches the node itself, but bs4 find/find_all search descendants only, so self is excluded (except the document root, which legitimately matches its own <html>). Byte-identical across the snapshot suite. Per-SERP median ~451ms -> ~239ms on the local subset; full suite 76s -> 45s. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
…ptcha) find(string=re.compile(...)) with no tag filter (utils.has_captcha) was matching by computing each element's full subtree text via the Python text walker, i.e. O(elements x subtree) per call -- the dominant cost in the profile (~40s of 54s). Scan text nodes directly instead (O(text)), matching bs4's NavigableString search. get_text keeps the Python walker (selectolax's native .text() changes whitespace in ways that alter downstream parsing -- local_results address/directions, a knowledge sub_type). Per-SERP median ~237ms (bs4) -> ~106ms; suite 76s -> 26s. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
The four parsers that derive a sub_type slug from a component heading were
fragile in two ways: (a) the slug came from a bare .replace(" ", sep), which
only normalizes ASCII space (a non-breaking or doubled-space heading would
slug to "co__op" or "behind\xa0the-scenes"); (b) three of the four discarded
the raw heading text entirely after slugging.
Changes:
- utils.slugify(text, sep): " ".join-style whitespace-robust slugify; replaces
the .replace(" ", sep) sites in knowledge, local_results, searches_related,
and perspectives.
- Preserve the raw component heading as details["heading"] in searches_related,
local_results, and perspectives (matching knowledge.py's existing convention,
3 prior uses). Top-level "title" is left alone: it already means "the result's
own title" (the business name, the story title), so a component-level header
belongs on details, not there.
- local_results._link_text_to_url: key off the stable structural classes
(a.L48Cpd = website, a.VDgVie = directions) instead of the localized visible
anchor text. The old text-keyed lookup silently dropped both fields whenever
get_text(strip=True) left stray whitespace on the key, and missed them on any
non-English SERP.
Snapshots regenerated for the deliberate enrichment (63 SERPs); diff verified
to contain only details["heading"]/sub_type/website/directions changes. Full
suite green (299 passed, 66 snapshots).
https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
Adds get_text/class_tokens/node_string/find_text/reparse_fragment/make_soup_native as public functions in _slx.py alongside the existing SoupNode class. These are what migrated parsers call directly. Each parser migration: drops bs4 import, adds `node: Node = cmpt.raw` at the top to escape the SoupNode wrapper, replaces bs4-style finds with native selectolax css/css_first plus the helpers above. The SoupNode class stays in place until every call site is native; then it's removed and make_soup swaps to return Node directly. First migration: promo.py (12-line template). https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
…general_questions Plus _slx: get_text returns None for None input (matches utils.get_text convention so parsers chain css_first+get_text cleanly), and a new has_text helper for the filter_empty_divs predicate. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
top_stories takes either Node or SoupNode (delegators recent_posts/latest_from/ local_news still pass SoupNode from the dispatcher; the hasattr unwrap absorbs the difference until those delegators stop being delegators). https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
shopping_ads: bs4 multi-token-class strings are EXACT (not AND), so
'mnr-c pla-unit' must reject 'mnr-c c3mZkd pla-unit' -- compound CSS
.mnr-c.pla-unit alone matches the latter via AND-of-tokens, which routed
modern PLA cards through the legacy parser. Narrow + verify exact tokens.
shopping_ads also inlines bs4 find_parent('div', {'class':'pla-unit'}) as
an upward tag+class walk.
https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
… next_sibling)
- find_subcomponents: explicit self-exclusion via mem_id (the gu-gels nested-.g bug).
- regex class match (re.compile('fG8Fp')) -> CSS substring [class*=...] (equivalent).
- top_menu.children + child.decompose() -> iter(include_text=False) + decompose.
- stars.next_sibling (text-inclusive) -> inline _next_sibling_with_text via
parent.iter(include_text=True).
- 'd4rhi' in attrs.get('class', []) -> 'd4rhi' in class_tokens(sub).
- list-of-classes find ({'class': ['P1usbc','IThcWe']}) -> CSS OR 'div.P1usbc, div.IThcWe'.
https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
…+ sep fixes)
Adds walk_descendants() helper to _slx for subtree-scoped pre-order DFS --
selectolax node.traverse() walks the ENTIRE document from this point forward
(not just the subtree), and node.css('span, div, a') with commas returns
results GROUPED BY SELECTOR, not document order. Both shapes break the panel
text walk that needs descendants-only in document order.
Also: where I migrated utils.get_text(...) calls (utils default sep=' '), pass
the separator explicitly; the _slx.get_text default is sep='' (bs4 .text style).
Knowledge titles like 'Donald Trump 45th and 47th U.S. President' need sep=' '.
https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
Adds next_sibling/previous_sibling/next_siblings helpers to _slx (text-inclusive by default, matching bs4 .next_sibling semantics -- selectolax .next may skip text nodes). knowledge_rhs walks next_siblings of the description's parent and peeks the previous_sibling for the submenu; both routes now go through these helpers. local_results: recursive=False find_all becomes iter(include_text=False) + tag filter; class_token list checks use class_tokens(); the rest is straightforward. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
news_quotes: children destructure preserved (text-inclusive iter); .div / .span first-tag access -> css_first; inline _is_tag helper for the bs4 isinstance filter. notices: copy.copy(cmpt) -> reparse_fragment (bs4 cloning semantics for the extract-from-clone pattern); div.extract() -> div.remove(recursive=False); class_= kwargs become CSS class selectors. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
ai_overview: find_parent x2 (inline _find_parent_tag_class walking ancestors
on tag+class), _drop_nested_descendants uses mem_id sets, _doc_position uses
mem_id-based sibling index. find_all([list],{class}) -> CSS comma-join.
ads: AD_SUBTYPE_SELECTORS converted to dict of CSS selector strings (was
Selector NamedTuples); all utils.* calls inlined as native + _slx helpers.
shopping_ads gains the hasattr(raw) unwrap so callers can pass either a
SoupNode (component dispatch) or a native Node (ads delegating internally).
All 39 component_parsers are now native. utils.* helpers, classifiers, and
extractors still operate via the SoupNode adapter; component dispatch still
hands parsers a SoupNode, which they unwrap via cmpt.raw.
https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
Migrates the integration backbone to native selectolax Node: - _slx: subtree_first/subtree_css/walk_descendants/next_sibling/next_siblings helpers. walk_descendants replaces selectolax Node.traverse() (which walks the entire document from this point forward, not just the subtree). - classifiers/main.py + footer.py: every classifier method takes a Node; _ComponentSignals walks via walk_descendants (subtree-only). - extractors/__init__.py + extractor_main/header/footer/rhs/serp_features.py: all bs4 patterns replaced with native CSS + helpers. Critical fix: bs4 check_dict_value(attrs, 'class', ['M8OgIe']) is EXACT list equality, not token membership -- restored exact match. - components.py: Component.elem typed as Node; dispatch passes Node to parsers; reorder_by_dom_position uses mem_id; child-iteration uses iter(include_text=False). - utils.py: SoupNode wrappers stripped; helpers take Node natively (with a transitional _unwrap for any SoupNode that may still reach them). - parsers.py: make_soup returns Node; parse_serp type-annotated as str | Node. - All 39 component_parsers' 'node = cmpt.raw' boilerplate dropped via scripted refactor (cmpt is now Node directly). local_results uses subtree_first/css for cmpt-level queries that could match self (the gu-gels/places bug shape). - tests/test_ads.py, test_extractor_main.py, test_utils.py: switch from bs4-style .find() to utils.get_div() / native API. Suite: 299 passed, 66 snapshots, ruff clean. SoupNode class still present in _slx.py for any unmigrated callers/tests; final cleanup (drop SoupNode, drop bs4/lxml from runtime deps, README) is the next commit. https://claude.ai/code/session_01RdPLdE9zJx9mK4rKaKtpJh
# Conflicts: # CHANGELOG.md # scripts/bench_parse.py
v0.9.0: absorb demos and dev scripts into the package and skills
…unpushed draft 034)
Caller attribution found ClassifyMain.available_on is the most expensive get_text caller (full-component text fallback run on nearly every component); scope it as a third optimization lever alongside _ComponentSignals and the extractor review.
Measured: 2 available_on components in the 87-SERP corpus, both caught by the cheap span.mgAbYb heading path; the full-component /Available on text fallback never fires corpus-wide. Points Lever 3 at gating the fallback (not removing it) plus a targeted test for the non-mgAbYb shape.
Bump polars, syrupy, lxml (hold typer + ruff for cooldown)
Integrate v0.9.0 into dev (selectolax parse rewrite, in-package demos, kp-wholepage parsing)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.