Fix layout element rules dropped when a doc also has parse rules - #133
Open
AmirF194 wants to merge 1 commit into
Open
Fix layout element rules dropped when a doc also has parse rules#133AmirF194 wants to merge 1 commit into
AmirF194 wants to merge 1 commit into
Conversation
_load_jsonl_dataset routed a (category, pdf) group to LayoutDetectionTestCase only when it had layout rules and no parse rules. A group with both (e.g. a layout doc whose order rules land in parse_rules since their type isn't "layout") fell to the parse branch instead, and ParseTestCase.test_rules can't hold LayoutTestRule entries, so the layout rules were silently dropped and never reached Visual Grounding scoring. Route on layout_rules alone and keep parse_rules in the same test_rules list, matching load_test_case's sidecar-format handling a few lines below, which already does this correctly. Fixes run-llama#131
Author
|
This is still waiting on a maintainer to approve running CI for a first-time contributor, so lint and both test jobs haven't executed yet. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_load_jsonl_datasetonly built aLayoutDetectionTestCasewhen a group had layout rules and no parse rules (if layout_rules and not parse_rules:). A doc whoselayout.jsonlrows mixtype: "layout"element rules with atype: "order"rule (which lands inparse_rulessince its type isn't"layout") falls to theelsebranch instead, andParseTestCase.test_rulesis typedlist[ParseRule] | None, so it can't holdLayoutTestRuleentries. The layout element rules are dropped on the floor, and the run still reportssuccess: true.load_test_casea few lines below (the sidecar.test.jsonpath) already does this right:if has_layout_rules: return LayoutDetectionTestCase(..., test_rules=test_rules, ...), passing the full mixed rule list through.LayoutDetectionTestCase.test_rulestype-routes each entry (type == "layout"vs everything else) andEvaluationRunner._has_mixed_rules+ the multi-task path already exist to score both halves. This PR makes_load_jsonl_datasetmatch: route onlayout_rulesalone and passlayout_rules + parse_rulesthrough together.Added
tests/parse_bench/test_cases/test_loader.py(there wasn't one for this function): a doc with both rule types now keeps both intest_rulesand still resolves asLayoutDetectionTestCase, plus two cases confirming layout-only and parse-only docs are unaffected. The new test fails without the change (doc becomes aParseTestCase, layout rule gone).ruff check/ruff format --checkand the full suite (405 passed, 1 skipped) are clean on 3.12 and 3.13.I didn't touch the CHANGELOG since no other bugfix PR in the history seems to add an entry there directly.