Give charts a single path in the layout prompt: tabulate - #94
Conversation
The benchmark scores charts by looking for their data points in HTML
tables - ChartDataPointRule only reads tables. But the prompt never
clearly tells the model that a chart must become a table: the chart
instruction is worded as if tabulating were optional ('For charts/graphs
being converted to tables: ...'), and a separate instruction says to
describe figures in brackets ('Describe images/figures ... like
[Figure: description]') - and a chart is a figure.
So a model can read a chart correctly and still just write
'[Figure: bar chart of revenue by year]'. When it does, every chart
rule fails, because the scorer can't read data out of a bracket
description:
chart_data_point on '[Figure: bar chart ..., 2015 is 100]' -> fail
chart_data_point on the same data as an HTML table -> pass
The chart score then reflects which instruction the model followed,
not how well it read the chart.
Two wording changes so the prompt asks for exactly one thing:
- the chart instruction becomes a direct command: 'Convert
charts/graphs/figures to tables, using flat combined column
headers ...'
- the describe instruction no longer mentions figures: 'Describe
images/pictures ... like [Picture: description]'
The wording carries into the GEMINI/ABS variants automatically (they
are built by replacing only the coordinate bullet) and is pinned by a
new test. Chart scores go up for models that were describing charts
instead of tabulating them. The same wording exists verbatim in several
per-provider plain-parse prompts and is left unchanged here to keep the
diff small.
|
Nice catch — the chart rules really do only read tables, so those two bullets were a coin flip on the score. Pulled the branch: suite passes, ruff clean. One thing before merge: chart regions are ground-truthed as
Optional: Don't worry about the per-provider plain-parse prompts — I'll handle those separately. Keeping this to the shared layout prompt is the right scope. |
Problem
The benchmark scores charts by looking for their data points in HTML
tables —
ChartDataPointRuleonly reads tables. But the prompt neverclearly tells the model that a chart must become a table:
"For charts/graphs being converted to tables: use flat combined column
headers ..."
images/figures briefly in square brackets like [Figure: description]" —
and a chart is a figure.
So a model can read a chart correctly and still just write
[Figure: bar chart of revenue by year]. When it does, every chart rulefails, because the scorer can't read data out of a bracket description:
The chart score then reflects which instruction the model followed, not how
well it read the chart.
Fix
Two wording changes so the prompt asks for exactly one thing:
charts/graphs/figures to tables, using flat combined column headers ..."
images/pictures briefly in square brackets like [Picture: description]"
The wording carries into the GEMINI/ABS prompt variants automatically, and
a new test pins it.
Impact
Chart scores go up for models that were describing charts instead of
tabulating them, so before/after numbers aren't directly comparable for the
chart category. The same wording also appears in several per-provider
plain-parse prompts (anthropic, openai, google, gemma4, nemotron_omni,
dots_ocr); this PR changes only the shared layout prompt to keep the diff
small — happy to update the others too if you'd like consistency.
Testing
New
test_layout_prompt_wording.pypins the new chart instruction and the[Picture:]wording across all four prompt variants. Full suite passes;ruff clean.