Catalog-metadata extraction — model comparison
Which model best extracts catalog metadata (title, document type, date, subjects, named entities, issuing body) from government documents. 27 models × 13 documents, generated 2026-06-25. Companion to the Interpolation eval (reasoning over tables and charts in PDFs).
Reading this table
Click any column to sort. Accuracy = macro mean of per-field agreement vs a frontier-consensus silver reference (panel gpt-4.1 / claude-opus-4-8 / gemini-2.5-flash, grok-4 arbitrating). Rows marked * are panel members — they define the reference, so their accuracy is circular/inflated; judge them by cost, not accuracy. Completed = documents that returned a usable result / attempted. Valid schema = share of outputs that parsed into the controlled {value, code} schema (instruction-following, not correctness). $/1k docs = approximate cost per 1,000 documents (observed tokens × public list prices); locals run free. Sample = 13 docs (directional).
All models — hosted & local
01 · sortable| Model | Tier | Accuracy▼ | Completed | Valid schema | Latency | Resp tok | $/1k docs |
|---|---|---|---|---|---|---|---|
| anthropic:claude-opus-4-8 | hosted | 0.93* | 10/10 | — | 8.3s | 689 | $89.37 |
| gemini:2.5-flash | hosted | 0.86* | 10/10 | — | 7.8s | 526 | $1.91 |
| xai:grok-4 | hosted | 0.85 | 13/13 | — | 6.9s | 231 | $9.04 |
| openai:gpt-4.1 | hosted | 0.84* | 10/10 | — | 4.1s | 327 | $6.12 |
| anthropic:claude-sonnet-4-6 | hosted | 0.83 | 10/10 | — | 10.6s | 588 | $14.76 |
| mistral:large | hosted | 0.78 | 12/13 | 100% | 28.1s | 483 | $7.02 |
| anthropic:claude-haiku-4-5 | hosted | 0.77 | 13/13 | — | 4.9s | 478 | $4.39 |
| openai:gpt-4o-mini | hosted | 0.57 | 13/13 | — | 3.0s | 182 | $0.37 |
| ollama:granite3.3-8b | local | 0.57 | 10/10 | — | 14.5s | 277 | free |
| ollama:gemma2-9b | local | 0.54 | 10/10 | — | 10.7s | 212 | free |
| ollama:qwen2.5-7b | local | 0.49 | 10/10 | — | 10.0s | 219 | free |
| gemini:2.5-pro | hosted | — | 0/3 | — | 0.0s | 0 | — |
| ollama:apertus-8b | local | — | 10/10 | — | 17.3s | 314 | free |
| ollama:mistral:7b | local | — | 10/10 | — | 13.6s | 278 | free |
| ollama:qwen2.5-coder-7b | local | — | 10/10 | — | 10.4s | 228 | free |
| ollama:gemma-sealion-27b | local | — | 10/10 | — | 69.8s | 348 | free |
| ollama:phi3-mini | local | — | 9/10 | — | 6.6s | 277 | free |
| ollama:qwen2.5-3b | local | — | 9/10 | — | 5.6s | 282 | free |
| ollama:olmo2:7b | local | — | 9/10 | — | 7.6s | 141 | free |
| ollama:eurollm-9b | local | — | 8/10 | — | 12.6s | 263 | free |
| ollama:apertus-70b | local | — | 8/10 | — | 357.4s | 307 | free |
| ollama:climategpt-70b | local | — | 8/10 | — | 163.9s | 356 | free |
| ollama:llama3 | local | — | 6/10 | — | 7.4s | 159 | free |
| ollama:climategpt-13b | local | — | 6/10 | — | 23.9s | 175 | free |
| ollama:apertus-70b-q3 | local | — | 4/10 | — | 902.8s | 178 | free |
| ollama:climategpt-7b | local | — | 3/10 | — | 7.2s | 112 | free |
| ollama:deepseek-r1-8b | local | — | 1/10 | — | 23.8s | 499 | free |
Per-field accuracy — every model
02Accuracy is field-dependent, so the cheapest correct pipeline routes each field to the best (non-panel) model for it rather than picking one model overall. Each cell is per-field agreement accuracy; the highlighted cell is the best non-panel model for that field (where you would route it).
| Field | opus° | gemini° | grok-4 | gpt-4.1° | sonnet | mistral | haiku | 4o-mini | granite | gemma | qwen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| jurisdiction_code | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.90 | 0.80 | 0.90 | 1.00 | 0.70 |
| language_code | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.90 | 0.90 | 1.00 | 0.90 |
| document_type_code | 1.00 | 1.00 | 1.00 | 0.78 | 0.89 | 0.83 | 0.78 | 0.89 | 0.44 | 0.50 | 0.67 |
| coverage_codes | 1.00 | 0.29 | 1.00 | 1.00 | 1.00 | 0.83 | 0.83 | 0.30 | 0.40 | 0.22 | 0.30 |
| coverage_text | 0.93 | 0.80 | 1.00 | 1.00 | 0.93 | 0.90 | 0.64 | 0.64 | 0.43 | 0.58 | 0.79 |
| issuing_body_name | 1.00 | 1.00 | 0.88 | 0.94 | 0.94 | 0.88 | 1.00 | 0.62 | 0.81 | 0.79 | 0.50 |
| title | 0.81 | 1.00 | 0.56 | 0.81 | 0.69 | 0.83 | 0.75 | 0.31 | 0.38 | 0.44 | 0.25 |
| subjects | 0.75 | 0.71 | 0.45 | 0.62 | 0.59 | 0.43 | 0.60 | 0.36 | 0.41 | 0.42 | 0.43 |
| key_points | 0.81 | 0.78 | 0.65 | 0.73 | 0.74 | 0.59 | 0.72 | 0.42 | 0.52 | 0.47 | 0.36 |
| abstract_summary | 1.00 | 1.00 | 1.00 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.50 | 0.00 | 0.00 |
° panel / reference-defining model (gpt-4.1, opus, gemini-flash) — its accuracy is circular and shown muted, not eligible as a routing target. Highlighted = best non-panel model per field. Columns ordered by macro accuracy.
The cost story
The current catalog pipeline runs on grok-4.20 (~grok-4 tier, ~0.74 accuracy here). Field-completeness alone suggested cheap/free models were equivalent — but accuracy tells a sharper story: free local models emit perfect schema yet get more values wrong (≈0.49–0.57), and gpt-4o-mini (~24× cheaper) drops to ≈0.57. The strongest genuinely-cheap, non-panel option is claude-haiku-4-5 (≈0.77).
Because accuracy is field-dependent (jurisdiction codes are easy for everyone; document type and title are not), the cheapest correct pipeline is per-field routing (table 02) rather than one model overall. Reference is frontier-consensus silver on 13 docs — directional, not a human gold set.