Evaluation

bibr's evaluation harness scores saved extraction JSON against printed-paper ground truth. It reports title, DOI, abstract, author, and reference metrics, plus paper-level pass rates and cohort coverage. Primary metrics participate in the optional --threshold mean gate. Diagnostic metrics support analysis and do not participate in that gate.

Source checkout required

Run uv run python -m evaluation.evaluate from a source checkout with the development dependencies installed. The harness is not part of the published bibr package.

Bring your own ground truth

Reference corpora and extraction captures are not distributed with bibr. Supply independently prepared gold exports with --gold-dirs. The harness fails when no gold directory exists or no predictions match ground truth.

Quick start

# Use your own predictions and independently prepared gold exports
uv run python -m evaluation.evaluate \
    --results-dir outputs/local \
    --gold-dirs /path/to/gold \
    --output outputs/local-eval.json

Gold directories contain one bibr-shaped JSON export per paper. The harness matches predictions by normalized DOI, falling back to filename, and prints aggregate metrics. Keep gold independent of the predictions being evaluated; copying an extraction into gold would preserve the same errors on both sides.

Fixed cohorts and regression gates

uv run python -m evaluation.evaluate \
    --results-dir outputs/local \
    --gold-dirs /path/to/gold \
    --expected-ids /path/to/expected-paper-ids.json \
    --output outputs/local-eval.json \
    --threshold 0.80

--expected-ids accepts a JSON file containing a list of paper IDs or an object with members or ids. Expected IDs without a prediction are added with zero scores on the paper-level floor metrics. Otherwise, missing files would disappear from the denominator. --ids-file separately restricts scoring to prediction filenames listed one per line (without .json).

The --threshold flag exits with status 1 if any primary metric's non-null mean is below the supplied value. It does not gate missing/unmatched counts, abstention rate, or paper-level pass rate. Inspect those fields too when comparing runs. Choose a threshold from a recorded baseline scored with the same metric version and cohort; 0.80 above is only an example.

Public CI tests the evaluator using small synthetic examples. It checks scoring behavior and software contracts; passing CI does not establish extraction accuracy for your papers.

Ground truth

Always provide --gold-dirs explicitly. Gold represents what is printed in your papers, including reference fields; enrichment output is not extraction ground truth. Keep the gold revision fixed when comparing saved runs and record how each field was checked against its source.

Running extraction for evaluation

Evaluate against exports produced by your own bibr chew run:

uv run bibr chew papers/ -o outputs/local/
uv run python -m evaluation.evaluate \
    --results-dir outputs/local/ \
    --gold-dirs /path/to/gold

A bibr batch --out directory works as --results-dir too. The evaluator reads every *.json file in it except run_info.json, the batch's own run record, and validation_report.json. They are not scored, not listed as predictions and not part of predictions_tree_sha256. The batch's other records (outcomes.jsonl, runs.jsonl, tables/) are not *.json files.

Metrics reference

Metric scores are in [0, 1], with higher being better. A metric can be null when gold provides nothing to score; those entries are excluded from its mean. The current definitions are metrics_version: 5, recorded in every saved evaluation, including section-text results. Re-score predictions when definitions change rather than comparing means from different versions: scores with metrics_version 4 and 5 do not compare. Version 5 skips a batch's run_info.json, counts the gold references of a paper with no matched pair carrying a field as misses in the micro-averaged reference scores, and tokenizes section text in every script (see below).

Title

Metric	Tier	Description
`title_soft`	primary	Exact match after case, punctuation, and whitespace normalization, or a word-aligned prefix retaining at least 70% of the longer normalized title. The shorter side needs at least two words, or six characters in supported scripts without inter-word spaces.

DOI

Metric	Tier	Description
`doi_match`	primary	1.0 for equal normalized DOIs. An asserted DOI against empty gold scores 0; when both are empty there is no score.

Abstract

Metric	Tier	Description
`abstract_rouge_l`	primary	ROUGE-L F1 score using whitespace tokenization and longest common subsequence
`abstract_ned`	diagnostic	Normalized Edit Distance similarity (1 - NED), following OmniDocBench methodology

Predictions are scored from metadata.abstract only. Exports older than schema 11 name that block info, and the evaluator still reads it. The evaluator does not recover an absent exported abstract from section text. Gold may use its abstract-typed sections because some gold records store their abstract there.

Authors

Metric	Tier	Description
`authors_fullname_f1`	primary	F1 on full printed names ("given family") using fuzzy similarity — boundary-insensitive to the given/family split
`authors_f1`	diagnostic	F1 on normalized family names with fuzzy, globally greedy bipartite matching
`first_author`	diagnostic	Fuzzy similarity (token_sort_ratio) on the first author's family name

Per-author front-matter fields (affiliation_sim, email_f1, orcid_f1, corresponding_acc) and keywords_f1 are also emitted as diagnostics; they score only over papers whose gold carries the field.

References

Metric	Tier	Description
`ref_matching_f1`	primary	F1 for individual reference matching using a four-level cascade: DOI match (1.0) > fuzzy title match (0.9) > first-author + year (0.7) > unstructured citation string (0.6). Uses globally-greedy bipartite matching
`ref_title_acc`	primary	Among matched GT references that have titles, fraction with a correct title (fuzzy token_sort_ratio >= 85)
`ref_year_acc`	primary	Among matched GT references that have years, fraction with a correct year
`ref_doi_recall`	primary	Among all gold references with DOIs, fraction recovered with the correct DOI. Unmatched gold references with DOIs count as misses.
`ref_count_ratio`	diagnostic	min/max ratio of reference counts
`ref_author_acc`	diagnostic	Among matched gold references with authors, fraction where every gold surname is recalled
`ref_journal_acc`	diagnostic	Among matched GT references with a container/journal, fraction correct
`ref_volume_acc`	diagnostic	Among matched GT references with a volume, fraction correct
`ref_pages_acc`	diagnostic	Among matched GT references with page ranges, fraction correct

Sparse GT fields

The ref_*_acc metrics use matched pairs where gold has the field. ref_doi_recall instead uses all gold references with a DOI. Saved results include field coverage, counts, and micro-averaged scores to make these denominators visible.

A paper whose gold references carry a field but whose matched pairs do not, for example because extraction lost the bibliography, scores 0.0 on that ref_*_acc metric. The micro average (micro_mean, micro_correct, micro_denominator) counts that paper's gold references with the field as misses, so it does not leave out the papers the mean scores lowest. A paper whose gold lacks the field is null and counts in neither.

Output format

--output writes a JSON artifact containing:

Fields	Meaning
`metrics_version`, `generated_at`, `bibr_commit`	Metric definitions, scoring timestamp, and commit of the evaluator checkout
`predictions_dir`, `predictions_tree_sha256`	Prediction source and content digest; the evaluator commit alone does not identify the code that produced these predictions
`papers_attempted`, `papers_evaluated`, `papers_missing`, `missing_ids`	Cohort accounting, including missing expected predictions
`papers_unmatched`, `unmatched_ids`	Predictions that matched no gold record and were excluded from aggregates
`metrics`	Per-metric mean, p10, median, min, max, and non-null count `n`, with additional abstention and reference-coverage fields where applicable
`pass_rate`, `pass_rate_n`	Fraction passing the paper-level floors and the exact denominator used
`abstention_rate`, `n_abstained`, `abstained_ids`	Front-matter abstentions and their share of evaluated predictions
`per_paper`	Individual scores, flags, and reference-field counts for investigation

pass_rate uses these floors: title_soft >= 0.9, doi_match >= 1.0, authors_fullname_f1 >= 0.9, and ref_matching_f1 >= 0.8. A null field is excluded from that paper's floor checks. Missing expected predictions and front-matter abstentions count as failures in the headline rate. The legacy survivors-only rate is retained separately as pass_rate_excl_abstained with its own denominator; it is not interchangeable with pass_rate.

Front-matter abstentions suppress affected field scores from ordinary metric means. mean_incl_abstained reports the companion mean including their pre-suppression scores. Always read mean accuracy alongside abstention and cohort coverage.

Section-text evaluation

The same entry point can score saved exports against *.sectiongold.json artifacts:

uv run python -m evaluation.evaluate \
    --results-dir outputs/local \
    --sections \
    --section-gold-dirs /path/to/section-gold \
    --output outputs/local-sections.json

This mode emits section recall, coverage, and a drop report. It has a separate output shape from metadata scoring, and returns before the metadata --threshold gate. Per-type section recall is diagnostic.

Recall and precision count tokens. Section text is casefolded and NFKD normalized, diacritics are removed from Latin letters, and the text is split on runs of non-word characters ([\W_]+), so letters and digits of every script count. "Straße" is one token, and combining marks on letters of other scripts stay inside their word. Scripts written without spaces between words (Han, hiragana and katakana, Thai) give one token per character.

Scoring GROBID output

The harness can score GROBID with the same metrics as bibr. evaluation.grobid_run sends each PDF to a GROBID server's /api/processFulltextDocument with consolidation off (bibr is scored before enrichment) and the printed reference and affiliation strings included. It writes one TEI file per paper, a failure record for each paper that did not convert, and a manifest.json with the GROBID version, the request parameters, and each PDF's SHA-256, wall time and attempts. evaluation.grobid_tei then writes each TEI file as a bibr export, with extraction.producer naming GROBID and extraction.converter the converter:

uv run python -m evaluation.grobid_run --grobid-url http://localhost:8070 \
    --pdf-dir papers/ --ids-file cohort-ids.txt --out grobid-tei/
uv run python -m evaluation.grobid_tei --tei-dir grobid-tei/ --out grobid-json/
uv run python -m evaluation.evaluate \
    --results-dir grobid-json/ \
    --gold-dirs /path/to/gold \
    --expected-ids grobid-tei/manifest.json \
    --output grobid-eval.json

Score both tools against the same --expected-ids list. cohort-ids.txt holds the paper ids (PDF stems) of the cohort bibr is scored on, one per line, with # starting a comment line. The runner then processes exactly those papers, and the manifest's ids are the same ids as bibr's --expected-ids file. A paper GROBID failed on has no prediction file, and the manifest keeps it in the denominator as a failure. Without --ids-file every PDF in --pdf-dir enters the manifest, so a GROBID failure on a PDF outside the cohort would count against GROBID but not against bibr.

A failure is recorded per paper, whatever its cause, and the run goes on. The converter does the same for a TEI file it cannot read or convert. --resume continues an interrupted run in the same --out directory, over the same paper ids, so pass the --pdf-dir and --ids-file the run started with. It refuses other paper ids, a server that reports another GROBID version, and other request parameters. It also refuses a --grobid-image other than the one the manifest records, including one given when the first run named none. A resume without --grobid-image keeps the recorded image and checks none. The version check reads only the version string the server reports, so two images of one GROBID release, such as its CRF-only and full images, are told apart only when the first run and every resume name --grobid-image. The server's model configuration is not checked.

For the headline comparison, read pass_rate in both runs, and compare bibr's mean_incl_abstained with GROBID's mean for the front-matter fields. pass_rate counts missing papers and abstentions as failures for both tools. bibr's per-field mean leaves out the papers where its front matter abstained, but converted GROBID output never abstains, so GROBID's mean covers every paper it returned. The evaluator writes mean_incl_abstained only for the fields an abstention suppresses, and only when bibr abstained on at least one paper. Where it is absent, bibr's mean already covers every paper.

The converter carries over every field the metrics credit when GROBID emits it, including reference DOIs at any level, editors kept apart from authors, and given names. GROBID does not classify body sections, so its body text is typed unknown. In the section-text benchmark only the label-independent layout_recall is comparable between the two tools.

The public JATS benchmark

The bibr JATS benchmark is the whole-paper test on the limitations page, made public: 300 CC BY papers from PMC Open Access, gold built by a script from the publishers' JATS and checked against the PDF text, per-paper scores for bibr and GROBID 0.9.0, and the scripts behind them. Its README shows how to download the exact PDFs and score your own outputs with evaluation.evaluate. Scores taken after the benchmark was published are not held-out results.

Interpreting results

Compare runs on the same papers, gold revision, metric version, and extraction configuration. Record the extraction build and model identities separately from the scoring commit. Review missing and unmatched papers before interpreting accuracy changes, and use per_paper to separate OCR/native-text failures, metadata errors, and reference segmentation or parsing errors. Historical scores without this provenance are not a baseline for current main.