Architecture

Pipeline overview

bibr processes scientific papers through six logical phases. The shared stage plan lives in bibr/pipeline/plans.py; local and served pipelines use the same parsing, extraction, validation, and export stages.

File / bytes --> Validate
                   ├── DOCX / JATS / HTML / ePub --> Native parse ─────────┐
                   └── PDF --> Render --> Layout --> Native text / OCR ──┤
                                                                        v
                                                      Structure / sentence segmentation
                                                                        v
                                                      Extract --> Identity validation
                                                                        v
                                                      Core checkpoint --> Enrich --> Export

Two runtime options are available:

bibr chew / Python library (bibr/local/pipeline.py) -- LocalPipeline manages model lifetimes on one machine. OCR uses the selected Paddle-first runtime, an explicit HTTP backend, or a cloud vision provider.
bibr serve (bibr/serve/app.py) -- LitServe HTTP API with disk-backed multipart ingress and one spawned inference worker hosting the pipeline GPU models (layout, segmenter).

Pipeline shutdown releases owned models, servers, and clients. Cancellation during local server startup or sentence segmentation waits for the underlying worker to finish before disposing its result or releasing the model lock. Cancellation can therefore take as long as the blocking operation; server startup still relies on the selected backend's own timeout.

Pipeline stages

1. Validate (`bibr/input/validate.py`)

Checks MIME type, file corruption, and encryption. Refuses unsupported formats (.exe, .zip, .tex, legacy .doc). Supports .pdf/.docx/.xml/.html/.htm/.epub — .xml as JATS.

2. Input (`bibr/ocr/`, `bibr/input/docx_native.py`)

DOCX and JATS XML files are parsed natively (python-docx, and a JATS <article> parser respectively) to extract headings and text structure, skipping OCR entirely
HTML (.html/.htm) and ePub (.epub) files are also parsed natively — HtmlHandlingStage (bibr/pipeline/stages/html.py), backed by bibr/input/html_native.py and bibr/input/epub_native.py — skipping OCR the same way as DOCX/JATS
PP-DocLayoutV3 detects PDF regions (headings, body text, tables, formulas, figures, and more); regions then flow through native-text inspection and recognition
OCR_BACKEND=paddle selects one Paddle-first OCR runtime transactionally at startup; explicit paddle-*, glm-*, and cloud backends remain available through OcrOptions

PDF limits. PIPELINE_MAX_PAGES limits the processed page count (default 200). Pages beyond the cap are dropped without a warning in the console or the export; for a long document, compare the length of the export's extraction.pages list with the PDF's page count. Render-pixel and dimension limits reject oversized pages before rasterization. Size memory for the processed pages and concurrent files.

DOCX limits. A table cell spanning more than 1,000 grid columns is read as 1,000 wide. A table's grid cells are its rows × its widest row, plus 100 for each column; a merged cell counts once per grid cell it covers, text included. A table of more than 1,000,000 grid cells, or one that takes the document past 4,000,000 grid cells or 64 MiB of table HTML, is dropped with a DOCX_TABLE_DROPPED warning; its caption stays in the body text. The HTML is measured as rendered: escaped text and cell markup in every grid cell, at the bytes per character its widest character needs. Pictures past the first 1,000 are dropped (DOCX_FIGURES_DROPPED). Figures carry at most 128 MiB of image data in total, an image counting once per figure that shows it; the figures past that keep their caption without an image (DOCX_FIGURE_IMAGES_OMITTED). A footnote or endnote referenced more than once is read once, with one foot reference from each paragraph that cites it. A note, hyperlink, text box or equation nested inside another of its kind is read as part of the outermost one, and a picture inside nested drawings is one figure.

HTML and ePub limits. An HTML file over 48 MiB is rejected at validation as invalid input, before the pure-Python html5lib parser reads it. HTML is read in the encoding its byte-order mark names (UTF-8, or UTF-16 as Word's "Save as Unicode" writes it), else its <meta charset>, else as UTF-8 when the bytes decode as UTF-8 and as windows-1252 when they do not; an ePub chapter in the encoding its byte-order mark or XML declaration names.

Headings and reference lists in DOCX and HTML. DOCX headings come from Word's heading styles (Title, Heading 1-6) and outline levels; HTML headings from <h1>-<h6>. A manuscript whose author bolded the headings by hand has neither, so in a document with no heading style, outline level or heading element a short paragraph that is bold throughout is a level-1 heading (not one opening "Figure" or "Table", ending in , ; :, or a sentence of more than six words ending in a full stop). In any document, a paragraph whose whole text is a references heading ("References", "6. Literatur", the languages of the reference-heading grammar) heads the reference list. A list a reference manager generated is found even under no heading: in DOCX, paragraphs in the Bibliography, EndNote Bibliography or Citavi Bibliography Entry style or inside an EndNote, Zotero, Mendeley, Citavi, Papers or Word bibliography field; in HTML, Word's MsoBibliography and EndNoteBibliography paragraphs and a container named refs, references, bibliography, csl-bib-body, ref-list or reflist (pandoc and Quarto's citeproc output). Such a list under no references heading gets a "References" section. A paragraph without a year after the list ends it, and goes back to the section open before the references heading. HTML text outside any paragraph element (<div>text</div>, a <dd>) is read as a paragraph of its own, as a browser lays it out.

HTML table limits (HTML and ePub tables, and the table HTML OCR returns). A rowspan ends at the table's last row, as a browser draws it, and more than 100 header rows are read as data rows. A table may be at most twice as wide as its widest row of cells plus 20 columns, and hold at most 20 slots for each cell and row it has plus 1,000. A colspan ends at that width, so a footnote row with colspan="100" under five columns gives 30 columns. A table wider than that all the same (one wide row padding many short rows, or rowspans piling up row after row) gets no contents. A table is also measured as its HTML renders, as DOCX tables are: escaped text and cell markup in every slot, a spanned cell's text once for every slot it fills, at the bytes per character its widest character needs. A table of more than 16 MiB gets no contents either. In HTML and ePub input, neither does a table that takes the document past 64 MiB of tables or 4,000,000 table cells (rows × widest row); an ePub's chapters are one document. Like a table without cells, an HTML table then keeps its caption and markup when the caption labels it ("Table 3. …") and is dropped otherwise, and a TABLE_CONTENTS_OMITTED warning counts these tables. An OCR table that cannot be read is dropped with OCR_TABLE_DROPPED.

JATS table limits. JATS cells are read without their spans, and each row is padded to the widest. The HTML table limits apply: at most 20 slots for each cell and row plus 1,000, 16 MiB per table measured as rendered, and 64 MiB and 4,000,000 cells for the document's tables together. A <table> nested in another is read once: its rows follow the row holding it, and its text is left out of the cell around it. A table over the limits gets no contents and a TABLE_CONTENTS_OMITTED warning; its table-wrap is kept when it has a label or caption and dropped otherwise.

Native text bypass and recognition. bibr/ocr/pdf_inspection.py inspects embedded PDF text, metadata, outline headings, and reference-line geometry under one PDFium walk. With OCR_NATIVE_TEXT_ENABLED=true (the default), qualifying text regions are filled from the PDF text layer and bypass OCR. Acceptance uses character-count and printable-text checks (OCR_NATIVE_TEXT_MIN_CHARS=20, OCR_NATIVE_TEXT_MIN_PRINTABLE_RATIO=0.85) plus guards for corrupt text. A scanned page whose text layer is invisible text over a page-sized image (the layer an OCR engine adds to a scan) is read by OCR instead (OCR_NATIVE_TEXT_REJECT_INVISIBLE_LAYER=true, the default); its layer text is used only when OCR returns no text for the page, or for a region whose OCR request failed; the page then carries an OCR_TEXT_LAYER_FALLBACK warning, and the OCR cache does not store the file. Remaining regions use the selected OCR runtime. Disabling the bypass leaves metadata, outline, and geometry inspection available.

Scan path (bibr/scan/). Each inspected page is classed as born-digital, born-digital with a broken text layer (most of its text regions failed the corruption gates), or a scan (no text layer, or an invisible OCR layer). Two opt-in steps act on scanned pages only. With OCR_CONSENSUS_BACKEND set (glm-http or paddle-http with OCR_CONSENSUS_URL, or a cloud vision provider), a second recognizer re-reads every OCR'd text region of a scanned page; regions whose two readings differ by at least OCR_CONSENSUS_THRESHOLD (normalized edit distance, default 0.15) are ranked and the worst OCR_CONSENSUS_ESCALATE_SHARE (default 7%) escalated. An OCR_ESCALATION_BACKEND re-reads those and the reading closest to the other two is kept; without one the primary reading stays and the region is flagged. A GLM-OCR server used this way needs --max-model-len above the GLM profile's 16384 generation tokens, or every request is refused. Each compared region carries _ocr_consensus, and the paper an OCR_RECOGNIZERS_DISAGREE warning. With PIPELINE_SCAN_ARTICLE_SPLIT=true, the parse stage drops the regions of neighbouring articles from scanned pages: the previous article's tail when it holds that article's references, floats above the paper's title, and everything from a later doc_title that follows the paper's own references (SCAN_ARTICLE_SPLIT warning).

3. Structure (`bibr/structure/pdf_parser.py`)

PDFParser converts OCR regions into PaperContents:

Maps region labels to content handlers via LABEL_TREATMENT dispatch dict
Runs sentence segmentation through the pipeline's wtpsplit-lite segmenter (local or served)
Reconstructs heading hierarchy, incorporates PDF outline hints, and handles multi-study scopes
Extracts tables, figures, footnotes, URL links, and inline figure/table references
Assigns captions and preserves ordered physical parts when figures or tables span multiple regions/pages

4. Extract (`bibr/extract/extractor.py`, `bibr/paper.py`)

Post-parse pipeline runs after structure parsing:

Section classification -- three-tier cascade maps headers to canonical IMRaD categories: alias lookup table, then a trained classifier model, then LLM fallback (section_classifier.py; see Classifiers)
Study hierarchy -- regex markers such as Study 1 and Experiment A establish separate section scopes before classification (section_tree.py)
Metadata extraction -- selected front-matter rows ground title, authors, abstract, and publication fields. JATS and HTML/ePub can supply preparsed metadata, avoiding the core metadata LLM call
Paper classification -- the default MiniLM multitask model predicts paper type and OECD domains from title/abstract; confidence gates and LLM fallback are described in Classifiers
Reference extraction -- segmentation (default geom, a local geometry model, cascading through region anchors -> LLM -> CRF when geometry is absent or unconfident, then checked against a segmentation of the section's printed lines that pools the geometry model's line votes with numbering, author/year onsets, hanging indent, layout boxes and DOI links, and replaces a fallback, under-yielding or clearly weaker result) locates each reference; parsing (default ner, a local encoder-CRF model, with llm for opt-in batched LLM parsing) extracts structured fields
Citation linking -- 3-tier hybrid approach: numeric bracket/superscript citations, author-year citations, then LLM fallback (citation_linker.py)
Equation extraction -- regex + LLM fallback for statistical reporting decomposition (equation_extractor.py)
IMRaD enforcement -- deduplication of Abstract and References by classification-source trust; repeated Methods/Results/Discussion remain valid
Source ownership -- abstract spans, author grounding, and integrity-statement evidence are resolved against the selected article block. Ambiguous front matter can yield validation issues instead of metadata taken from another article in the file
Field decisions -- the title, authors, abstract, keywords, publication date, journal, publisher, paper type, integrity statements, structured funding and affiliations are each decided once, by one rule in bibr/extract/field_decisions.py, from the candidates their producers propose: the model's answer and its grounding repairs, the printed-row title fallbacks (title_candidates.py), the Abstract and Keywords sections, the PDF doc-info, the empty-author recovery and CRediT statement, the paper classifier, the integrity-statement resolver. The rule is the field's only writer; its receipt (the candidates considered, the one used, the rule) stays on Paper.field_decisions and gives extraction.fields its source and rule. The DOI is the identity stage's; the fields a record is built with (volume, issue, pages, ISSN, licence, language, identifiers) have one producer each; the JATS and HTML readers build the record they hand over; a run without an LLM decides no structured funding or affiliations. The structured-integrity call adds contribution roles to the decided authors afterwards, in place; the author receipt records it as a contribution_roles transform

Identity validation and a core checkpoint run before enrichment. The identity stage alone sets the paper's DOI: it chooses among the DOIs the paper prints (sentences, page furniture, the input's structured metadata and, for a PDF, the text layer of pages 1 and 2) and records every candidate in extraction.identity.receipt. A DOI only the PDF's metadata or a link target carries is recorded in the receipt as an agreement-only row: it takes no part in the selection and is never exported. Extraction and enrichment have separate completion evidence, so a failed or delayed external lookup need not force OCR and extraction to run again. The integrity-statement resolver defaults to PIPELINE_INTEGRITY_STATEMENT_MODE=shadow: it records typed evidence while preserving the compatibility scalar fields.

5. Enrich (`bibr/enrich/references.py`)

Opt-in Crossref (and optional bibr-resolver) enrichment of extracted references:

DOI lookup for direct matches
Bibliographic search as fallback (fuzzy title matching)
Matches stay in bib_match and metadata_match; explicit fill/replace consolidation can merge accepted reference fields into bib
Crossref records also bring identifiers: author ORCIDs and affiliation ROR IDs, funders (Open Funder Registry DOI, ROR ID, awards) and the version-of-record license, which land on the match rows
With enrichment on, affiliation strings and funder names are matched to ROR (bibr/clients/ror.py, ROR_ENRICH, default on) into affiliation_match and funding_match. Only ROR's own chosen match is kept. Each process sends at most 1000 ROR requests per 5 minutes (2000 with ROR_CLIENT_ID; ROR_REQUESTS_PER_5MIN sets it), half ROR's limit of 2000 per IP address; a 429 from ROR pauses ROR lookups in the process for its Retry-After (at most 5 minutes) and halves the process's budget (not below 50), and the first lookup after an hour without a 429 doubles it back, one doubling per such hour; a 403 means ROR has blocked the address and pauses ROR lookups for 5 minutes without halving; a string whose request could not start within the paper's ROR time budget is left unmatched at once instead of waiting; strings not matched within ROR_ENRICH_TIMEOUT, or whose lookup failed, stay unmatched with a warning, and a malformed ROR record counts as no match. ROR never marks enrichment partial
Off by default. CROSSREF_ENRICH=true enables it for a deployment; per run, bibr chew --crossref / --no-crossref, chew(crossref=True|False) and the serve API's crossref form field override the setting either way (extraction.settings.crossref_enrich in the output records the effective value)
A served request can enable both reference parsing and Crossref even when REF_PARSE_STRATEGY=off is the deployment default. Enrichment is skipped when the effective reference parsing strategy remains off.
When enabled, the up-front network work (resolver health probe and title searches, Crossref bulk DOI lookup) starts as soon as the references are parsed and overlaps the rest of the extract stage (citation linking, structured integrity), so it no longer adds serial wall time after extraction; the core checkpoint still sees unenriched references

6. Export (`bibr/export/`)

JSON (bibr/export/json_export.py):

JSON-serializable dict matching the bibr v12.3 paper schema
Top-level keys, grouped by role: the paper and its input file — paper_id, schema_version, source; what the paper says — metadata, author, affiliation, funding, text, section, url, bib, xref, figure, table, footnote, eq; what external registries returned — metadata_match, affiliation_match, funding_match, bib_match; how bibr produced the output — extraction. Every key is always present.
Content rows carry no processing fields. Engines, settings, timings, LLM usage, enrichment, warnings (each a stable code and a message), qualification provenance, the output-validation result, diagnostics receipts (section classification, citation-detector tiers, paper-classification confidences, consolidation), the page and bbox of every figure/table piece (float_parts) and the opt-in layout payloads all live under extraction, keyed by the content rows' IDs. A Paper exported outside the pipeline gets a minimal extraction block.
Every record table has an integer primary key named after it (text_id, xref_id, url_id, …); other *_id columns are foreign keys. Absent values are null, never an empty string. Closed vocabularies (section_type, bib_type, paper_type, OECD labels, …) are enums in the schema, and every field carries a description.
xref, url and eq rows carry start/end character spans within their sentence's text. Printed values keep normalized companions where one is unambiguous: metadata.published_date (ISO 8601), license_url/license_spdx, author.credit_roles (CRediT URIs), and declared language/pmid/pmcid/arxiv.
Every bounding box is [x0, y0, x1, y1] in PDF points on the page as displayed, measured from its top-left corner; extraction.pages gives each page's width and height.
Figure and table rows describe the whole object: a figure assembled from several panel crops gets a composited image, and a table continued across pages keeps each printed piece's HTML in html and all rows in contents.
Schema version: 12.3
metadata is scalar-only (no nested objects or lists of objects) so R consumers can as.data.frame(metadata). Pipeline telemetry lives under extraction; the input file's identity under source.
Every id is a 1-based position in document order; sections, figures and tables are renumbered at export, so no id has gaps or depends on how the input was parsed
Captions and footnotes are not sections. Their sentences are text rows after the body, with a null section_id; figure and table rows point at their caption row with text_id, and their section_id is the section they are printed in; footnote has one row per footnote or endnote (printed label, text_id), which foot references target
Enrichment matches are in a separate top-level bib_match array (flat, keyed by bib_id + service)
paper_id is required: the --paper-id, else the input file's stem (its name without the extension); bibr batch sets it to the corpus-unique id its JSON file is named after.
Optional extraction.regions debug payload (per-region bbox/font/content) is opt-in via include_regions=True on Paper.export_to_json() / export_paper_to_json() / the include_regions form field on POST /papers/extract / the --regions CLI flag. It also carries raw_ocr_content, the unmodified model response of each OCR'd region, so diagnostics can compare the original model output with canonical content (they differ where Paddle normalization decoded a table or formula).

Parquet (bibr/export/tables.py): bibr tables <exports> --out DIR, bibr.write_tables(), and bibr batch (into <out>/tables/, after every run) write a corpus as one Parquet file per table: paper (one row per paper), each record and match table, and the extraction_* processing lists. Every row starts with paper_id; column types come from the export models, so all files share one schema whatever papers they hold.

Reference lists (bibr/export/citations.py): Result.to_csl_json(), to_bibtex() and to_ris(), bibr.write_references(), bibr references, the serve route GET /papers/jobs/{id}/references and the MCP tools export_references and save_references write an export's references as CSL-JSON, BibTeX or RIS for reference managers. They read the export and never change it; see the Python guide.

Within major version 12, the schema is additive-only: new optional fields and new enum values may appear in any 12.x release, and readers must ignore keys they don't recognize and accept enum values they don't know. Dispatch on the presence of a root schema_version key, never on parsing its value — pre-v11 payloads have no such key at all. See CHANGELOG.md for the v12 break and forward-versioning policy.

Two JSON Schema documents are generated from the export models. bibr-export-v12.schema.json is exact to what the current release writes: unknown keys are rejected and schema_version is 12.3. bibr-export-v12-reader.schema.json is the reader contract: every object allows unknown keys, every enum accepts values it does not know yet (the known ones are listed as examples), and schema_version may be any 12.x. The PaperExport and PaperExportReader models in bibr.export apply the same two rules in Python. Both documents carry a stable $id under https://bibr.org/schema/, so downstream readers can reference them instead of copying them. The superseded v11 and v10 schemas stay published, frozen.

OCR selection and evidence

PDF OCR starts with PP-DocLayoutV3 regions. OCR_BACKEND=paddle then selects a concrete runtime transactionally at startup: Linux x86_64 tries paddle-vllm (PaddleOCR-VL-1.6) when an NVIDIA GPU with at least 8 GB VRAM is detected, then glm-llama; without that GPU it uses glm-llama directly. Apple Silicon tries paddle-rapid-mlx, paddle-mlx-vlm, glm-rapid-mlx, then glm-llama. The selected backend/model/profile becomes the OCR runtime identity used in the OCR-cache key and export extraction.ocr, so cache entries and provenance cannot be confused across recognizers or normalizers. There is no silent per-request GLM fallback after a concrete runtime has passed startup.

OCR cache entries retain page-attempt and page-failure counts with extraction warnings. Cache hits apply the current OCR_MIN_SUCCESS_RATE, so tightening the threshold also rejects cached OCR that falls below it. Cache format 10, which stores the warnings with their codes, invalidates older entries.

Paddle table output uses OTSL markers (such as <fcel>, <lcel>, <nl>, and <ecel>) that bibr decodes into canonical HTML. Paddle formula output has one outer Markdown/LaTeX fence or balanced display delimiter removed; the LaTeX body is otherwise preserved. The normalized value feeds parsing, while the raw model output of either profile remains available in raw_ocr_content through extraction.regions.

Key data structures

PaperContents (`bibr/paper_contents.py`)

Holds parsed text content built by the native input parsers or PDFParser:

sections -- list of PaperSection (header, section_type, classification_score)
sentences -- list of PaperSentence (text, section_id, paragraph_id, page_number)
tables -- list of PaperTable (caption, HTML markup, cells, physical parts)
links -- list of PaperURLLink
figures -- list of PaperFigure (caption, optional base64 image, physical parts)
xrefs -- list of PaperXref (inline citations, table/figure refs)
equations -- list of PaperEquation (lhs, comp, rhs)
text_df, links_df, sentences_df, equations_df -- cached DataFrame properties

Paper (`bibr/paper.py`)

Top-level dataclass wrapping PaperContents + PaperMetadata:

metadata -- PaperMetadata (title, authors, DOI, keywords, references, matches, paper type, OECD domain)
contents -- PaperContents
export_to_json() -- serialize to dict

PaperMetadata (`bibr/models.py`)

Pydantic model for the paper's own fields and reference list:

doi, title, keywords -- core identifiers
paper_type, paper_type_confidence -- paper type and confidence from the local classifier or LLM fallback
oecd_l1, oecd_l2, oecd_confidence -- OECD Frascati domain taxonomy
authors -- list of PaperAuthor (given, family, affiliation, email, orcid, role)
references -- list of PaperReference (BibTeX fields + nested match dict with ExternalMatch per service)

CanonicalSection (`bibr/paper_contents.py`)

IMRaD+ section classification enum:

Value	Description
`title`	Paper title (root/level-0 heading)
`abstract`	Abstract / Summary
`intro`	Introduction / Background
`method`	Methods / Materials
`results`	Results / Findings
`discussion`	Discussion / Conclusion
`references`	References / Bibliography
`acknowledgment`	Acknowledgments
`funding`	Funding information
`keywords`	Keywords
`endnote`	Supplementary material, future work, outlook
`appendix`	Appendix / Supporting Information
`data_availability`	Data availability / code availability
`author_contributions`	Author Contributions / CRediT statement
`coi`	Conflict of Interest / Competing Interests
`ethics`	Ethics statement / IRB approval / Informed consent
`footnote`	A printed Footnotes or Notes heading
`table`	A printed Tables heading (captions are not sections)
`figure`	A printed Figures heading (captions are not sections)
`unknown`	Unclassified (fallback)

External services

Service	Purpose	Required
OCR runtime (`paddle-` default; `glm-` explicit/fallback)	PDF recognition after layout detection	`bibr chew` selects a local runtime; `bibr serve` proxies to an explicit external OCR service
LLM API (Google/OpenAI/Anthropic/Groq/Ollama or managed local server)	Metadata extraction and fallback tasks	Standard extraction path; `no_llm=True` disables these tasks
Crossref API	Reference enrichment (DOI lookup + search)	Optional
Redis	Response caching (`bibr serve` mode)	Optional

Deployment architecture

`bibr` (single-machine)

Pipeline orchestrator (bibr/local/pipeline.py). Loads models sequentially to fit within limited GPU memory. Components:

Module	Purpose
`bibr/local/cli/`	CLI entry point (`bibr chew`, `bibr batch` and the other subcommands)
`bibr/local/pipeline.py`	`LocalPipeline` orchestrator with memory management
`bibr/local/layout.py`	PP-DocLayoutV3 layout detector
`bibr/local/segmenter.py`	wtpsplit-lite sentence segmenter
`bibr/local/ocr.py`, `bibr/local/vllm_ocr.py`, `bibr/local/mlx_vlm_ocr.py`, `bibr/local/rapid_mlx.py`, `bibr/local/ocr_cloud.py`	OCR backends behind `bibr/ocr/registry.py`: managed `paddle-vllm` / `glm-llama` / `paddle-*-mlx` / `glm-rapid-mlx` servers, `paddle-http` / `glm-http` external servers, and the cloud vision LLMs
`bibr/pipeline/`	The stage pipeline (`stages/`), run context, resources and caches shared by the local and LitServe entry points

Memory management modes control GPU VRAM usage: aggressive (load/unload per phase), balanced (keep layout + segmenter resident; OCR also stays resident across chunks unless a local LLM server needs the VRAM), keep_all (everything loaded).

Auto-selection uses aggressive with ≤8 GB system RAM, a CUDA card below 12 GB that a managed local LLM server shares, or a CUDA card of 8 GB or less; otherwise it uses balanced. With a cloud LLM and non-aggressive memory mode, PIPELINE_STREAM_BACKHALF=true allows completed files' parsing, extraction, enrichment, and export to overlap subsequent OCR work. Managed local LLMs retain the stage barrier for the OCR-to-LLM memory handoff.

`bibr serve` (LitServe)

LitServe separates the HTTP API process from its spawned inference worker. Public multipart bodies never cross that process boundary. The request flow is:

POST /papers/extract
  -> API admission + explicit multipart bounds
     (one file, eight unique options at most 64 bytes each)
  -> one multipart spool (at most 1 MiB in memory)
  -> owner-only disk file (50 MiB file limit; 51 MiB body envelope)
  -> leased UUID/size/SHA-256/options descriptor
  -> LitServe multiprocessing queue
  -> worker verifies one read, deletes file, and runs the pipeline

/_bibr/inference is the private LitServe descriptor route and returns 404 when called over HTTP. It exists only so the API process can dispatch the opaque descriptor through LitServe's in-process route adapter. A descriptor contains no file bytes and no filesystem path.

Component	Module	Purpose
`build_server()`	`bibr/serve/app.py`	Builds the `litserve.LitServer`, public multipart route, private descriptor dispatch, and owned upload lifecycle
`UploadStore` / `InferenceDispatchTracker`	`bibr/serve/ingress.py`	Persists bounded uploads, queues opaque descriptors, and cleans files on consumption, cancellation, failure, or shutdown
`BibrPipelineAPI`	`bibr/serve/deployments/pipeline.py`	Main processing orchestrator
`LayoutDetector`	`bibr/serve/deployments/layout.py`	PP-DocLayoutV3 layout detection (GPU/CPU)
`SentenceSegmenter`	`bibr/serve/deployments/segmenter.py`	wtpsplit-lite sentence segmentation (GPU/CPU)

Exactly one inference worker is pinned in build_server(), with no setting to change it: another worker means another model copy (~1.5 GB RSS), CUDA context, and GpuBatcher, so batches shrink as workers rise. Async I/O and the GPU batchers provide concurrency within that one worker, and the heavy CPU stages already use every core from one process — see Production deployment. Failed inference workers fail-stop by default (PIPELINE_RESTART_WORKERS=false). LitServe 0.2.17 worker replacement cannot reliably notify the API waiter owned by a dead worker, so true remains an unsupported opt-in until the locked real-process death-path gate proves completion notification.

Async jobs persist through the same upload store and submit the same descriptor to LitServe. The queue remains in the HTTP API process; JOBS_STORE=memory keeps results there too, while JOBS_STORE=redis shares status and results between replicas. The server pins exactly one API process per instance, including with jobs disabled. Upload leases keep queued and dispatched descriptors out of stale sweeping. LitServe is pinned below 0.3 because this boundary relies on 0.2.x manager/worker helpers; the spawned success and death-path compatibility tests must be migrated before that version constraint is relaxed.

Configuration

Settings are managed by GlobalSettings (bibr/config.py) using pydantic-settings. Values are read from environment variables or a .env file. Each pipeline owns a settings snapshot; per-run choices such as page range and reference strategies travel through RunConfig, without changing process-global defaults.

Key settings:

Variable	Description	Default
`LLM_PROVIDER`	LLM backend	`google`
`LLM_MODEL`	LLM model name	`gemini-3.5-flash-lite`
`LLM_REASONING_EFFORT`	Default reasoning effort for OpenAI models	`minimal`
`LLM_REASONING_EFFORT_AUTHORS`	Reasoning effort override for author extraction	`low`
`LLM_REASONING_EFFORT_CITATIONS`	Reasoning effort override for citation resolution	`low`
`OCR_BACKEND`	OCR runtime selector/backend	`paddle`
`OCR_BASE_URL`	Base URL for an external HTTP OCR server	`http://localhost:8080`
`CROSSREF_ENRICH`	Enable Crossref/resolver reference enrichment (per-run override: `--crossref`/`--no-crossref`, `chew(crossref=...)`, serve `crossref` field)	`false`
`EQUATION_EXTRACTION`	Enable equation extraction	`true`
`FIGURE_IMAGES`	Include figure images in output, as `data:` URIs	`false`
`REF_SEG_STRATEGY`	Reference segmentation strategy (`geom`, `region`, `llm`, or `crf`)	`geom`
`REF_PARSE_STRATEGY`	Reference parsing strategy (`ner`, `llm`, `llm-chunked`, `ref-llm`, or `off`)	`ner`
`REF_TRAINING_DATA_DIR`	Save raw bib text + LLM extracts (parser data); LLM segmentation spans in `segmentation/` subdir (segmenter data)	disabled
`ENVIRONMENT`	Runtime mode (`development`/`production`)	`development`

For the complete reference, run bibr config example --full or see the Settings reference.

Architecture

Pipeline overview

Pipeline stages

1. Validate (bibr/input/validate.py)

2. Input (bibr/ocr/, bibr/input/docx_native.py)

3. Structure (bibr/structure/pdf_parser.py)

4. Extract (bibr/extract/extractor.py, bibr/paper.py)

5. Enrich (bibr/enrich/references.py)

6. Export (bibr/export/)