Skip to content

Development setup

Clone and install

Use Python 3.11–3.14 and uv. Python 3.12 is the primary CI environment; some optional model-serving runtimes have narrower Python and platform support. Install libmagic on Linux/macOS if your system does not already provide it (see Installation).

git clone https://github.com/scienceverse/bibr.git
cd bibr

uv sync --locked --extra all --all-groups

--extra all selects the cloud + ML superset (batch, cache, demo, mcp, torch). The hardware-specific serving extras (local, vllm, local-mlx, gpu) are opt-in; --all-extras also resolves if you want every one, subject to their platform markers. local is an Apple Silicon serving extra; it does not install a universal in-process OCR engine. The nuextract extra, needed only by the experimental LLM_STRUCTURED_BACKEND=nuextract-native backend, is opt-in too.

For core code or documentation work without local ML models:

uv sync --locked --group docs

Tests set placeholder configuration in tests/conftest.py and mock external services. API keys and model downloads are needed only for the live workflows you choose to run. Use uv run bibr setup to configure one of those workflows.

Running tests

uv run --locked --extra all pytest -m "not slow"

This selects the main CI suite. Core-only CI also runs the non-slow suite with uv run --locked pytest -m "not slow"; tests that need absent optional dependencies should use pytest.importorskip(). Slow tests are excluded by the -m expression, not by pytest's default configuration. Read a slow test's requirements before enabling it: it may need model downloads, an OCR service, GPU hardware, or LLM credentials. Unit tests stay offline: an autouse guard fails any test that opens a non-loopback socket or resolves an external name, and global Settings changes plus bibr.* logger levels are restored after every test. Tests that intentionally reach the network take the network mark, which opts them out of the socket guard.

Linting and formatting

uv run ruff check .
uv run ruff format .
uv run ruff format --check .  # non-mutating CI check

bibr uses Ruff for both. Line length is 100 characters, double quotes, Python 3.11+ target.

Previewing the docs

uv run --locked --group docs mkdocs serve
uv run --locked --group docs mkdocs build --strict

Some reference pages (reference/settings.md, reference/cli.md, reference/schema.md) are generated at build time from scripts/docs_ref_core.py reading live code and docstrings — don't edit the built output directly. To change what those pages say, edit scripts/docs_ref_core.py or the docstrings/field descriptions it reads from. The docs build also validates internal links; add pages to mkdocs.yml when they should appear in navigation.

For the documentation contracts and CLI examples:

uv run --locked pytest tests/test_docs_sync.py

Conventions

  • Line length 100, double quotes, py311+ target (see pyproject.toml for the full Ruff rule set)
  • Lazy __getattr__ imports in __init__.py for subpackages with heavy dependencies (transformers, torch, etc.), so import bibr stays light
  • Exceptions derive from BibrError; use the appropriate input, processing, upstream-service, or configuration error in bibr/exceptions.py
  • Settings in bibr/config.py lazily exposes GlobalSettings; settings can also be passed explicitly to pipeline instances. Read environment variables and .env through this configuration layer

Publishing a release

The release workflow builds the wheel and source archive once, checks their contents and PyPI metadata, and installs the wheel in a clean environment. It publishes those same files through PyPI Trusted Publishing with attestations. The package description comes from pyproject.toml; the PyPI project page comes from README.md. Update both before releasing, including installation instructions and links that work outside GitHub.

The registered PyPI publisher must match these values:

Setting Value
PyPI project bibr
GitHub owner/repository scienceverse/bibr
Workflow filename release.yml
GitHub environment pypi

The pypi environment allows v* tags. No PyPI API token is needed. Keep the repository variable PUBLISH_PYPI=true for normal publishing. A maintainer's version tag starts the release workflow, which publishes only after its tests and distribution checks pass. Enable this once during publishing setup:

gh variable set PUBLISH_PYPI --repo scienceverse/bibr --body true

Leave publishing enabled after a successful release. Set the variable to false only when deliberately suspending PyPI delivery; tags created while it is disabled skip the PyPI upload. Manual release rehearsals never publish.

Container registry delivery is a separate opt-in: PUBLISH_GHCR=true enables edge and release uploads, including manual container workflow runs. It is disabled for the initial public 0.5.0 launch; users can build the containers from the public source. CI still builds the serve image and blocks fixable HIGH/CRITICAL vulnerabilities before CI / required passes.

A manual container workflow run publishes only a commit that is on main. A release run also needs the vX.Y.Z tag on that commit and a matching project.version, and it refuses to move an already published vX.Y.Z image tag to a different digest. The X.Y, X and latest image tags move only when X.Y.Z is the newest release tag on main in that series, so publishing a backport or an older release never moves them backwards. Release images are built without any layer cache, so a rebuild never reproduces a published digest: if promotion fails after vX.Y.Z was pushed, point the remaining tags at that digest by hand with docker buildx imagetools create instead of rerunning the job.

Before enabling GHCR, configure a clean package with the intended visibility and repository Actions access, then verify anonymous pulls for public images. Do not expose a legacy private package's old versions as part of that setup. See Docker deployment.

  1. Update the package version, lockfile, changelog, and public documentation on main. Update version and date-released in CITATION.cff for the release, and check its authors and license. Wait for CI / required to pass on the exact commit to be released.
  2. Rehearse the release from main with gh workflow run release.yml --repo scienceverse/bibr --ref main. Wait for the Ubuntu, macOS, Windows, and distribution checks to pass. A manual rehearsal cannot publish to PyPI, GHCR, or GitHub Releases.
  3. After release approval, create and push an annotated vX.Y.Z tag on the verified commit, with X.Y.Z matching project.version. The workflow rejects mismatched tags and commits that are not reachable from main. A pre-release version such as 1.2.0rc1 goes to PyPI and becomes a GitHub pre-release that is never marked Latest; it publishes no container image, because the container channel publishes only X.Y.Z.
  4. Watch the tag-triggered Release workflow to completion. PyPI receives the verified distributions and GitHub Release assets are attached after each enabled delivery channel succeeds. If GHCR is enabled, the release container must also pass its digest scan before its version tags are promoted. A failed enabled channel blocks finalization; only deliberately disabled channels may be skipped.
  5. Verify the live PyPI description and install that exact version from PyPI in a fresh environment.
  6. If Zenodo integration is enabled, verify that the GitHub release was archived successfully and that its version, authors, license, and DOI are correct.

If publication partially succeeds, rerun only the failed jobs of that same run. Do not rerun a successful PyPI upload or move a published release tag. PyPI files are immutable; package or description corrections require a new version.

Zenodo archiving and citation metadata

The root CITATION.cff provides citation metadata for GitHub and Zenodo. Keep it as the single metadata source; a .zenodo.json file would override it in Zenodo.

To enable automatic archiving, connect the maintainer's GitHub account to Zenodo, open the profile menu's GitHub page, select Sync now, and enable scienceverse/bibr. This is a one-time account setup; it does not require a Zenodo token in GitHub Actions. Subsequent GitHub releases are archived automatically. See Zenodo's repository setup guide.

Enabling integration does not create a DOI for releases that already exist. For an earlier release, use Zenodo's manual software upload workflow with that release's source archive, or wait for the next normal release.

After the first archive is published, add its project-wide concept DOI badge to the README. Research using a particular release should cite that release's version DOI. Only add a top-level doi to CITATION.cff if it identifies the listed version; remove or update it when preparing a different release. See Zenodo's DOI versioning guidance.

Submitting a pull request

  • Branch from main, keep PRs focused on a single concern, and use descriptive branch names without codex/ or claude/ prefixes
  • Add tests for new functionality
  • CI must be green, including the strict docs build (mkdocs build --strict)
  • Update user-facing docs when CLI options, defaults, input formats, output fields, or deployment behavior change
  • A CLA is required before your first PR can be merged — see CONTRIBUTING.md for details