Skip to content

Release 2026.3.3 - #993

Merged
bencap merged 113 commits into
mainfrom
release-2026.3.3
Nov 16, 2026
Merged

bencap merged 113 commits into
mainfrom
release-2026.3.3

Conversation

@bencap

@bencap bencap commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Features

Bug Fixes

Maintenance

Breaking API changes

  • DoiIdentifier and RawReadIdentifier no longer have id, and ExternalGeneIdentifier no longer has referenceHtml. DOIs are returned upper-cased, including in the DOI statistics keys.
  • Publication lookups that fail upstream return 502 or 504 instead of 404 or 500.
  • A missing experiment on POST /score-sets/ returns the same 404 detail as a private one, and removing an unlinked collection member as a non-editor returns 403 instead of 409.

Deployment

  • Needs a maintenance window. 2dfdc4a88870 drops the shared identifier tables and d0c23896b0c5 rewrites the citation tables, both of which 2026.3.2 reads, so the running version breaks once they're migrated.
  • Five revisions above main's head 6987f253e02a: 2dfdc4a88870 → d0c23896b0c5 → e59e1cc49c9c → c1bdc0bd24db → 020b7e003efd. None needs an index built by hand. d0c23896b0c5 refuses to run if experiment_set_publication_identifiers has rows.
  • Take an RDS snapshot first.
  • Deploy the worker with VariantEffect/mavedb-deployment#62. Every environment needs the mapping-worker process from this release on, or mapping jobs wait in arq:queue:mapping-lane forever.
  • Dependencies: arq 0.25.0 → 0.26.3, and starlette-context removed.
  • New optional settings, with defaults: CLINGEN_LOOKUP_READ_TIMEOUT and DB_WORKER_LOCK_TIMEOUT_SECONDS.
  • Canonical log fields change (refactor(logging): own the logging context and stop key overwrites #963). The fields that metric filters and alarms use keep their names; the per-check permission fields move under permissions.*, and several worker fields are renamed or removed.
  • Deploy with Release 2026.3.2 mavedb-ui#775, and push the mapper image from Release 2026.3.3 dcd_mapping2#129.
  • After merge: rebase release-2026.4.0 onto main, and poetry publish --build from main.

Issues closed by this release

Generated from the issues linked to PRs merged into release-2026.3.3 for this release, and rewritten on every push. Issues still linked from an open PR are left out. To change the list, change those PRs' linked issues or Closes lines.

bencap and others added 30 commits October 22, 2026 15:09
Saving a file in the editor sorted its imports, but bin/check never checked
import order, so edits picked up reordering unrelated to the change. Selecting
"I" makes the CLI enforce the same order the editor applies.

This commit is the rule change and Ruff's fixes only, in 87 files: reorders
within import groups and dropped blank lines after imports.
FastAPI runs async def handlers on the event loop, and most of ours ran
synchronous SQLAlchemy queries there, so one slow query stalled every
in-flight request on its process. The publication fetchers were async
in name only: they made synchronous HTTP calls to NCBI, bioRxiv and
Crossref on the loop, with no timeout.

Routes that use the database are now def, so FastAPI runs them in its
thread pool. The helpers they awaited, which never awaited anything
real, are sync. MONDO and NCBI Taxonomy use a sync httpx client, and
routes queue jobs through enqueue_pipeline_for_score_set_from_thread,
which writes the pipeline records in the calling thread and hands only
the ARQ call to the loop.

The two upload routes stay async to read the multipart form and run
everything after it, including the CSV rebuild and S3 upload, in the
thread pool. Making them def would mean declaring every form field,
which changes the OpenAPI schema. The schema is unchanged.

Disease search could be async, since it only calls OLS, but it's def to
keep one client style in lib/mondo_ols.py; its comment says when to
revisit that. Tests that were async only because these helpers were are
now sync, with the same assertions.
Walks the app's routes and fails when an async endpoint or dependency
takes a session from get_db, so sync database work can't drift back
onto the event loop. A def dependency that uses the session passes,
since it runs in the thread pool.

FastAPI keeps each included router behind a wrapper rather than copying
its routes into app.routes, so the walk follows original_router and
asserts it finds at least 150 routes; without that, a walk that saw
nothing would pass. It fails on the code before the conversion and on a
route or dependency made async again.
Replaces "never write an async def with no real await" with the rule
the conversion settled on: def by default, async only where the code
can't work without an await, database work kept out of it, and a
comment saying why. The routes rule names the guard test, what may go
on its allowlist, and the thread-pool variant of the pipeline kickoff.

Expect: agents write new routes and helpers as def, and add a comment
giving the reason whenever they write an async route.
…ports

build(lint): sort imports with Ruff's isort rules
…nc-routes-off-loop

perf: run blocking route work in the thread pool
The target gene identifier statistics test updated the first target of
TEST_MINIMAL_ACC_SCORESET and TEST_MINIMAL_SEQ_SCORESET in place, so any
score set test that ran later in the same process created its score set
with three external identifiers and failed its response comparison.

Full-suite runs collect test_score_set.py before test_statistics.py, so
CI never ran a score set test after the mutation. Listing the files in
another order, or a different xdist split, did.

The test now deep-copies the target and passes it as a targetGenes
update.
Model mixins lived in db/mixins.py, and the external identifier base for #944
needs a home too. Splitting mixins between db/ and models/ by whether they're
MaveDB-specific would leave two places named mixins and a judgment call for
each new one. Every model mixin now lives in models/mixins/<concept>.py, and
db/ keeps what every table builds on: Base, MaterializedView and sessions.
The database rule says so, and now loads for db/ too.

ValidTime's code is unchanged apart from its module docstring.
Identifier search fans out from identifiers to the records that cite
them, so it needs a request-scoped read check per record type. Each
viewer delegates to its module's has_permission and memoizes the
result, as the existing viewers do.
… that use them

DOI, raw read and target gene identifiers each lived in a shared table
that records linked to by id, though every column but the identifier
was constant, null or derivable from it. Each identifier now lives on
the row that cites it, so under row-level security (#808) it is an
owned row, visible exactly when its record is.

The six DOI and raw read link tables hold the identifier, keyed by
(<parent>_id, identifier). The three offset tables merge into
target_gene_external_identifiers, keyed by (target_gene_id, db_name).
Writes build link rows from the submitted strings, replacing the
find-or-create helpers. Links are computed on serialization from
lib/external_identifier_urls.py: raw read links follow the accession
type (#445) and gene identifier links, always null before, are filled.

DOIs are validated by a reusable Doi field type and stored bare and
upper-cased, the registry's form, so a record can't cite one DOI twice.
Identifier search returns only identifiers on records the caller can
read. Publications keep their shared table (#945), since their rows
hold fetched metadata.

BREAKING CHANGE: DoiIdentifier and RawReadIdentifier lose id,
ExternalGeneIdentifier loses referenceHtml, and DOIs come back
upper-cased.
Before this change the api accepted DOIs written with a doi: or
resolver prefix and stored them as given. The fold only upper-cased
them, so such a row would have become HTTPS://DOI.ORG/..., with a
broken link and no match against the bare form new submissions
store. Strip the prefixes idutils.normalize_doi strips. The 2026.3.0
dump has none, but production accepts them until this ships.
A search for % or _ matched every identifier. Read access is checked
in Python before the result limit, so that loaded every record that
cites one, on a route anyone can call.
The DOI and raw read link models were listed both among the
association tables and in their own union, and the count narrowed
from one to the other with a type: ignore. Give them their own typed
map, so a new link goes in one place and mypy checks it.
The DOI and raw read search routes each spelled out the same three
link-record joins. identifier_sources_on_records builds them from the
three link models, so a new record type changes one function.
…load

The three offsets these replace were one row each, so joining them in
was cheap. External identifiers are a collection of up to three rows
per gene, and joining a collection in multiplies the rows of every
other joined collection on the target gene and its score set. Load it
in its own query instead.
The search checks read access through the permission viewers, which
import fastapi. View models import lib/identifiers for publication
authors, so they could no longer load without the server extras, and
the core-dependency test jobs failed to collect. The search now lives
in lib/identifier_search, which only the routers import.
The label validator was the only caller of
sanitize_target_sequence_label, which lived in lib/identifiers.py with
the publication code. The validator now does the cleanup itself, as the
sequence validator beside it does.

The test compared the label with the helper's own output, so it passed
whatever the helper did. It now asserts the cleaned label.
The create and update models declared sequenceType as a plain string,
so clients couldn't see the accepted values. The schema now lists them,
built from valid_sequence_types so the two can't drift. The validator
still checks the value with its own message and accepts any case.

The enum is on the input models only. "infer" is never stored, so the
response models keep a plain string.
lib/identifiers.py served the shared identifier tables. Now that DOIs,
raw reads and target gene identifiers are stored on the rows that use
them, nearly all that's left is publication code, under a name that
doesn't say so. Each piece moves, unchanged, to the domain it belongs
to:

- ExternalPublication and the fetch functions go to
  lib/external_publications.py, beside the Crossref and Rxiv clients.
- Finding and storing publications goes to a new lib/publications.py.
- target_gene_external_identifiers goes to lib/target_genes.py, and
  distinct_identifiers to lib/utils.py.

Callers import from the new modules, and test_identifiers.py becomes
test_publications.py.
Citations referenced a publication by its numeric id, so writing one
meant reading the publication first. Under row-level security (#808) a
publication is visible only through a citation the caller can see, so
that read fails for a publication cited only by others' private records,
and for a brand-new one.

Score set, experiment and calibration citations now reference
(db_name, identifier), a foreign key to a new unique key on
publication_identifiers. Foreign key checks ignore row security.
ensure_publication_keys resolves every list a record cites in one pass
and stores new publications with ON CONFLICT DO NOTHING, which skips a
row whether or not the caller can see it. Citations are built from the
keys, so the api never reads a publication to cite it. The unused
experiment_set_publication_identifiers table is dropped.

A publication database failure now raises PublicationServiceError (or
PublicationServiceTimeoutError) where the request goes out. One handler
maps it to 502 or 504 with the `detail` the ui shows. This replaces
per-route try/except blocks, which the calibration routes and
publication search lacked, so those returned 500 when a publication
database failed.

Also fixes along the way:
- A publication listed twice in one submission, such as a PMID and its
  url, is cited once instead of colliding on the primary key.
- A partial score set update keeps its primary publication. It read the
  flag from a temporary attribute on the shared publication object,
  which publications loaded from the database don't have.
primary on score set and experiment citations was nullable, and some
legacy rows hold NULL. Reads already treat NULL as secondary, and a
record that lists one publication in both lists cites it once as
primary, so NULL never meant anything else. The migration stores NULL
as false and makes the column NOT NULL with a false default, so the
column says what reads assume.
Two requests reached a ValueError in lib/score_calibrations.py and
returned 500:

- An admin deleting a primary calibration. Admins may delete any
  calibration, and the delete helper refuses primaries. The route now
  answers 400 and asks for a demotion first. Other users still get the
  403 from the permission check.
- A PUT without scoreSetUrn. The route treated it as optional, but the
  update replaces the calibration and the modify helper requires it.
  The route now answers 422, as the create route does.
Most calibration routes declared only 404, which the router already
declares for every route, so OpenAPI left out the 400, 401, 403 and
gateway errors they return. Each route now declares what it can return:
access control errors where it needs a login or a permission, 400 where
it rejects a request, and gateway errors where it reaches the
publication or disease services. Behaviour doesn't change.
When NCBI failed, we told the user their PMID didn't exist. eutils
raises EutilsRequestError for an error with a JSON body, such as a rate
limit (429), and a global handler turned that into a 404. An error page
it can't parse becomes EutilsNCBIError, which fetch_pubmed_article read
as "no article". Either way a valid PMID looked missing, and the lookup
stopped before trying the other publication databases.

NCBI answers an unknown PMID with an empty article set, not an error.
Identifiers are validated before any fetch, so every eutils error here
means PubMed failed. Both now raise PublicationServiceError, which
answers 502 and lets the lookup try the other databases. The
EutilsRequestError handler had no other use and is removed.
The Crossref and Rxiv clients called requests.get with no timeout, so
a database that accepted the connection and then stalled held the
request open indefinitely, and the 504 for a publication database
timeout could only fire on a failed connection. Both clients now wait
10 seconds. eutils, which fetches from PubMed, doesn't take a timeout.
View models import PublicationAuthors from lib/external_publications.py,
and they must import without the server extras. The publication service
error helper imported the request logging context, which needs FastAPI
and Starlette, so view model tests failed to collect in CI's
core-dependencies job.

The helper now logs without the request context. The exception handler
already records the exception in that context, so the helper's own
save_to_logging_context call was redundant.
Row-level security policies (#809) need to know who is asking. The
principal belongs to the request, but Postgres only sees transactions,
and after a mid-request commit the session's next transaction may run on
a different pooled connection. So bind_principal stores the caller in
session.info, and an after_begin listener re-sends it at the start of
every transaction as app.user_id and app.active_roles.

The values are set with set_config(..., true) and bound parameters, so
they last only for the transaction. A session-level SET would stay on
the pooled connection and leak to the next request. Pinning one
connection per request was rejected: it holds a connection through long
streaming downloads and breaks behind a transaction-mode pooler.

Anonymous callers bind nothing, the same as a session nobody binds, so
anonymous reads one way everywhere. Savepoints don't re-send, because
their enclosing transaction already carries the principal.
get_current_user now binds the caller it resolves on the request's
session. That is the same session the route receives, because FastAPI
caches deps.get_db per request, so every transaction the route runs,
including those after a mid-request commit, carries the caller. Active
roles come only from UserData.active_roles: the JWT path already refuses
a role the user doesn't hold, and does so before anything is bound.

A route that never depends on authentication binds nothing and is
anonymous at the database, even when the request carries credentials.
…ntext

lib/logging/context.py imported starlette-context and lib/logging
re-exported LoggedRoute, so any lib module that recorded logging context
needed FastAPI installed. That kept lib/permissions and a dozen other lib
modules out of the core-dependencies install.

The context now lives in a stdlib ContextVar, and RequestContextMiddleware
replaces starlette-context's middleware and plugins with the same
behaviour: fresh X-Correlation-ID and X-Request-ID per request, echoed on
the response. LoggedRoute is imported from its own module.

Context writes follow one rule per key. Saving a different value to a key
that already holds one is a collision: strict mode raises, production
keeps the newer value and lists the key in context_collisions. increment
and a capped append cover repeated writes, and record_exception keeps
captured_exception_info as the latest error plus the first few in
captured_exceptions.

The error middleware also writes the canonical request log for an
exception raised after the response started, because the app-level
handler that sees it next runs outside the request context.
Nothing imports it now that the logging context and request middleware
are our own. Removes starlette-context 0.3.6 from the server extra and
the lockfile.
jstone-dev and others added 18 commits November 9, 2026 09:18
…t tests

calibration_preference_key and visible_superseding_calibration_urn were
tested only through routes and a single pairing in variant details. The
api and the ui both rely on the preference order, and the second
function is the one place that decides whether a response may name a
draft replacement, so each now has a direct test.
Co-authored-by: Benjamin Capodanno <31941502+bencap@users.noreply.github.com>
Without passive_deletes, deleting a replaced calibration through the
session first nulled its replacement's replaces_id, so the delete
succeeded and the version link silently disappeared. Only
delete_score_calibration refused it; an admin script or another path
would not. The ORM now leaves the column alone and the foreign key
refuses the delete. Deleting the whole score set still removes every
version.

A check constraint also stops a calibration naming itself as the one
it replaces. The same-score-set rule stays in the lib: a composite
foreign key would need its own unique key and the lib would still
have to translate the error.
…ement

Matches the 409 wording: a published replacement is named with the
next step, while a draft's URN stays out because only the caller can
tell whether the user may read it.
A superseded version's columns looked the same as its replacement's,
and versions usually share a title, so nothing told a reader which
interpretation was current. Calibration groups now carry a
`superseded` column, and CSV namespace entries a `superseded` flag.
Superseded versions are offered but unchecked, as research-use-only
ones are; the variant CSV used to drop them without saying why.

The public dump tests now cover version chains: a draft replacement
adds no columns, names no namespace and leaves its predecessor
current, a superseded calibration doesn't raise the VA-Spec layer,
and main.json names only a published replacement.
The worker's process pool had four processes while the worker runs at
most two jobs, and each process kept the memory of the largest task it
had run (a parsed mapping response) until the container restarted. On
production that held about 4 GB per idle worker, and the mapper's BLAT
was OOM-killed on the shared host.

The pool now has one process per job slot and each process exits after
one task, which in Python 3.11 also starts processes with spawn instead
of copying the worker. startup and standalone_ctx build it from one
helper. MAX_JOBS moves to settings/constants.py so the lifecycle hooks
can size the pool without importing the worker classes.
The mapper aligns one score set at a time, so mapping jobs running in
several worker slots queue at the mapper while their time budget runs.
A job that times out is retried, but its request is still in the
mapper's queue, so each retry adds a duplicate the mapper serves anyway.
Adding worker processes for throughput would make that worse.

map_variants_for_score_set now goes to its own ARQ queue, consumed by a
mapping-lane worker (mavedb.worker.MappingLaneWorkerSettings) that runs
one job at a time. The budget then measures the mapping itself, and the
main worker's slots go to work that doesn't need the mapper. A Redis
lock around the mapper call was rejected: jobs would hold worker slots
while waiting for it.

Every environment that runs a worker needs the new process, or mapping
jobs wait in its queue. The main worker still registers the job, so
mapping jobs already on the default queue at deploy time still run.
bin/check-todos fails when a line added since the merge base has a
TODO, FIXME, XXX or HACK that doesn't cite an issue as TODO(#123) or
TODO(owner/repo#123). It checks only added lines, so the TODOs already
in the tree pass until someone edits them. bin/check runs it, and the
CI check job fetches full history so it can find the PR's merge base.

The same script lives in the other MaveDB repos and the workspace.

Expect: agents write new TODOs as TODO(#N) and file or find an issue
first, instead of leaving bare TODOs.
Nothing calls it; visibility is checked through the permissions module.
Rewrites TODOs that already named an issue into one form, TODO(#N):,
cites newly filed or existing issues for TODOs that had none, re-points
TODOs whose issue was closed with the work left undone, and removes
TODOs that were already done or described work nobody needs. TODOs with
no clear issue are left for later; bin/check-todos flags them only when
they're edited.
…nce-source

feat(vep): serve how each VEP consequence was resolved
…ool-recycle

fix(worker): recycle pool processes and give mapping its own lane
…sue-refs

ci: require an issue reference on new TODOs
@bencap bencap added this to the R2026.4 milestone Nov 12, 2026
@bencap bencap added the core: release A release PR label Nov 12, 2026
…settings

#987 moved MAX_JOBS to settings/constants.py so the lifecycle hooks could import it. Merging release-2026.3.3 into #991's branch brought the old definition back into worker.py, which Ruff flags as a redefinition (F811). Both were 2, so nothing changes at runtime.
@coveralls

coveralls commented Nov 12, 2026 •

Copy link
Copy Markdown

Coverage Report for CI Build 43929699309

Coverage increased (+0.8%) to 91.195%

Details

  • Coverage increased (+0.8%) from the base build.
  • Patch coverage: Could not be determined — this PR's diff is too large for GitHub to return (406 error at GitHub).
  • 517 coverage regressions across 41 files.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

517 previously-covered lines in 41 files lost coverage.

Top 10 Files by Coverage Loss Lines Losing Coverage Coverage
src/mavedb/routers/score_sets.py 105 86.89%
src/mavedb/lib/score_sets.py 65 77.02%
src/mavedb/lib/external_publications.py 60 80.95%
src/mavedb/worker/lib/managers/job_manager.py 26 90.68%
src/mavedb/routers/score_calibrations.py 25 92.54%
src/mavedb/lib/exceptions.py 24 66.67%
src/mavedb/models/experiment.py 17 79.35%
src/mavedb/lib/target_genes.py 15 69.05%
src/mavedb/lib/score_calibrations.py 14 96.39%
src/mavedb/routers/experiments.py 13 91.9%

Coverage Stats

Coverage Status
Relevant Lines: 19647
Covered Lines: 17917
Line Coverage: 91.19%
Coverage Strength: 0.91 hits per line

💛 - Coveralls

@bencap
bencap marked this pull request as ready for review November 16, 2026 18:51
@bencap
bencap merged commit bfc0414 into main Nov 16, 2026
13 checks passed
@bencap
bencap deleted the release-2026.3.3 branch November 16, 2026 20:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment