Repository navigation
Release 2026.3.3 - #993
Merged
Merged
Release 2026.3.3#993
Conversation
Saving a file in the editor sorted its imports, but bin/check never checked import order, so edits picked up reordering unrelated to the change. Selecting "I" makes the CLI enforce the same order the editor applies. This commit is the rule change and Ruff's fixes only, in 87 files: reorders within import groups and dropped blank lines after imports.
FastAPI runs async def handlers on the event loop, and most of ours ran synchronous SQLAlchemy queries there, so one slow query stalled every in-flight request on its process. The publication fetchers were async in name only: they made synchronous HTTP calls to NCBI, bioRxiv and Crossref on the loop, with no timeout. Routes that use the database are now def, so FastAPI runs them in its thread pool. The helpers they awaited, which never awaited anything real, are sync. MONDO and NCBI Taxonomy use a sync httpx client, and routes queue jobs through enqueue_pipeline_for_score_set_from_thread, which writes the pipeline records in the calling thread and hands only the ARQ call to the loop. The two upload routes stay async to read the multipart form and run everything after it, including the CSV rebuild and S3 upload, in the thread pool. Making them def would mean declaring every form field, which changes the OpenAPI schema. The schema is unchanged. Disease search could be async, since it only calls OLS, but it's def to keep one client style in lib/mondo_ols.py; its comment says when to revisit that. Tests that were async only because these helpers were are now sync, with the same assertions.
Walks the app's routes and fails when an async endpoint or dependency takes a session from get_db, so sync database work can't drift back onto the event loop. A def dependency that uses the session passes, since it runs in the thread pool. FastAPI keeps each included router behind a wrapper rather than copying its routes into app.routes, so the walk follows original_router and asserts it finds at least 150 routes; without that, a walk that saw nothing would pass. It fails on the code before the conversion and on a route or dependency made async again.
Replaces "never write an async def with no real await" with the rule the conversion settled on: def by default, async only where the code can't work without an await, database work kept out of it, and a comment saying why. The routes rule names the guard test, what may go on its allowlist, and the thread-pool variant of the pipeline kickoff. Expect: agents write new routes and helpers as def, and add a comment giving the reason whenever they write an async route.
…ports build(lint): sort imports with Ruff's isort rules
…nc-routes-off-loop perf: run blocking route work in the thread pool
The target gene identifier statistics test updated the first target of TEST_MINIMAL_ACC_SCORESET and TEST_MINIMAL_SEQ_SCORESET in place, so any score set test that ran later in the same process created its score set with three external identifiers and failed its response comparison. Full-suite runs collect test_score_set.py before test_statistics.py, so CI never ran a score set test after the mutation. Listing the files in another order, or a different xdist split, did. The test now deep-copies the target and passes it as a targetGenes update.
Model mixins lived in db/mixins.py, and the external identifier base for #944 needs a home too. Splitting mixins between db/ and models/ by whether they're MaveDB-specific would leave two places named mixins and a judgment call for each new one. Every model mixin now lives in models/mixins/<concept>.py, and db/ keeps what every table builds on: Base, MaterializedView and sessions. The database rule says so, and now loads for db/ too. ValidTime's code is unchanged apart from its module docstring.
Identifier search fans out from identifiers to the records that cite them, so it needs a request-scoped read check per record type. Each viewer delegates to its module's has_permission and memoizes the result, as the existing viewers do.
… that use them DOI, raw read and target gene identifiers each lived in a shared table that records linked to by id, though every column but the identifier was constant, null or derivable from it. Each identifier now lives on the row that cites it, so under row-level security (#808) it is an owned row, visible exactly when its record is. The six DOI and raw read link tables hold the identifier, keyed by (<parent>_id, identifier). The three offset tables merge into target_gene_external_identifiers, keyed by (target_gene_id, db_name). Writes build link rows from the submitted strings, replacing the find-or-create helpers. Links are computed on serialization from lib/external_identifier_urls.py: raw read links follow the accession type (#445) and gene identifier links, always null before, are filled. DOIs are validated by a reusable Doi field type and stored bare and upper-cased, the registry's form, so a record can't cite one DOI twice. Identifier search returns only identifiers on records the caller can read. Publications keep their shared table (#945), since their rows hold fetched metadata. BREAKING CHANGE: DoiIdentifier and RawReadIdentifier lose id, ExternalGeneIdentifier loses referenceHtml, and DOIs come back upper-cased.
Before this change the api accepted DOIs written with a doi: or resolver prefix and stored them as given. The fold only upper-cased them, so such a row would have become HTTPS://DOI.ORG/..., with a broken link and no match against the bare form new submissions store. Strip the prefixes idutils.normalize_doi strips. The 2026.3.0 dump has none, but production accepts them until this ships.
A search for % or _ matched every identifier. Read access is checked in Python before the result limit, so that loaded every record that cites one, on a route anyone can call.
The DOI and raw read link models were listed both among the association tables and in their own union, and the count narrowed from one to the other with a type: ignore. Give them their own typed map, so a new link goes in one place and mypy checks it.
The DOI and raw read search routes each spelled out the same three link-record joins. identifier_sources_on_records builds them from the three link models, so a new record type changes one function.
…load The three offsets these replace were one row each, so joining them in was cheap. External identifiers are a collection of up to three rows per gene, and joining a collection in multiplies the rows of every other joined collection on the target gene and its score set. Load it in its own query instead.
The search checks read access through the permission viewers, which import fastapi. View models import lib/identifiers for publication authors, so they could no longer load without the server extras, and the core-dependency test jobs failed to collect. The search now lives in lib/identifier_search, which only the routers import.
The label validator was the only caller of sanitize_target_sequence_label, which lived in lib/identifiers.py with the publication code. The validator now does the cleanup itself, as the sequence validator beside it does. The test compared the label with the helper's own output, so it passed whatever the helper did. It now asserts the cleaned label.
The create and update models declared sequenceType as a plain string, so clients couldn't see the accepted values. The schema now lists them, built from valid_sequence_types so the two can't drift. The validator still checks the value with its own message and accepts any case. The enum is on the input models only. "infer" is never stored, so the response models keep a plain string.
lib/identifiers.py served the shared identifier tables. Now that DOIs, raw reads and target gene identifiers are stored on the rows that use them, nearly all that's left is publication code, under a name that doesn't say so. Each piece moves, unchanged, to the domain it belongs to: - ExternalPublication and the fetch functions go to lib/external_publications.py, beside the Crossref and Rxiv clients. - Finding and storing publications goes to a new lib/publications.py. - target_gene_external_identifiers goes to lib/target_genes.py, and distinct_identifiers to lib/utils.py. Callers import from the new modules, and test_identifiers.py becomes test_publications.py.
Citations referenced a publication by its numeric id, so writing one meant reading the publication first. Under row-level security (#808) a publication is visible only through a citation the caller can see, so that read fails for a publication cited only by others' private records, and for a brand-new one. Score set, experiment and calibration citations now reference (db_name, identifier), a foreign key to a new unique key on publication_identifiers. Foreign key checks ignore row security. ensure_publication_keys resolves every list a record cites in one pass and stores new publications with ON CONFLICT DO NOTHING, which skips a row whether or not the caller can see it. Citations are built from the keys, so the api never reads a publication to cite it. The unused experiment_set_publication_identifiers table is dropped. A publication database failure now raises PublicationServiceError (or PublicationServiceTimeoutError) where the request goes out. One handler maps it to 502 or 504 with the `detail` the ui shows. This replaces per-route try/except blocks, which the calibration routes and publication search lacked, so those returned 500 when a publication database failed. Also fixes along the way: - A publication listed twice in one submission, such as a PMID and its url, is cited once instead of colliding on the primary key. - A partial score set update keeps its primary publication. It read the flag from a temporary attribute on the shared publication object, which publications loaded from the database don't have.
primary on score set and experiment citations was nullable, and some legacy rows hold NULL. Reads already treat NULL as secondary, and a record that lists one publication in both lists cites it once as primary, so NULL never meant anything else. The migration stores NULL as false and makes the column NOT NULL with a false default, so the column says what reads assume.
Two requests reached a ValueError in lib/score_calibrations.py and returned 500: - An admin deleting a primary calibration. Admins may delete any calibration, and the delete helper refuses primaries. The route now answers 400 and asks for a demotion first. Other users still get the 403 from the permission check. - A PUT without scoreSetUrn. The route treated it as optional, but the update replaces the calibration and the modify helper requires it. The route now answers 422, as the create route does.
Most calibration routes declared only 404, which the router already declares for every route, so OpenAPI left out the 400, 401, 403 and gateway errors they return. Each route now declares what it can return: access control errors where it needs a login or a permission, 400 where it rejects a request, and gateway errors where it reaches the publication or disease services. Behaviour doesn't change.
When NCBI failed, we told the user their PMID didn't exist. eutils raises EutilsRequestError for an error with a JSON body, such as a rate limit (429), and a global handler turned that into a 404. An error page it can't parse becomes EutilsNCBIError, which fetch_pubmed_article read as "no article". Either way a valid PMID looked missing, and the lookup stopped before trying the other publication databases. NCBI answers an unknown PMID with an empty article set, not an error. Identifiers are validated before any fetch, so every eutils error here means PubMed failed. Both now raise PublicationServiceError, which answers 502 and lets the lookup try the other databases. The EutilsRequestError handler had no other use and is removed.
The Crossref and Rxiv clients called requests.get with no timeout, so a database that accepted the connection and then stalled held the request open indefinitely, and the 504 for a publication database timeout could only fire on a failed connection. Both clients now wait 10 seconds. eutils, which fetches from PubMed, doesn't take a timeout.
View models import PublicationAuthors from lib/external_publications.py, and they must import without the server extras. The publication service error helper imported the request logging context, which needs FastAPI and Starlette, so view model tests failed to collect in CI's core-dependencies job. The helper now logs without the request context. The exception handler already records the exception in that context, so the helper's own save_to_logging_context call was redundant.
Row-level security policies (#809) need to know who is asking. The principal belongs to the request, but Postgres only sees transactions, and after a mid-request commit the session's next transaction may run on a different pooled connection. So bind_principal stores the caller in session.info, and an after_begin listener re-sends it at the start of every transaction as app.user_id and app.active_roles. The values are set with set_config(..., true) and bound parameters, so they last only for the transaction. A session-level SET would stay on the pooled connection and leak to the next request. Pinning one connection per request was rejected: it holds a connection through long streaming downloads and breaks behind a transaction-mode pooler. Anonymous callers bind nothing, the same as a session nobody binds, so anonymous reads one way everywhere. Savepoints don't re-send, because their enclosing transaction already carries the principal.
get_current_user now binds the caller it resolves on the request's session. That is the same session the route receives, because FastAPI caches deps.get_db per request, so every transaction the route runs, including those after a mid-request commit, carries the caller. Active roles come only from UserData.active_roles: the JWT path already refuses a role the user doesn't hold, and does so before anything is bound. A route that never depends on authentication binds nothing and is anonymous at the database, even when the request carries credentials.
…ntext lib/logging/context.py imported starlette-context and lib/logging re-exported LoggedRoute, so any lib module that recorded logging context needed FastAPI installed. That kept lib/permissions and a dozen other lib modules out of the core-dependencies install. The context now lives in a stdlib ContextVar, and RequestContextMiddleware replaces starlette-context's middleware and plugins with the same behaviour: fresh X-Correlation-ID and X-Request-ID per request, echoed on the response. LoggedRoute is imported from its own module. Context writes follow one rule per key. Saving a different value to a key that already holds one is a collision: strict mode raises, production keeps the newer value and lists the key in context_collisions. increment and a capped append cover repeated writes, and record_exception keeps captured_exception_info as the latest error plus the first few in captured_exceptions. The error middleware also writes the canonical request log for an exception raised after the response started, because the app-level handler that sees it next runs outside the request context.
Nothing imports it now that the logging context and request middleware are our own. Removes starlette-context 0.3.6 from the server extra and the lockfile.
…t tests calibration_preference_key and visible_superseding_calibration_urn were tested only through routes and a single pairing in variant details. The api and the ui both rely on the preference order, and the second function is the one place that decides whether a response may name a draft replacement, so each now has a direct test.
Co-authored-by: Benjamin Capodanno <31941502+bencap@users.noreply.github.com>
Without passive_deletes, deleting a replaced calibration through the session first nulled its replacement's replaces_id, so the delete succeeded and the version link silently disappeared. Only delete_score_calibration refused it; an admin script or another path would not. The ORM now leaves the column alone and the foreign key refuses the delete. Deleting the whole score set still removes every version. A check constraint also stops a calibration naming itself as the one it replaces. The same-score-set rule stays in the lib: a composite foreign key would need its own unique key and the lib would still have to translate the error.
…ement Matches the 409 wording: a published replacement is named with the next step, while a draft's URN stays out because only the caller can tell whether the user may read it.
A superseded version's columns looked the same as its replacement's, and versions usually share a title, so nothing told a reader which interpretation was current. Calibration groups now carry a `superseded` column, and CSV namespace entries a `superseded` flag. Superseded versions are offered but unchecked, as research-use-only ones are; the variant CSV used to drop them without saying why. The public dump tests now cover version chains: a draft replacement adds no columns, names no namespace and leaves its predecessor current, a superseded calibration doesn't raise the VA-Spec layer, and main.json names only a published replacement.
The worker's process pool had four processes while the worker runs at most two jobs, and each process kept the memory of the largest task it had run (a parsed mapping response) until the container restarted. On production that held about 4 GB per idle worker, and the mapper's BLAT was OOM-killed on the shared host. The pool now has one process per job slot and each process exits after one task, which in Python 3.11 also starts processes with spawn instead of copying the worker. startup and standalone_ctx build it from one helper. MAX_JOBS moves to settings/constants.py so the lifecycle hooks can size the pool without importing the worker classes.
The mapper aligns one score set at a time, so mapping jobs running in several worker slots queue at the mapper while their time budget runs. A job that times out is retried, but its request is still in the mapper's queue, so each retry adds a duplicate the mapper serves anyway. Adding worker processes for throughput would make that worse. map_variants_for_score_set now goes to its own ARQ queue, consumed by a mapping-lane worker (mavedb.worker.MappingLaneWorkerSettings) that runs one job at a time. The budget then measures the mapping itself, and the main worker's slots go to work that doesn't need the mapper. A Redis lock around the mapper call was rejected: jobs would hold worker slots while waiting for it. Every environment that runs a worker needs the new process, or mapping jobs wait in its queue. The main worker still registers the job, so mapping jobs already on the default queue at deploy time still run.
bin/check-todos fails when a line added since the merge base has a TODO, FIXME, XXX or HACK that doesn't cite an issue as TODO(#123) or TODO(owner/repo#123). It checks only added lines, so the TODOs already in the tree pass until someone edits them. bin/check runs it, and the CI check job fetches full history so it can find the PR's merge base. The same script lives in the other MaveDB repos and the workspace. Expect: agents write new TODOs as TODO(#N) and file or find an issue first, instead of leaving bare TODOs.
Nothing calls it; visibility is checked through the permissions module.
Rewrites TODOs that already named an issue into one form, TODO(#N):, cites newly filed or existing issues for TODOs that had none, re-points TODOs whose issue was closed with the work left undone, and removes TODOs that were already done or described work nobody needs. TODOs with no clear issue are left for later; bin/check-todos flags them only when they're edited.
…sioned-calibrations
…nce-source feat(vep): serve how each VEP consequence was resolved
…ool-recycle fix(worker): recycle pool processes and give mapping its own lane
…sue-refs ci: require an issue reference on new TODOs
Coverage Report for CI Build 43929699309Coverage increased (+0.8%) to 91.195%Details
Uncovered ChangesNo uncovered changes found. Coverage Regressions517 previously-covered lines in 41 files lost coverage.
Coverage Stats
💛 - Coveralls |
This was referenced Nov 13, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
supersededCalibrationUrn,supersedingCalibrationUrn,hasPublishedReplacementandpublishedDate. Ships with feat(calibration): create, list and pick versions of published calibrations mavedb-ui#759.(db_name, identifier)instead of by id. Publication database failures now answer 502/504 from one handler; PubMed rate limits and outages were reported as a missing PMID (404), and calibration saves and publication search returned 500. Crossref and bioRxiv requests time out after 10 seconds. A publication listed twice in one submission is cited once instead of failing the save.app.user_id,app.active_roles). Nothing reads them until row-level security policies land.totalUnmapped, the number of published score sets named for the gene that aren't mapped to it, so the gene page can point to them. Ships with feat(gene): point to score sets not yet mapped to the gene mavedb-ui#761.vepblock ofGET /variants/{urn}andGET /alleles/{digest}says how the consequence was resolved:consequenceSource,consequenceTermsandmatchedTranscript. Ships with fix(variant): prefer a transcript VEP call over a genomic worst case mavedb-ui#766.Bug Fixes
GET /score-sets/{urn}/variants, so score sets of about 500k variants load instead of timing out at the load balancer. The body is byte for byte what it was. On a 648k-variant score set, the first byte arrives in 0.4 s instead of 17.5 s, and peak memory drops from 1.9 GB to 340 MB.POST /variants/clingen-allele-id-lookups, removed in 2026.3.0, answers 410 Gone with deprecation headers pointing atGET /clingen-alleles/{clingen_allele_id}/measurements, instead of a bare 405. MobiDetails and ClinGen's Variant Curation Interface still call it.Maintenance
ContextVarwith our own request middleware, each worker job gets its own context, and writes to a key that already holds a different value are recorded as collisions instead of silently overwriting. Removes thestarlette-contextdependency.mavedb.libimports without the server extra.bin/checkrequires an issue reference on new TODOs.Breaking API changes
DoiIdentifierandRawReadIdentifierno longer haveid, andExternalGeneIdentifierno longer hasreferenceHtml. DOIs are returned upper-cased, including in the DOI statistics keys.POST /score-sets/returns the same 404 detail as a private one, and removing an unlinked collection member as a non-editor returns 403 instead of 409.Deployment
2dfdc4a88870drops the shared identifier tables andd0c23896b0c5rewrites the citation tables, both of which 2026.3.2 reads, so the running version breaks once they're migrated.main's head6987f253e02a:2dfdc4a88870→d0c23896b0c5→e59e1cc49c9c→c1bdc0bd24db→020b7e003efd. None needs an index built by hand.d0c23896b0c5refuses to run ifexperiment_set_publication_identifiershas rows.mapping-workerprocess from this release on, or mapping jobs wait inarq:queue:mapping-laneforever.starlette-contextremoved.CLINGEN_LOOKUP_READ_TIMEOUTandDB_WORKER_LOCK_TIMEOUT_SECONDS.permissions.*, and several worker fields are renamed or removed.release-2026.4.0ontomain, andpoetry publish --buildfrommain.Issues closed by this release
Generated from the issues linked to PRs merged into
release-2026.3.3for this release, and rewritten on every push. Issues still linked from an open PR are left out. To change the list, change those PRs' linked issues or Closes lines.