Repository navigation
direct: use the postgres branch when its create request times out - #7602
Merged
Merged
Conversation
Contributor
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 26 tests: regression=2 unreleased=0 coverage=24 failing=0
TestAccept/bundle/resources/postgres_branches/create_504/DMS=: branch ✅ | main (bdc5f0e) ❌ | latest (v1.26.0) ❌main (bdc5f0e): latest (v1.26.0): same output as main (bdc5f0e) TestAccept/bundle/resources/postgres_branches/create_504/DMS=true: branch ✅ | main (bdc5f0e) ❌ | latest (v1.26.0) ❌main (bdc5f0e): latest (v1.26.0): same output as main (bdc5f0e) |
Collaborator
✅ Integration test reportCommit: 1b7ce29
Top 46 slowest tests (at least 2 minutes):
|
Sankalp-Mittal
left a comment
Contributor
There was a problem hiding this comment.
🤖 AI review from review-factory: 3 independent reviewer models (Claude Sonnet 5.5, Claude Opus 5.5, Claude Fable 5.2) with cross-verification. Only findings that reviewers agreed on are posted.
Postgres branch create now recovers from transient errors by reading the branch; reviewers flagged untested paths and an unsuppressed warning.
Findings: 2 major, 3 minor, 1 nit.
andrewnester
approved these changes
Nov 17, 2026
Creating a postgres branch sometimes fails with 504 DEADLINE_EXCEEDED, "The create request timed out and may still be completing. Check whether the resource was created before retrying." The deploy failed even when the branch had been created. On a transient create error, read the branch: if it exists, wait until it is READY and use it; if it does not, repeat the create (retrySafe). With replace_existing the request is repeated directly, since a branch found then may predate it. cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing Co-authored-by: Isaac <no-reply@databricks.com>
cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing Co-authored-by: Isaac <no-reply@databricks.com>
Addresses review: a found branch in DELETED/ARCHIVED never becomes READY, so fail at once instead of polling for 10 minutes; retry transient errors on the GET used to check the branch. cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing Co-authored-by: Isaac <no-reply@databricks.com>
- Repeat the create request inside DoCreate while the branch is missing, and treat ALREADY_EXISTS on a repeat as the earlier request having landed, so the next read finds the branch. The repeat used to go back to the caller's retry, where ALREADY_EXISTS ended the deploy. - Use a found branch only if it was created at or after the first request (with a minute of clock skew), so a branch that predates it is not taken over. - Once the branch is ours, return it even if the wait for READY ends early, so state records it and the next deploy does not fail with ALREADY_EXISTS. - Pass a cancelled context through instead of reporting a failed existence check. - Log using the recovered branch at Info, so acceptance goldens are not affected when the cloud deploy recovers. - Make the wait timeout and poll interval variables, and cover every recovery path in unit tests; add acceptance scenarios for replace_existing and a failed read. - Add retry.Pause, the log-and-wait step of retry.With, for loops like this one. cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing TestAccept/bundle/resources/postgres_databases/update Co-authored-by: Isaac <no-reply@databricks.com>
denik
force-pushed
the
postgres-branch-create-504
branch
from
November 17, 2026 13:23
50c2d41 to
6cb360a
Compare
On Windows the clock can return the same instant twice, so with a zero timeout the wait read the branch once more. cloud: TestAccept/bundle/resources/postgres_branches TestAccept/bundle/resources/postgres_roles/replace_existing TestAccept/bundle/resources/postgres_databases/update Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
When the request to create a postgres branch fails with a transient error (5xx, 408, inactivity timeout), the direct engine now looks for the branch before giving up. It repeats the request while the branch is missing:
READY, use it. If it isINIT, wait forREADY(up to 10 minutes); if the wait ends early, use it anyway, with a warning, so state records it. In any other state (DELETED,ARCHIVED, ...), fail with the original error.create_time, allowing a minute of clock skew): fail with the original error. It isn't taken over.retry.MaxRetriestimes. A repeat that fails withALREADY_EXISTSmeans the earlier request has landed in the meantime; the next look finds that branch and uses it.replace_existing: true: the error is markedretrySafe, andDeploymentUnit.Createrepeats the create without reading the branch. A branch found at this point may predate the request, and repeating a replace is safe.libs/retrygainsPause, the log-and-wait step ofretry.With, which the loop above uses so its retries print the usualretrying after ...warning.Follow-up to #7557, which retried transient errors while polling create operations and on the project create request. Branches were left out there because they weren't failing yet.
Judgment calls:
READYstays inline inDoCreate, as the create-operation wait on the success path does, rather than moving toWaitAfterCreate. The latter would read every created branch once more, for a path that only runs after a failed create.Why
The Lakebase API sometimes answers a branch create with a 504 that says the branch may still be created. The deploy failed even when the branch had been created, and dependent resources failed with it.
Dashboard:
Error: cannot create resources.postgres_branches.main: The create request timed out and may still be completing. Check whether the resource was created before retrying. (504 DEADLINE_EXCEEDED)onPOST [DATABRICKS_URL]/api/2.0/postgres/projects/test-pg-proj-[UNIQUE_NAME]/branches?branch_id=mainTests: postgres_databases/update/DMS=true, postgres_roles/replace_existing/DMS=.
Tests
bundle/direct/dresources/postgres_branch_test.godriveDoCreateagainst a testserver with scripted responses. They cover every recovery path:INITthenREADY,INITpast the timeout,DELETED, a failed read, a cancelled context,replace_existing, a non-transient error, a repeat that succeeds, a repeat that getsALREADY_EXISTS, a branch that stays missing, and a branch that predates the request.bundle/resources/postgres_branches/create_504, with four 504 scenarios:replace_existing: the create is repeated without a read.bundle/resources/postgres_*,bundle/deploy,bundle/upload,bundle/syncandbundle/resourcesacceptance tests, and thebundle/direct/...andlibs/retryunit tests, pass.postgres_branches,postgres_roles/replace_existingandpostgres_databases/updateon every cloud.This pull request and its description were written by Isaac.