Skip to content

fix(start): reconcile dead-lettered submissions to Error - #1040

Draft
sbalabanov wants to merge 7 commits into
wt/sqc009from
wt/sqc011
Draft

sbalabanov wants to merge 7 commits into
wt/sqc009from
wt/sqc011

Conversation

@sbalabanov

@sbalabanov sbalabanov commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

TL;DR: Report failed starts as Error instead of leaving submissions stuck at Accepted.

Summary

A failed start message used to be discarded when no Request had been saved, leaving the submission at Accepted. The DLQ now records a complete failed Request and sends an Error status without restarting validation or normal work. Existing batch owners and terminal winners are preserved.

Builds on #1039.

Repro

  1. Accept a valid submission, but let its initial request save fail until the original message reaches start_dlq.
  2. Restore storage and consume that retained message while no Request exists.
  3. Before: the message was discarded and the submission stayed Accepted. After: the same request is recorded as Error and an Error log is sent; no validation or normal-work command is published.

Test Plan

  • The service fixture starts with a retained start message and no Request, then checks the Error row/log and absence of validation or merge-check work.
  • Primary replay coverage confirms a delayed original delivery cannot restart the failed request.
  • DLQ tests cover failed writes/publication, lost create replies, input conflicts, concurrent claims, completed-batch ownership, cancellation, and terminal winners.
  • Eight focused core/controller/pipeline test targets passed uncached; real service integration and repository checks also passed locally.

Revert Plan

Revert this change without deleting terminal Request rows or retained DLQ messages. Failed submissions are not automatically resubmitted.

Issue Links

None — repository audit remediation without an external tracker ticket.

Summary:
Intent:
- Resume retained start deliveries whose request creation never completed, without requiring another user submission or inventing a failed outcome.

Changes:
- Reuse start for final recovery and classify the authoritative snapshot after Create or AlreadyExists.
- Preserve immutable inputs and later-stage ownership; replay stored terminal facts through the shared terminal-log publisher.
- Replace the unused start ID-only decoder and cover recovery through the actual pipeline binding and running service.

Test Plan:
- TestRetainedStartCreatesRequestAndHandsOffToValidate seeds a full start_dlq payload before service startup with no request row, then verifies acknowledgment, the original immutable inputs, and the normal validate command.
- TestController_Process_ReplaysFailedHandOff exercises dependency failures and lost acknowledgment on both input topics; TestController_Process_AuthoritativeState proves later/cancelling ownership and stored terminal diagnostics are preserved.

Revert Plan:
Revert this change while retaining final-DLQ records and existing request rows. Already-admitted work continues under its current owner.
@sbalabanov

sbalabanov commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

[independent-integration-review] test/integration/submitqueue/orchestrator/suite_test.go:155-163: this loop can wait until the test runner times out even after start recovery succeeds. In a full-suite run with earlier cancellation integration cases, request 911's Started/Validating logs and merge-conflict-check hand-off existed, but queue GC had already removed its acked start_dlq delivery state and consumed validate message. sql.ErrNoRows therefore never reaches the loop's only exit; the payload assertion at line 172 is transient for the same reason.

Please replace the ephemeral ack-row poll and consumed-message read with durable recovery/hand-off evidence (for example, await the authoritative Request and the downstream merge-conflict-check payload, which this fixture does not consume, then compare its request identity and original change). Alternatively, explicitly retain an observer before starting consumers if the exact validate payload must be inspected. No sleep or test-order dependency is needed. Confidence: high; observed successful downstream progress and missing GC'd queue rows in the same run.

Summary:
Replace transient queue receipt and validate-message assertions with the authoritative request and unconsumed cross-service hand-off. Complete the consumers using durable acknowledgment watermarks, then explicitly remove consumed evidence to exercise garbage-collected queues without timing or test-order dependencies.

Test Plan:
TestRetainedStartCreatesRequestAndHandsOffToValidate passes with consumed start and validate evidence removed, preserving the request ID, queue, ordered changes, and merge strategy in the downstream request. All 19 orchestrator and service integration targets pass.

Revert Plan:
Revert this test-only change; production recovery behavior is unchanged.
@sbalabanov

Copy link
Copy Markdown
Contributor Author

[agent-response SQC-011] Addressing this review. Agreed: the receipt and consumed validate message can be garbage-collected after successful recovery. Fixed in 0d365e1e.

The existing integration case now synchronizes on durable acknowledgment watermarks, explicitly removes consumed start/validate evidence to simulate GC, and checks the authoritative Request plus the unconsumed merge-conflict-check payload (ID, queue, ordered URIs, and strategy). No sleeps, test-order changes, or additional test cases.

./tool/bazel test //submitqueue/orchestrator/... //test/integration/submitqueue/orchestrator:go_default_test --test_output=errors passed all 19 targets, including the real service integration with GC simulation. Format, lint, tidy, Gazelle, and mock checks also passed.

@sbalabanov

Copy link
Copy Markdown
Contributor Author

[independent-review] Should fix — public PR description boundary (95% confidence).

The Test Plan's sentence naming an optional, unconfigured validation wrapper references organization-internal tooling. This repository's AGENTS.md also applies its public-content rule to PR descriptions. Please remove that sentence or describe the wrapper generically; preserve the native Bazel commands and truthful results. This needs only a body edit, not a code change or new tests.

The complete updated code review read all 12 assigned files and reran all four affected targets, including fresh Docker integration. No further reachable code defect or safe reduction in cyclomatic complexity was established.

Generated with the repository's .agents/skills/autoreview workflow; finding rechecked against PR #1040 at 0d365e1e195f8e6fc0e7f5c35550ec9b0e05082d.

@sbalabanov

Copy link
Copy Markdown
Contributor Author

[agent-response SQC-011] Addressing this review. Agreed: removed only the optional-wrapper sentence from the Test Plan. Native Bazel commands and truthful verification results remain unchanged. This is a description-only correction; no code, commits, or tests changed.

Summary:
Keep start recovery tests focused on request state, log status, version, and replay instead of requiring retained terminal diagnostic contents.

Test Plan:
Existing start and pipeline tests pass without a terminal-details fixture or timestamp, error, and metadata fidelity assertions. No runtime code or new tests.

Revert Plan:
Revert this test-only change.
Summary:
Replace start dead-letter forward processing with Error-only reconciliation. A missing request is recorded as a complete terminal snapshot using the original immutable inputs, preventing a delayed primary delivery from restarting it. Existing requests use conditional writes and the established batch-ownership and terminal-winner rules.

Test Plan:
Focused start, dead-letter, and pipeline tests cover Error-only publication, immutable inputs, retries, concurrent creation and claims, terminal winners, and delayed primary delivery. The service fixture checks the failed Request and Error log without validate or merge-check work.

Revert Plan:
Revert this change without deleting terminal Request rows or retained dead letters. Failed submissions are not automatically resubmitted.
Summary:
Keep successful and cancelled batches authoritative while their requests await terminal status projection. Failed batches still permit Error reconciliation, abandoned Creating associations remain unowned, and terminal Request winners retain idempotent status repair.

Test Plan:
The existing Start DLQ table now checks successful/cancelled batch ownership, failed-batch reconciliation, and an existing Error winner. The successful/cancelled owner cases fail against the previous guard. Uncached DLQ, Conclude, Start, and pipeline tests pass; existing Conclude cases cover Landed and Cancelled projection.

Revert Plan:
Revert this ownership guard change; rollback does not rewrite stored data.
Summary:
Integrate the terminal-status repair base without changing the reviewed Error-only start reconciliation or completed-batch ownership guard. The merged base removes diagnostic-retention storage and names terminal status resends explicitly.

Test Plan:
Eight focused request-core, controller, and pipeline targets pass uncached. Gazelle, the generated BUILD-file check, and lint pass. The merge matches the precomputed tree and leaves every reviewed start-reconciliation file unchanged.

Revert Plan:
Revert this merge relative to its first parent without deleting stored Request rows.
@cla-assistant

cla-assistant Bot commented Nov 12, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

1 similar comment
@cla-assistant

cla-assistant Bot commented Nov 12, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@sbalabanov sbalabanov changed the title fix(start): recover retained requests in place fix(start): reconcile dead-lettered submissions to Error Nov 12, 2026

@sbalabanov sbalabanov left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reduce the size of the change. Do not rewrite comments unless absolutely needed. Do not add nil or empty checks that were not there before. Make change focused to the problem.

Summary:
Keep the primary start change to its authoritative snapshot and immutable-input/state fence. Restore existing comments, publishing helper, decoder and tests instead of rewriting them, and remove newly added empty-input and unsupported-state validation.

Remove the duplicate mock pipeline scenario and unrelated documentation updates. Retain the complete missing-Request Error snapshot, conditional Error reconciliation, completed-batch ownership and real service fixture.

Test Plan:
All 19 orchestrator and service integration test targets pass uncached, including the real service Error row/log and no-forward-progress fixture. Eight focused core/controller/pipeline targets also pass uncached. Formatting and Gazelle generation completed.

Revert Plan:
Revert this focused cleanup without deleting terminal Request rows or retained dead letters.
@sbalabanov

Copy link
Copy Markdown
Contributor Author

[agent-response SQC-011] Addressing this review in d1179a40: 15 to 12 files, with about 38% fewer changed lines against the parent. Restored existing comments, helpers and tests; removed the new primary empty-ID/DLQ empty-URI checks, duplicate wiring test and unrelated documentation edits.

Kept the pre-existing DLQ ID requirement and the authoritative snapshot, Error-row and ownership/version guards. Removing those could restart failed submissions or overwrite a winning outcome. All 19 local targets passed, including service integration.

[addressed by agent]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants