Repository navigation
fix(start): reconcile dead-lettered submissions to Error - #1040
sbalabanov wants to merge 7 commits into
Conversation
Summary: Intent: - Resume retained start deliveries whose request creation never completed, without requiring another user submission or inventing a failed outcome. Changes: - Reuse start for final recovery and classify the authoritative snapshot after Create or AlreadyExists. - Preserve immutable inputs and later-stage ownership; replay stored terminal facts through the shared terminal-log publisher. - Replace the unused start ID-only decoder and cover recovery through the actual pipeline binding and running service. Test Plan: - TestRetainedStartCreatesRequestAndHandsOffToValidate seeds a full start_dlq payload before service startup with no request row, then verifies acknowledgment, the original immutable inputs, and the normal validate command. - TestController_Process_ReplaysFailedHandOff exercises dependency failures and lost acknowledgment on both input topics; TestController_Process_AuthoritativeState proves later/cancelling ownership and stored terminal diagnostics are preserved. Revert Plan: Revert this change while retaining final-DLQ records and existing request rows. Already-admitted work continues under its current owner.
|
[independent-integration-review] Please replace the ephemeral ack-row poll and consumed-message read with durable recovery/hand-off evidence (for example, await the authoritative Request and the downstream merge-conflict-check payload, which this fixture does not consume, then compare its request identity and original change). Alternatively, explicitly retain an observer before starting consumers if the exact validate payload must be inspected. No sleep or test-order dependency is needed. Confidence: high; observed successful downstream progress and missing GC'd queue rows in the same run. |
Summary: Replace transient queue receipt and validate-message assertions with the authoritative request and unconsumed cross-service hand-off. Complete the consumers using durable acknowledgment watermarks, then explicitly remove consumed evidence to exercise garbage-collected queues without timing or test-order dependencies. Test Plan: TestRetainedStartCreatesRequestAndHandsOffToValidate passes with consumed start and validate evidence removed, preserving the request ID, queue, ordered changes, and merge strategy in the downstream request. All 19 orchestrator and service integration targets pass. Revert Plan: Revert this test-only change; production recovery behavior is unchanged.
|
[agent-response SQC-011] Addressing this review. Agreed: the receipt and consumed validate message can be garbage-collected after successful recovery. Fixed in 0d365e1e. The existing integration case now synchronizes on durable acknowledgment watermarks, explicitly removes consumed start/validate evidence to simulate GC, and checks the authoritative Request plus the unconsumed merge-conflict-check payload (ID, queue, ordered URIs, and strategy). No sleeps, test-order changes, or additional test cases.
|
|
[independent-review] Should fix — public PR description boundary (95% confidence). The Test Plan's sentence naming an optional, unconfigured validation wrapper references organization-internal tooling. This repository's The complete updated code review read all 12 assigned files and reran all four affected targets, including fresh Docker integration. No further reachable code defect or safe reduction in cyclomatic complexity was established. Generated with the repository's |
|
[agent-response SQC-011] Addressing this review. Agreed: removed only the optional-wrapper sentence from the Test Plan. Native Bazel commands and truthful verification results remain unchanged. This is a description-only correction; no code, commits, or tests changed. |
Summary: Keep start recovery tests focused on request state, log status, version, and replay instead of requiring retained terminal diagnostic contents. Test Plan: Existing start and pipeline tests pass without a terminal-details fixture or timestamp, error, and metadata fidelity assertions. No runtime code or new tests. Revert Plan: Revert this test-only change.
Summary: Replace start dead-letter forward processing with Error-only reconciliation. A missing request is recorded as a complete terminal snapshot using the original immutable inputs, preventing a delayed primary delivery from restarting it. Existing requests use conditional writes and the established batch-ownership and terminal-winner rules. Test Plan: Focused start, dead-letter, and pipeline tests cover Error-only publication, immutable inputs, retries, concurrent creation and claims, terminal winners, and delayed primary delivery. The service fixture checks the failed Request and Error log without validate or merge-check work. Revert Plan: Revert this change without deleting terminal Request rows or retained dead letters. Failed submissions are not automatically resubmitted.
Summary: Keep successful and cancelled batches authoritative while their requests await terminal status projection. Failed batches still permit Error reconciliation, abandoned Creating associations remain unowned, and terminal Request winners retain idempotent status repair. Test Plan: The existing Start DLQ table now checks successful/cancelled batch ownership, failed-batch reconciliation, and an existing Error winner. The successful/cancelled owner cases fail against the previous guard. Uncached DLQ, Conclude, Start, and pipeline tests pass; existing Conclude cases cover Landed and Cancelled projection. Revert Plan: Revert this ownership guard change; rollback does not rewrite stored data.
Summary: Integrate the terminal-status repair base without changing the reviewed Error-only start reconciliation or completed-batch ownership guard. The merged base removes diagnostic-retention storage and names terminal status resends explicitly. Test Plan: Eight focused request-core, controller, and pipeline targets pass uncached. Gazelle, the generated BUILD-file check, and lint pass. The merge matches the precomputed tree and leaves every reviewed start-reconciliation file unchanged. Revert Plan: Revert this merge relative to its first parent without deleting stored Request rows.
|
|
1 similar comment
|
|
sbalabanov
left a comment
There was a problem hiding this comment.
reduce the size of the change. Do not rewrite comments unless absolutely needed. Do not add nil or empty checks that were not there before. Make change focused to the problem.
Summary: Keep the primary start change to its authoritative snapshot and immutable-input/state fence. Restore existing comments, publishing helper, decoder and tests instead of rewriting them, and remove newly added empty-input and unsupported-state validation. Remove the duplicate mock pipeline scenario and unrelated documentation updates. Retain the complete missing-Request Error snapshot, conditional Error reconciliation, completed-batch ownership and real service fixture. Test Plan: All 19 orchestrator and service integration test targets pass uncached, including the real service Error row/log and no-forward-progress fixture. Eight focused core/controller/pipeline targets also pass uncached. Formatting and Gazelle generation completed. Revert Plan: Revert this focused cleanup without deleting terminal Request rows or retained dead letters.
|
[agent-response SQC-011] Addressing this review in d1179a40: 15 to 12 files, with about 38% fewer changed lines against the parent. Restored existing comments, helpers and tests; removed the new primary empty-ID/DLQ empty-URI checks, duplicate wiring test and unrelated documentation edits. Kept the pre-existing DLQ ID requirement and the authoritative snapshot, Error-row and ownership/version guards. Removing those could restart failed submissions or overwrite a winning outcome. All 19 local targets passed, including service integration. [addressed by agent] |
TL;DR: Report failed starts as Error instead of leaving submissions stuck at Accepted.
Summary
A failed start message used to be discarded when no Request had been saved, leaving the submission at Accepted. The DLQ now records a complete failed Request and sends an Error status without restarting validation or normal work. Existing batch owners and terminal winners are preserved.
Builds on #1039.
Repro
start_dlq.Test Plan
Revert Plan
Revert this change without deleting terminal Request rows or retained DLQ messages. Failed submissions are not automatically resubmitted.
Issue Links
None — repository audit remediation without an external tracker ticket.