Repository navigation
Conversation
Fixtures that poll a non-blocking std TcpListener read the accepted stream with set_read_timeout. macOS accept(2) copies O_NONBLOCK to the accepted socket (Linux does not), so those reads failed with WouldBlock at once and tests failed at random on macOS under parallel load. Add a small accept_blocking helper per crate (store-aws, adapter-axum tests, cli tests) that restores blocking mode, route all nine accept sites through it, and cover it with a test that fails on macOS when the accepted stream does not wait for its read timeout.
Four production_hosting tests drive the native proxy fixture, which binds its upstream socket to 127.0.0.2 or 127.0.0.3 so the server sees distinct peers. Linux routes all of 127.0.0.0/8 to lo; macOS lo0 has only 127.0.0.1, so the bind fails with EADDRNOTAVAIL, the relay drops the connection, and the tests see an empty response or TLS UnexpectedEof. Ignore them off Linux with a reason that gives the lo0 alias commands needed to run them with --ignored.
The SDK's 100 ms attempt deadline runs from the start of the attempt.
Under parallel load the child can be descheduled between the TCP connect
and the request write, so the SDK correctly abandons the connection
before sending anything. The fixture treated that as fatal ("SDK closed
before request headers"), which failed the test about once in 20 full
store-aws runs on macOS.
The test now uses a fixture that accepts every attempt and never
answers, recording whether the client closed each connection itself. The
operation budget is 10 s, so only the attempt deadline can close a held
connection within the 5 s bound. The test asserts that every attempt was
closed that way, that the SDK made 1 to 3 attempts, and that any full
request named the fixture ARN. Removing the attempt timeout from the
provider makes the test fail.
macOS marks a new socket close-on-exec only after socket() returns. When a parallel test spawned a fixture process in that gap, the child inherited another test's port reservation and kept it bound after the test released it. That test's fixture server then failed to bind (AddrInUse); lsof showed another fixture process holding the port as a CLOSED socket. Separately, a released reservation could be served by another test's server during a "nothing answers" check. Together these failed 6 of 20 macOS runs. Reservations and fixture spawns now hold one lock, so they never overlap. Tests that assert nothing answers keep their reservation bound, but not listening, until the check is done.
production_hosting: client streams from Server::socket and raw_request also take the reservation/spawn lock, so a concurrently spawned fixture cannot inherit one and keep the connection open after the test drops it. Because a held reservation always refuses connections, "nothing answers" checks now also assert that the server never reached its listener bind. Secrets Manager: the holding fixture drains every queued connection after the child exits before counting attempts, so a starved fixture thread cannot undercount them.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
accept(2)copiesO_NONBLOCKfrom a listener to the accepted socket; Linux does not. Fixtures that poll a non-blocking stdTcpListenerand then read the accepted stream withset_read_timeoutgotWouldBlockimmediately on macOS, so store-aws, startup and generated-proof tests failed at random under parallel load. Linux CI never shows this. Each affected crate now gets one smallaccept_blockinghelper that restores blocking mode, and every affected accept site uses it.production_hostingtests that fail on macOS have a different cause.tests/fixtures/native_proxy.rsbinds its upstream socket to127.0.0.2/127.0.0.3. Linux routes all of127.0.0.0/8tolo, but macOSlo0has only127.0.0.1, so the bind fails withEADDRNOTAVAIL(errno 49). The relay discards that error and drops the connection, and the test sees an empty response or a TLSUnexpectedEof. The tests need distinct source IPs by design, so they are now ignored off Linux, with a reason that gives thelo0alias commands needed to run them.Secrets Manager attempt-timeout test. The SDK's 100 ms attempt deadline runs from the start of the attempt. Under parallel load the child process can be descheduled between the TCP connect and the request write, so the SDK correctly drops the connection having sent nothing. A temporary probe caught a connection closed with 0 bytes, 200 ms after accept. The old fixture treated that as fatal ("SDK closed before request headers"). The new fixture accepts every attempt and never answers. With a 10 s operation budget, only the attempt deadline can close a held connection within 5 s. The test asserts that this happened for every attempt, that there were 1–3 attempts, and that any full request named the fixture ARN. Deleting
operation_attempt_timeoutfrom the provider makes it fail.production_hosting port reservations. macOS marks a new socket close-on-exec only after
socket()returns, unlike Linux's atomicSOCK_CLOEXEC. A fixture process spawned by a parallel test in that gap inherited another test's port reservation and kept it after the test released it.lsofat a failure showed another fixture process holding the port. Both failure modes come from this one leak:AddrInUse).Port reservations, client streams and fixture spawns now hold one lock, so none of them can be inherited. Tests that assert nothing answers keep their reservation bound, but not listening, until the check is done. They also assert that the server never reached its listener bind, because a held port refuses connections whatever the server does.
Stacked on #510 (
feat/kv-record-contract). That is the lowest branch that has all the affected fixtures (dynamodb_records_tests.rsfirst appears there). Above #510, the only change to these files is unrelated new tests added toproduction_hosting.rsby #454, #517 andrc/identity.git merge-treeshows this PR merges cleanly into each of them.Changes
edgezero-store-aws/src/test_net.rs(new)accept_blockinghelper plus a regression test. Module declared#[cfg(test)]inlib.rs, gated onsecrets-managerordynamodb-kv.edgezero-store-aws/src/{secrets_manager,dynamodb_tests,dynamodb_records_tests}.rsaccept_blocking.dynamodb_records_tests.rs(added in #510) has the same pattern but was not in the issue's table.edgezero-adapter-axum/tests/fixtures/blocking_accept.rs(new)#[path]edgezero-adapter-axum/tests/{native_aws_startup,native_celld_startup}.rsedgezero-cli/tests/support/blocking_accept.rs(new)#[path](cfg(unix), like its callers)edgezero-cli/tests/{native_store,dynamodb,celld}_generated_proof.rsedgezero-adapter-axum/tests/production_hosting.rs#[cfg_attr(not(target_os = "linux"), ignore = "…")]on the four source-bound proxy tests. One lock covers port reservations, client streams (connect()) and fixture spawns (Server::spawn,kill). Newreserved_address()holds the port, bound but not listening, through every "nothing answers" check (allfailed()callers plus three direct checks); those checks also assert the log has nostartup bind.edgezero-store-aws/src/secrets_manager.rsholding_fixture, which drains every queued connection before counting) instead of the 350 msDelayresponse. The timeout child's operation budget goes from 500 ms to 10 s so it cannot end an attempt; the 100 ms attempt timeout is unchanged.Sites checked and left unchanged, because they don't read an accepted std stream:
accept()returnsWouldBlock):edgezero-store-celld/src/client.rs,edgezero-cli/src/config.rs, and the test-harnessmain.rs/net.rs.from_std:fixtures/native_proxy.rsandsrc/dev_server.rs.net.rs,overflow.rsandreadiness.rs.net/wire/accept.rshands accepted sockets to mio, which expects them non-blocking on every platform.edgezero-store-celld/src/client.rsis in the issue's table, but at this base it holds only a sentinel. Its HTTP fixture is tokio-based.Closes
Closes #503
This PR targets
feat/kv-record-contract, notmain, so GitHub will not close #503 automatically. Close it by hand once this lands.Test plan
All runs were on macOS (Apple Silicon) with the pinned Rust 1.98.1 and the default test thread count.
Before the fix (base
f2cacc67):edgezero-store-aws --all-features --libfailed 5 of 5 runs, with 7–39 of 99 tests failing (read request headers: Os { code: 35, kind: WouldBlock }).native_aws_startup+native_celld_startupfailed 5 of 5.WouldBlockin the fixture).production_hostingfailed the same 4 tests in 3 of 3 runs.relay, the swallowed error wasAddrNotAvailablefor source127.0.0.2.Regression test: with the helper reduced to a plain
accept(), the new test fails in every crate ("read returned after ~5µs, before its 200 ms timeout"). With the helper it passes.After the fix, 5 runs each, all passed (at
6ac23fb7; store-aws and production_hosting were rerun at the final head below):cargo test -p edgezero-store-aws --all-features --lib(100 passed)cargo test -p edgezero-store-celld --all-features --libcargo test -p edgezero-adapter-axum --features aws-stores,aws-dynamodb,celld-kv,test-utils --test native_aws_startup --test native_celld_startupcargo test -p edgezero-adapter-axum --no-default-features --features axum,test-utils --test production_hosting(33 passed, 4 ignored)--ignored, as intest.yml)Lint and format:
cargo fmt --all -- --checkcargo clippy -D warningson the touched crates with--all-targets --all-features, plus store-aws withsecrets-manager,dynamodb-kvandappconfig-agenteach on its own, and the clicli,edgezero-adapter-axumtest set.Flaky tests fixed (full-suite runs, default thread count):
d86a5d32)cargo test -p edgezero-store-aws --all-features --libcargo test -p edgezero-adapter-axum --no-default-features --features axum,test-utils --test production_hosting09b09d9e, which lacked only the review hardening)Attempt-timeout negative control: with
operation_attempt_timeoutremoved from both client configs insecrets_manager.rs, the new test fails with "the attempt deadline, not the 10 s operation budget, ends every unanswered attempt". The old test also failed on this mutation; its only problem was the panic on a connection closed before the request.Known limitations:
accept().Checklist
{id}syntax (not:id) (not applicable)edgezero_core(nothttpcrate) (not applicable)KvRegistry/ConfigRegistry/SecretRegistry(not applicable)