Skip to content

Fix macOS-only test fixture failures - #519

Draft
prk-Jr wants to merge 5 commits into
feat/kv-record-contractfrom
fix/503-macos-nonblocking-accept
Draft

prk-Jr wants to merge 5 commits into
feat/kv-record-contractfrom
fix/503-macos-nonblocking-accept

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • On macOS, accept(2) copies O_NONBLOCK from a listener to the accepted socket; Linux does not. Fixtures that poll a non-blocking std TcpListener and then read the accepted stream with set_read_timeout got WouldBlock immediately on macOS, so store-aws, startup and generated-proof tests failed at random under parallel load. Linux CI never shows this. Each affected crate now gets one small accept_blocking helper that restores blocking mode, and every affected accept site uses it.
  • The four production_hosting tests that fail on macOS have a different cause. tests/fixtures/native_proxy.rs binds its upstream socket to 127.0.0.2/127.0.0.3. Linux routes all of 127.0.0.0/8 to lo, but macOS lo0 has only 127.0.0.1, so the bind fails with EADDRNOTAVAIL (errno 49). The relay discards that error and drops the connection, and the test sees an empty response or a TLS UnexpectedEof. The tests need distinct source IPs by design, so they are now ignored off Linux, with a reason that gives the lo0 alias commands needed to run them.
  • Two more macOS flakes are fixed so the issue's "all pass on macOS" holds on every run:
    • Secrets Manager attempt-timeout test. The SDK's 100 ms attempt deadline runs from the start of the attempt. Under parallel load the child process can be descheduled between the TCP connect and the request write, so the SDK correctly drops the connection having sent nothing. A temporary probe caught a connection closed with 0 bytes, 200 ms after accept. The old fixture treated that as fatal ("SDK closed before request headers"). The new fixture accepts every attempt and never answers. With a 10 s operation budget, only the attempt deadline can close a held connection within 5 s. The test asserts that this happened for every attempt, that there were 1–3 attempts, and that any full request named the fixture ARN. Deleting operation_attempt_timeout from the provider makes it fail.

    • production_hosting port reservations. macOS marks a new socket close-on-exec only after socket() returns, unlike Linux's atomic SOCK_CLOEXEC. A fixture process spawned by a parallel test in that gap inherited another test's port reservation and kept it after the test released it. lsof at a failure showed another fixture process holding the port. Both failure modes come from this one leak:

      • A fixture server failed to bind its own port (AddrInUse).
      • A "nothing answers" check got a response. The original reservation was a listening std socket, so a leaked copy in another child accepted the connection.

      Port reservations, client streams and fixture spawns now hold one lock, so none of them can be inherited. Tests that assert nothing answers keep their reservation bound, but not listening, until the check is done. They also assert that the server never reached its listener bind, because a held port refuses connections whatever the server does.

  • Only test code changes. Production behavior is unchanged.

Stacked on #510 (feat/kv-record-contract). That is the lowest branch that has all the affected fixtures (dynamodb_records_tests.rs first appears there). Above #510, the only change to these files is unrelated new tests added to production_hosting.rs by #454, #517 and rc/identity. git merge-tree shows this PR merges cleanly into each of them.

Changes

Crate / File Change
edgezero-store-aws/src/test_net.rs (new) accept_blocking helper plus a regression test. Module declared #[cfg(test)] in lib.rs, gated on secrets-manager or dynamodb-kv.
edgezero-store-aws/src/{secrets_manager,dynamodb_tests,dynamodb_records_tests}.rs Fixture accept loops use accept_blocking. dynamodb_records_tests.rs (added in #510) has the same pattern but was not in the issue's table.
edgezero-adapter-axum/tests/fixtures/blocking_accept.rs (new) Same helper and test, included with #[path]
edgezero-adapter-axum/tests/{native_aws_startup,native_celld_startup}.rs Include the helper; fixture accept loops use it
edgezero-cli/tests/support/blocking_accept.rs (new) Same helper and test, included with #[path] (cfg(unix), like its callers)
edgezero-cli/tests/{native_store,dynamodb,celld}_generated_proof.rs Include the helper; all four accept calls use it
edgezero-adapter-axum/tests/production_hosting.rs #[cfg_attr(not(target_os = "linux"), ignore = "…")] on the four source-bound proxy tests. One lock covers port reservations, client streams (connect()) and fixture spawns (Server::spawn, kill). New reserved_address() holds the port, bound but not listening, through every "nothing answers" check (all failed() callers plus three direct checks); those checks also assert the log has no startup bind.
edgezero-store-aws/src/secrets_manager.rs Attempt-timeout test uses a fixture that never answers (holding_fixture, which drains every queued connection before counting) instead of the 350 ms Delay response. The timeout child's operation budget goes from 500 ms to 10 s so it cannot end an attempt; the 100 ms attempt timeout is unchanged.

Sites checked and left unchanged, because they don't read an accepted std stream:

  • Sentinels (assert that accept() returns WouldBlock): edgezero-store-celld/src/client.rs, edgezero-cli/src/config.rs, and the test-harness main.rs/net.rs.
  • Handed to tokio via from_std: fixtures/native_proxy.rs and src/dev_server.rs.
  • Already restore blocking themselves: the test-harness fixtures in net.rs, overflow.rs and readiness.rs. net/wire/accept.rs hands accepted sockets to mio, which expects them non-blocking on every platform.

edgezero-store-celld/src/client.rs is in the issue's table, but at this base it holds only a sentinel. Its HTTP fixture is tokio-based.

Closes

Closes #503

This PR targets feat/kv-record-contract, not main, so GitHub will not close #503 automatically. Close it by hand once this lands.

Test plan

All runs were on macOS (Apple Silicon) with the pinned Rust 1.98.1 and the default test thread count.

  • Before the fix (base f2cacc67):

    • edgezero-store-aws --all-features --lib failed 5 of 5 runs, with 7–39 of 99 tests failing (read request headers: Os { code: 35, kind: WouldBlock }).
    • native_aws_startup + native_celld_startup failed 5 of 5.
    • The DynamoDB and celld generated proofs each failed 1 of 3 (WouldBlock in the fixture).
    • production_hosting failed the same 4 tests in 3 of 3 runs.
    • With a temporary print in relay, the swallowed error was AddrNotAvailable for source 127.0.0.2.
  • Regression test: with the helper reduced to a plain accept(), the new test fails in every crate ("read returned after ~5µs, before its 200 ms timeout"). With the helper it passes.

  • After the fix, 5 runs each, all passed (at 6ac23fb7; store-aws and production_hosting were rerun at the final head below):

    • cargo test -p edgezero-store-aws --all-features --lib (100 passed)
    • cargo test -p edgezero-store-celld --all-features --lib
    • cargo test -p edgezero-adapter-axum --features aws-stores,aws-dynamodb,celld-kv,test-utils --test native_aws_startup --test native_celld_startup
    • cargo test -p edgezero-adapter-axum --no-default-features --features axum,test-utils --test production_hosting (33 passed, 4 ignored)
    • Generated native-store, DynamoDB and celld proofs (--ignored, as in test.yml)
  • Lint and format:

    • cargo fmt --all -- --check
    • cargo clippy -D warnings on the touched crates with --all-targets --all-features, plus store-aws with secrets-manager, dynamodb-kv and appconfig-agent each on its own, and the cli cli,edgezero-adapter-axum test set.
    • Full workspace gates and Linux: left to hosted CI.
  • Flaky tests fixed (full-suite runs, default thread count):

    Target Before After (final head d86a5d32)
    cargo test -p edgezero-store-aws --all-features --lib 1 of 20 failed (attempt-timeout test) 20 of 20 passed
    cargo test -p edgezero-adapter-axum --no-default-features --features axum,test-utils --test production_hosting 2 of 10 failed earlier; 6 of 20 on a larger sample (AddrInUse, or a "nothing answers" check got a response) 10 of 10 passed (also 40 of 40 at the previous head 09b09d9e, which lacked only the review hardening)
  • Attempt-timeout negative control: with operation_attempt_timeout removed from both client configs in secrets_manager.rs, the new test fails with "the attempt deadline, not the 10 s operation budget, ends every unanswered attempt". The old test also failed on this mutation; its only problem was the panic on a connection closed before the request.

Known limitations:

  • The regression test can only catch the bug on macOS. On Linux it passes with or without the fix.
  • It guards the helper, not a future fixture that calls plain accept().
  • Its copy runs once per test binary that includes it, which adds about 1 s in total.

Checklist

  • Changes follow CLAUDE.md conventions
  • No Tokio deps added to core or adapter crates
  • Route params use {id} syntax (not :id) (not applicable)
  • Types imported from edgezero_core (not http crate) (not applicable)
  • Store wiring goes through KvRegistry / ConfigRegistry / SecretRegistry (not applicable)
  • New code has tests
  • No secrets or credentials committed

Fixtures that poll a non-blocking std TcpListener read the accepted
stream with set_read_timeout. macOS accept(2) copies O_NONBLOCK to the
accepted socket (Linux does not), so those reads failed with WouldBlock
at once and tests failed at random on macOS under parallel load.

Add a small accept_blocking helper per crate (store-aws, adapter-axum
tests, cli tests) that restores blocking mode, route all nine accept
sites through it, and cover it with a test that fails on macOS when the
accepted stream does not wait for its read timeout.
Four production_hosting tests drive the native proxy fixture, which binds
its upstream socket to 127.0.0.2 or 127.0.0.3 so the server sees distinct
peers. Linux routes all of 127.0.0.0/8 to lo; macOS lo0 has only
127.0.0.1, so the bind fails with EADDRNOTAVAIL, the relay drops the
connection, and the tests see an empty response or TLS UnexpectedEof.

Ignore them off Linux with a reason that gives the lo0 alias commands
needed to run them with --ignored.
@prk-Jr prk-Jr self-assigned this Nov 20, 2026
@prk-Jr prk-Jr changed the title Restore blocking mode on macOS fixture streams Fix macOS-only test fixture failures Nov 20, 2026
The SDK's 100 ms attempt deadline runs from the start of the attempt.
Under parallel load the child can be descheduled between the TCP connect
and the request write, so the SDK correctly abandons the connection
before sending anything. The fixture treated that as fatal ("SDK closed
before request headers"), which failed the test about once in 20 full
store-aws runs on macOS.

The test now uses a fixture that accepts every attempt and never
answers, recording whether the client closed each connection itself. The
operation budget is 10 s, so only the attempt deadline can close a held
connection within the 5 s bound. The test asserts that every attempt was
closed that way, that the SDK made 1 to 3 attempts, and that any full
request named the fixture ARN. Removing the attempt timeout from the
provider makes the test fail.
macOS marks a new socket close-on-exec only after socket() returns. When
a parallel test spawned a fixture process in that gap, the child
inherited another test's port reservation and kept it bound after the
test released it. That test's fixture server then failed to bind
(AddrInUse); lsof showed another fixture process holding the port as a
CLOSED socket. Separately, a released reservation could be served by
another test's server during a "nothing answers" check. Together these
failed 6 of 20 macOS runs.

Reservations and fixture spawns now hold one lock, so they never
overlap. Tests that assert nothing answers keep their reservation bound,
but not listening, until the check is done.
production_hosting: client streams from Server::socket and raw_request
also take the reservation/spawn lock, so a concurrently spawned fixture
cannot inherit one and keep the connection open after the test drops it.
Because a held reservation always refuses connections, "nothing answers"
checks now also assert that the server never reached its listener bind.

Secrets Manager: the holding fixture drains every queued connection
after the child exits before counting attempts, so a starved fixture
thread cannot undercount them.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Test fixtures that accept from a non-blocking std listener fail on macOS

1 participant