Repository navigation
DefaultAsyncClientConnectionOperator: close the session when cancelled during the TLS handshake - #950
Open
shinseongsu wants to merge 1 commit into
Open
shinseongsu wants to merge 1 commit into
shinseongsu wants to merge 1 commit into
Conversation
…d during the TLS handshake Once the socket is connected, the returned future still pointed at the session request, which is already done. Cancelling it during the TLS handshake did nothing: the session stayed open until the handshake timed out, and if the handshake then succeeded the connection was left open outside the pool. Register a dependency that closes the connection before starting the handshake, and register the session request through a holder so it cannot replace that dependency when the connect completes early.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #943.
Once the socket is connected, the future returned by
connect()still has the session request as its dependency, and that request is already done. So cancelling during the TLS handshake doesn't touch the session: it stays open until the handshake times out. If the handshake finishes after the cancel,future.completed()is ignored and the connection is left open with nothing holding it. It isn't in the pool and doesn't show up in the pool stats.The change registers a dependency that closes the connection right before the handshake starts. The session request goes through a
ComplexCancellableholder, like in #943, so a connect that completes beforesessionRequester.connect()returns can't overwrite it.How I tested it
Pooling async client, max 20 per route, 20 s connect timeout, against a local TLS server that accepts the TCP connection and then holds the handshake. 20 requests, each cancelled after 300 ms. Client sockets counted with
lsof3 s after the last cancel (the count includes one probe request).On master the server sees the cancelled connections close only when the client is shut down. With the change each one is reset about 0.3 s after it was accepted.
I also ran it under load, 150 s per build: 10 threads, every request cancelled if it takes more than 300 ms, one request per connection, and the server delays the handshake by 2 s on about 10% of new connections. Prometheus scrapes every 2 s.
On master the client's open fds and the server's open connections grow by about 11 per second and end at ~1.7K, while the pool never sees more than 10 connections. With the change both stay flat (66 fds, 5 server connections). Throughput is the same, so the leaked sockets are pure overhead here, but in a long-running client they'd run into the fd limit eventually.
Single request, server holds the handshake for 1 s, cancel after 300 ms:
DefaultAsyncClientConnectionOperatorTest#testCancelDuringTlsHandshakeClosesSessionfails on master and passes with the change. httpclient5verify: 1161 tests, 0 failures. httpclient5-testing async tests: 407, 0 failures.The connection is closed with
CloseMode.IMMEDIATE, so the server gets an RST rather than a close_notify. That seemed right for a handshake that never finished, but let me know if you'd rather have GRACEFUL here.