You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The RDMA transport rejects fb_rpc. FlatBuffers also requires contiguous storage, which can exceed the RDMA memory pool's 2 MiB block limit.
What is changed and the side effects?
Changed:
Allow fb_rpc over RDMA when FlatBuffers support is enabled.
Register larger contiguous buffers separately for message construction and receive-side reassembly, with cleanup on failure and final reference release.
Side effects:
Performance effects:
Breaking backward compatibility:
Check List:
Please make sure your changes are compilable.
When providing us with a new feature, it is best to add related tests.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The change still has two active moderate issues that make the standard FlatBuffers test suite exit or fail on runners without RDMA hardware instead of skipping hardware-dependent coverage.
This pull request enables FlatBuffers RPC (fb_rpc) over RDMA and supports messages larger than the RDMA pool's 2 MiB limit through separately registered contiguous storage.
Changes:
Adds fb_rpc to RDMA-supported protocols when FlatBuffers is enabled.
Registers, retains, and cleans up oversized FlatBuffers builder and receive buffers.
Adds protocol, fallback, TCP, and RDMA integration tests.
File
Description
test/brpc_rdma_unittest.cpp
Extends RDMA protocol-admission coverage to verify support for fb_rpc.
test/brpc_flatbuffers_protocol_unittest.cpp
Adds FlatBuffers TCP fallback, RDMA round-trip, large-payload, and transport tests.
src/brpc/rdma/rdma_helper.cpp
Allows fb_rpc as an RDMA-supported protocol when FlatBuffers is enabled.
src/brpc/flatbuffers/message.cpp
Adds allocation and registration of oversized contiguous FlatBuffers storage with safe cleanup and reference lifetime management.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Real RDMA hardware and runtime validation are required to verify negotiation, SGE lkeys, fallback behavior, and asynchronous registered-memory cleanup, which static review cannot establish.
The reason will be displayed to describe this comment to others. Learn more.
[replied by brpc-oncall robot] The test restructuring looks good, but the pool-vs-registered-memory fallback in AllocateStorage can abort the process in recoverable situations, and the new RDMA+FlatBuffers code branches are still never compiled by CI.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
A critical teardown-order bug can crash at exit when a namespace-scope Message deregisters RDMA storage after GlobalRelease() has deleted the required mutex.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Avoid locking deleted RDMA mutex during global teardown
src/brpc/flatbuffers/message.cpp:59
A namespace-scope Message constructed before RDMA initialization can crash at exit if it later owns this separately registered storage. GlobalRelease() runs first and deletes the mutex that DeregisterMemoryForRdma() unconditionally locks. Guard deregistration with rdma::IsRdmaAvailable() so the deleter only frees the buffer after global teardown has already deregistered it.
[replied by brpc-oncall robot] Agree with the teardown-order issue above; it should be fixed before merge, so I'm withdrawing my earlier LGTM.
Guarding the deregistration with rdma::IsRdmaAvailable() is the right fix. GlobalRelease() stores g_rdma_available = false as its first step and then deregisters every MR in g_user_mrs itself before deleting the map and the mutex, and GlobalDisableRdma() is LOG(FATAL), so teardown is the only transition to "unavailable" — skipping DeregisterMemoryForRdma() there cannot leak a registration.
Please check the flag inside the deleter (at release time, as in free_storage) rather than at allocation time: storage is normally released long before teardown, and PoolExhaustionFallsBackToRegisteredStorage already relies on the real deregistration path to observe GetLKey(data) == 0 after the last reference is dropped, so that path must stay intact.
[replied by brpc-oncall robot] Agree with the teardown-order issue above; it should be fixed before merge, so I'm withdrawing my earlier LGTM.
Guarding the deregistration with rdma::IsRdmaAvailable() is the right fix. GlobalRelease() stores g_rdma_available = false as its first step and then deregisters every MR in g_user_mrs itself before deleting the map and the mutex, and GlobalDisableRdma() is LOG(FATAL), so teardown is the only transition to "unavailable" — skipping DeregisterMemoryForRdma() there cannot leak a registration.
Please check the flag inside the deleter (at release time, as in free_storage) rather than at allocation time: storage is normally released long before teardown, and PoolExhaustionFallsBackToRegisteredStorage already relies on the real deregistration path to observe GetLKey(data) == 0 after the last reference is dropped, so that path must stay intact.
@wwbmmm Fixed in aff9c68. In the RDMA helper, A process-lifetime mutex now synchronizes MR registration and deregistration with global cleanup. A Message destroyed after cleanup can safely free its buffer. If RDMA is disabled while the registry is still live, the last reference still deregisters the buffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
status/ready-to-mergeRequired approvals and checks are satisfied, with no blocking feedback remaining
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: resolve
Problem Summary:
The RDMA transport rejects
fb_rpc. FlatBuffers also requires contiguous storage, which can exceed the RDMA memory pool's 2 MiB block limit.What is changed and the side effects?
Changed:
fb_rpcover RDMA when FlatBuffers support is enabled.Side effects:
Performance effects:
Breaking backward compatibility:
Check List: