Repository navigation
fix(live): give each support_cfc model call a fresh request queue - #8172
Open
AkshayShah03 wants to merge 1 commit into
Open
AkshayShah03 wants to merge 1 commit into
AkshayShah03 wants to merge 1 commit into
Conversation
call_llm_async closes the LiveRequestQueue on turn_complete, then reused that closed queue for the next CFC model call. A tool that finished after turn_complete had its response sent into the closed queue, so the next connection received it again as a live tool response, on top of the copy already in its history. The reused queue also reported closed=True, which run_live treats as a client close and so would not reconnect. Create a new queue when the existing one is closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
2. Or, if no issue exists, describe the change:
Problem:
With
run_async(..., run_config=RunConfig(support_cfc=True)), a tool's response can reach the model twice since e5dd820 (live function calls run in background tasks).It happens when a tool finishes right after the model's
turn_complete:call_llm_asynccloses the invocation'sLiveRequestQueueonturn_complete.run_livemerge loop, which yields it and callslive_request_queue.send_content(...). That puts it in the closed queue, after the close request.run_asyncmakes another model call.live_request_queue is None. The new connection gets the response twice: once in its history (it was persisted as an event), and again whensend_to_modelreads the leftover request from the old queue and sends it as a live tool response.The reused queue also still reports
closed=True.run_livereads that as "the client closed the queue" and won't reconnect with a resumption handle.Repro, with a scripted live model that records what each connection receives. The
NON_BLOCKINGtoollookupreturns as the model sendsturn_complete:Before e5dd820 the same script opened a single connection, because that response never went through the flow.
Solution:
Each CFC model call opens a new live connection, and the history it gets comes from the session. So
call_llm_asyncnow also creates a freshLiveRequestQueuewhen the existing one is closed, not only when there is none. Anything left in the old queue is already in the new connection's history. A queue that is still open (not used byrun_asynctoday) is left as it is.I kept the extra model call after the late response. In
run_asynca function response normally gets a follow-up model call, so the model now answers with the tool's result. If you'd rather a response that arrives afterturn_completedidn't start another call, I'm happy to change it.Testing Plan
Unit Tests:
test_cfc_call_does_not_reuse_the_previous_calls_closed_queue(tests/unittests/flows/llm_flows/core/test_model_call.py) runs two CFC calls. The first sends a function response afterturn_complete. The test checks that the second call gets an open, empty queue. It fails on main (the second call sees the closed queue holding the close request and the stale response) and passes with this change.tests/unittests/flows/llm_flows/core/test_model_call.py: 21 passedtest_runners.py,test_agent_tool.py: 2107 passed, 1 skipped, 4 xfailedtests/unittests: 24775 passed. The 8 failures (test_generate_content_body.py,test_google_llm.py::test_generate_content_async_leaves_continuation_to_sdk,test_release_dependencies.py) fail the same way on main in my environment, so they are unrelated to this change.Manual End-to-End (E2E) Tests:
I ran the script above with
run_async+support_cfc=Trueagainst a scriptedBaseLlmwhoseconnect()records each connection's history and the live requests it receives. No real Live API call was made.Checklist