Skip to content

fix(live): give each support_cfc model call a fresh request queue - #8172

Open
AkshayShah03 wants to merge 1 commit into
google:mainfrom
AkshayShah03:fix/live-cfc-closed-queue-reuse
Open

AkshayShah03 wants to merge 1 commit into
google:mainfrom
AkshayShah03:fix/live-cfc-closed-queue-reuse

Conversation

@AkshayShah03

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

2. Or, if no issue exists, describe the change:

Problem:

With run_async(..., run_config=RunConfig(support_cfc=True)), a tool's response can reach the model twice since e5dd820 (live function calls run in background tasks).

It happens when a tool finishes right after the model's turn_complete:

  1. call_llm_async closes the invocation's LiveRequestQueue on turn_complete.
  2. The tool's response then comes through the run_live merge loop, which yields it and calls live_request_queue.send_content(...). That puts it in the closed queue, after the close request.
  3. The response is now the step's last event, so run_async makes another model call.
  4. That call reuses the same queue, because it only creates one when live_request_queue is None. The new connection gets the response twice: once in its history (it was persisted as an event), and again when send_to_model reads the leftover request from the old queue and sends it as a live tool response.

The reused queue also still reports closed=True. run_live reads that as "the client closed the queue" and won't reconnect with a resumption handle.

Repro, with a scripted live model that records what each connection receives. The NON_BLOCKING tool lookup returns as the model sends turn_complete:

main:
  conn 1: history FR ids=[]      live-sent FR ids=[]
  conn 2: history FR ids=['fc1'] live-sent FR ids=['fc1']   <- fc1 delivered twice
this PR:
  conn 1: history FR ids=[]      live-sent FR ids=[]
  conn 2: history FR ids=['fc1'] live-sent FR ids=[]

Before e5dd820 the same script opened a single connection, because that response never went through the flow.

Solution:

Each CFC model call opens a new live connection, and the history it gets comes from the session. So call_llm_async now also creates a fresh LiveRequestQueue when the existing one is closed, not only when there is none. Anything left in the old queue is already in the new connection's history. A queue that is still open (not used by run_async today) is left as it is.

I kept the extra model call after the late response. In run_async a function response normally gets a follow-up model call, so the model now answers with the tool's result. If you'd rather a response that arrives after turn_complete didn't start another call, I'm happy to change it.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally (apart from 8 that also fail on main, see below).

test_cfc_call_does_not_reuse_the_previous_calls_closed_queue (tests/unittests/flows/llm_flows/core/test_model_call.py) runs two CFC calls. The first sends a function response after turn_complete. The test checks that the second call gets an open, empty queue. It fails on main (the second call sees the closed queue holding the close request and the stale response) and passes with this change.

  • tests/unittests/flows/llm_flows/core/test_model_call.py: 21 passed
  • flows, live, streaming, runners, test_runners.py, test_agent_tool.py: 2107 passed, 1 skipped, 4 xfailed
  • full tests/unittests: 24775 passed. The 8 failures (test_generate_content_body.py, test_google_llm.py::test_generate_content_async_leaves_continuation_to_sdk, test_release_dependencies.py) fail the same way on main in my environment, so they are unrelated to this change.

Manual End-to-End (E2E) Tests:

I ran the script above with run_async + support_cfc=True against a scripted BaseLlm whose connect() records each connection's history and the live requests it receives. No real Live API call was made.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

call_llm_async closes the LiveRequestQueue on turn_complete, then reused
that closed queue for the next CFC model call. A tool that finished after
turn_complete had its response sent into the closed queue, so the next
connection received it again as a live tool response, on top of the copy
already in its history. The reused queue also reported closed=True, which
run_live treats as a client close and so would not reconnect.

Create a new queue when the existing one is closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants