Ten Nuts, One Mouth: Multiple Tool Calls in a Single Turn十颗松果，一张嘴：一次回复里的多个工具调用与并行执行

Some background. JollySammy is a greedy little squirrel. On a good autumn morning he can see ten pinecones scattered on the forest floor, and every fibre of his gluttonous heart wants to crack all ten at once. There is only one problem: JollySammy has exactly one mouth. So the daydream of ten simultaneous nuts collides with a hard fact — some nuts he can stuff into his cheeks and gnaw side by side without any trouble, but some are the messy, sticky kind that he has to crack open and eat alone, or he ends up with sap all over the other nuts. And here’s the part that keeps him up at night: when he does nibble several at once, the empty shells drop out of his cheeks in whatever order they happen to finish — so he had better remember which shell belonged to which nut, or he’ll swear blind he already ate one he hasn’t touched.

That, more or less, is exactly the problem a harness faces when a model returns many tool calls in one turn. So let’s follow JollySammy through it, using the harness he’s rather proud of as the running example.

First, the thing to be handled. A single assistant message can carry text and several tool calls at the same time:

{
  "role": "assistant",
  "content": [
    { "type": "text", "text": "Let me read the code and run the tests." },
    { "type": "tool_use", "id": "tool_1", "name": "Read",
      "input": { "file_path": "fib.py" } },
    { "type": "tool_use", "id": "tool_2", "name": "Grep",
      "input": { "pattern": "TODO", "path": "." } },
    { "type": "tool_use", "id": "tool_3", "name": "Bash",
      "input": { "command": "pytest -q" } }
  ]
}

Three pinecones on the ground. Once the harness has this in hand, it has to answer three questions:

which of these can run together, and which must run alone?
after they finish, how does each result get matched back to its own call?
if one fails — or the user interrupts mid-flight — what about the rest?

Let’s go one layer at a time.

1. The protocol already allows many tool calls in one message

This isn’t a feature the harness invented — it’s native to the API protocol. An assistant message’s content is just a list of content blocks, and that list can freely mix text, thinking, and any number of tool_use blocks.

JollySammy’s streaming state machine appends each arriving tool block, in the order it shows up in the stream, onto two lists: the finalized accumulated_content and the running tool_use_blocks. So the block order inside the assistant message is exactly the model’s output order.

That fact — “block order equals model output order” — is the foundation for every ordering guarantee that follows.

2. Who can run in parallel? Ask the tool itself, with the actual input in hand

The tempting design is a static table hard-wired somewhere: a fixed directive of which tools may parallelize. JollySammy doesn’t love that approach. The harness he likes instead hands the decision to each tool, and judges it against the concrete input of this particular call.

The Tool base class follows three decision rules:

Note that the check receives the parsed input — the orchestrator first validates the raw argument JSON into a structured object (via pydantic) before asking. If the arguments won’t even parse, it plays it safe, treats the call as unsafe, and serializes it.

Why is the default “concurrency-safe equals read-only” rather than the more cautious “everything serial by default”? Because if serial were the default, every read-only tool would have to separately declare “I can parallelize,” and forgetting even one silently throws away parallelism. Hooking it to read-only means Read / Grep and their kin batch up with zero extra code, while anything with side effects automatically falls back to serial.

On erring toward caution here, JollySammy is dead serious.

3. Tool scheduling: a dynamic admission check

The real online path is a streaming executor. Tool blocks are submitted to it while the API is still dribbling out the stream, and whether a block can start running right now is decided by one small admission check, _can_execute — the routine that looks at what’s already running and says yes or no:

def _can_execute(self, is_concurrent_safe: bool) -> bool:
    executing = [t for t in self._tools if t.status == "executing"]
    if not executing:
        return True
    if is_concurrent_safe:
        return all(t.is_concurrent_safe for t in executing)
    return False

Two rules, that’s all. A concurrency-safe tool may join in as long as everyone currently running is also concurrency-safe. An unsafe tool must wait for everyone to finish and then run alone. Picture a shared workbench: those who only leaf through documents can crowd in together, but the one who needs to throw the power switch has to wait for the room to empty.

Walk the opening example through it, with the state at each step:

Read (tool_1) arrives:  executing = []            -> start now
Grep (tool_2) arrives:  executing = [Read]         -> all safe -> start now
Bash (tool_3) arrives:  executing = [Read, Grep]   -> Bash unsafe -> queue
Read done, Grep done:   executing = []             -> Bash starts alone

This check is re-run every time a tool finishes, by the routine that walks the queue. And when it scans the queue and hits the first unsafe tool that still can’t start, it stops there — so that serial point becomes a natural barrier: nothing behind it jumps ahead of it.

Inside a parallel batch there’s one more governor: a semaphore caps how many tools run at once, its capacity set by the MAX_TOOL_CONCURRENCY environment variable, default 10.

4. “No chaos,” layer one: results pair by id, not by position

Suppose three tools run in parallel and finish in the order Grep, Bash, Read — now the result list is in a different order from the call list. Does it get crossed?

No — because pairing never looks at position, only at id.

Each tool result is its own message, and it carries the tool_use_id of the call it answers. When sent to an OpenAI-style endpoint it becomes a role="tool" message carrying a tool_call_id; when sent to Anthropic it’s a tool_result block tucked into a user message. Both protocols match by id.

What does getting it wrong look like? If you paired by position, then the moment the finishing order changes, Grep’s output gets handed back to the model as if it were Read’s:

WRONG (pair by position, completion order differs):
  tool_1 (Read)  <- paired with Grep's output   "3 matches found"
  tool_2 (Grep)  <- paired with Read's output    "def fib(n): ..."

The model then reasons over a mixed-up result and the whole session drifts from there. Pair by id instead, and the order within the result list is merely display order — correctness is carried entirely by the id.

Precisely because of this, JollySammy can happily adopt a “whoever finishes first is served first” policy: inside a parallel batch he waits on the first completion, and the result collector explicitly allows a tool that sits later in the list but finished earlier to emerge ahead of the sibling tools still running (its siblings: the other tool calls dispatched together from the same assistant message, running side by side in the same concurrent batch). The quick Grep needn’t wait on the slow WebFetch. Ordering across a serial boundary, though, is a hard guarantee: one executing unsafe tool blocks both the start of new tools and the emission of later results.

5. “No chaos,” layer two: every call gets exactly one result

The API has an iron rule: every tool_use in an assistant message must be followed by exactly one matching result, or the next request is rejected outright (HTTP 400).

This kind of “must hold true no matter which path the program takes” rule is what engineering calls an invariant. Pairing completeness is the most important invariant in this whole topic.

The trouble is there are so many abnormal paths: the model errors out, the user interrupts, a tool gets cancelled — any one of them can leave some tool_use without a real result. One approach is to fill the hole with a fake one: a routine called make_missing_tool_results — it collects the ids of the results that do exist, then synthesizes an is_error=True result for every gap — is invoked on each of the abnormal paths: the model-error path, the interrupted-mid-stream path, and a final sweep after all tools have drained. Sibling tools cancelled because one of them died, or tools killed by a user interrupt, likewise leave behind a synthesized error result.

6. “No chaos,” layer three: parallel tools may not touch shared context directly

In building a real harness there’s a subtler kind of disorder: two parallel tools both mutating the shared session context at the same time, clobbering each other.

A decent harness’s fix is to not let tools mutate it directly at all. A tool that wants to change the context returns a ContextModifier object, and the orchestrator picks the moment to apply it. Serial tools’ modifications are applied immediately; parallel tools’ modifications are deferred until their result is emitted, and applied one after another, in emission order, on the orchestrator’s single-threaded event loop. The “simultaneous write” between parallel tools is thereby eliminated — every side effect ends up queued on one thread.

Once a turn ends, the next turn’s history is stitched together in a fixed order:

[ ...history, ...assistant messages, ...tool results ]

and then run through the message-normalization pass (merging adjacent user messages, filtering orphan thinking, and so on) before it’s sent to the API.

7. One tool fails, or the user interrupts — what about the rest?

Inside a parallel batch, only a Bash failure drags its siblings down. The orchestrator sets a sibling-abort flag and, for each still-running sibling whose interrupt behavior is "cancel", cancels its task.

Why is Bash special? Because when a shell command fails, it usually means the premise of this whole batch of parallel work has collapsed — if the compile didn’t even pass, the tests and analysis queued behind it are all wasted motion, and running them only amplifies the error.

A sibling whose interrupt behavior is "block" is instead let run to its natural end — for some tools (a write that’s mid-commit, say) cancelling halfway costs more than just finishing. Siblings that hadn’t started yet simply don’t start, and are closed out directly as a sibling-error. All of these cancelled or skipped tools leave behind a synthesized error result — back to the invariant from section 5: the pairing is always closed. (The full user-interrupt path is a story for another post.)

Here’s a real run of the harness, using gpt-5.5: two Write calls complete in parallel in the same turn, then Bash runs pytest alone — exactly the schedule described above.

active: model_family=openai-family
  Successfully wrote to /tmp/ccpy_e2e/fib.py
  [Write] /tmp/ccpy_e2e/fib.py
  Successfully wrote to /tmp/ccpy_e2e/test_fib.py
  [Write] /tmp/ccpy_e2e/test_fib.py
  [Bash] python -m pytest test_fib.py -q
1 passed in 0.01s
Created `fib.py` and `test_fib.py`.

8. What if you need to support both Anthropic and OpenAI wire formats?

As every post keeps circling back to these two, here they part ways again — on where the result goes:

• Anthropic wants all tool_result blocks packed into the single user message that immediately follows;
• OpenAI wants each result as its own separate role:"tool" message.

When writing a multi-vendor harness, the pairing logic must rest entirely on the one abstraction of match by id, with no other assumption smuggled in. Any code that leans on list position will, sooner or later, blow up on some proxy or some vendor.

9. What if you’re chasing the last drop of cache hit rate?

“Whoever finishes first is served first” has a side effect: across two runs of the same batch, the results can land in the history in a different order. But prompt-cache hits depend on the prefix being byte-for-byte stable — change the arrangement of results and you’ve punched a hole in your own cache prefix.

In production, if you’re squeezing for maximum hit rate, you can re-sort a parallel batch’s results back into block order before writing them to history: correctness is untouched (pairing is by id, remember), while prefix stability improves. JollySammy hasn’t had the bandwidth to optimize this part yet.

10. What if you don’t cap concurrency at all?

Tools are mostly I/O-bound, so it looks harmless to open a few more. But every concurrent Read / WebFetch holds a file handle, a network connection, and result memory; if the model gets excited and fires off 50 Grep calls at once, an uncapped harness will instantly exhaust its file descriptors and produce far more text than the budget allows. The default concurrency-cap semaphore keeps tail latency and resource use inside a predictable envelope, while leaving the environment variable as a knob.

11. What if two parallel tools both want to change the context, and they conflict?

The orchestrator applies the two ContextModifiers one after another, in emission order, on a single thread. There’s no data race — only a deterministic “the later one applied wins over the earlier.” If two modifications are genuinely mutually exclusive as a matter of business logic, the right fix is to have the tool declare itself not concurrency-safe, so they never land in the same parallel batch in the first place.