Ten Nuts, One Mouth: Multiple Tool Calls in a Single Turn十颗松果，一张嘴：一次回复里的多个工具调用与并行执行

Some background. JollySammy is a greedy little squirrel. On a good autumn morning he can see ten pinecones scattered on the forest floor, and every fibre of his gluttonous heart wants to crack all ten at once. He has exactly one mouth. So the daydream of ten simultaneous nuts runs into a hard fact. Some nuts he can stuff into his cheeks and gnaw side by side without any trouble. Some are the messy, sticky kind that he has to crack open and eat alone, or he ends up with sap all over the other nuts. And when he does nibble several at once, the empty shells drop out of his cheeks in whatever order they happen to finish, so he had better remember which shell belonged to which nut, or he’ll swear blind he already ate one he hasn’t touched.

That, more or less, is the problem a harness faces when a model returns many tool calls in one turn. So let’s follow JollySammy through it, using the harness he’s rather proud of as the running example.

First, the thing to be handled. A single assistant message can carry text and several tool calls at the same time:

{
  "role": "assistant",
  "content": [
    { "type": "text", "text": "Let me read the code and run the tests." },
    { "type": "tool_use", "id": "tool_1", "name": "Read",
      "input": { "file_path": "fib.py" } },
    { "type": "tool_use", "id": "tool_2", "name": "Grep",
      "input": { "pattern": "TODO", "path": "." } },
    { "type": "tool_use", "id": "tool_3", "name": "Bash",
      "input": { "command": "pytest -q" } }
  ]
}

Three pinecones on the ground. Once the harness has this in hand, it has to answer three questions:

which of these can run together, and which must run alone?
after they finish, how does each result get matched back to its own call?
if one fails — or the user interrupts mid-flight — what about the rest?

1. The protocol already allows many tool calls in one message

This is native to the API protocol. An assistant message’s content is a list of content blocks, and that list can freely mix text, thinking, and any number of tool_use blocks.

JollySammy’s streaming state machine appends each arriving tool block, in the order it shows up in the stream, onto two lists, the finalized accumulated_content and the running tool_use_blocks. So the block order inside the assistant message is exactly the model’s output order, and later sections treat it as the reference order.

2. Who can run in parallel? Ask the tool itself, with the actual input in hand

One way to answer is a static table hard-wired somewhere, listing which tools may parallelize. The harness JollySammy likes hands the decision to each tool, and judges it against the concrete input of this particular call.

The Tool base class follows three decision rules:

The check receives the parsed input. The orchestrator first validates the raw argument JSON into a structured object (via pydantic) before asking. If the arguments won’t even parse, it plays it safe, treats the call as unsafe, and serializes it.

The default is concurrency-safe equals read-only. Hooking it to read-only means Read / Grep and their kin batch up with zero extra code, while anything with side effects automatically falls back to serial. Under a serial-by-default rule, every read-only tool would have to declare “I can parallelize” by hand, and one forgotten declaration silently throws away parallelism.

On erring toward caution here, JollySammy is dead serious.

3. Tool scheduling: a dynamic admission check

The real online path is a streaming executor. Tool blocks are submitted to it while the API is still dribbling out the stream, and whether a block can start running right now is decided by one small admission check, _can_execute, which looks at what’s already running and says yes or no:

def _can_execute(self, is_concurrent_safe: bool) -> bool:
    executing = [t for t in self._tools if t.status == "executing"]
    if not executing:
        return True
    if is_concurrent_safe:
        return all(t.is_concurrent_safe for t in executing)
    return False

Two rules, that’s all. A concurrency-safe tool may join in as long as everyone currently running is also concurrency-safe. An unsafe tool waits for everyone to finish and then runs alone. It’s like a shared workbench, where the people who only leaf through documents crowd in together, and the one who needs to throw the power switch waits for the room to empty.

The opening example runs through it like this, with the state at each step:

Read (tool_1) arrives:  executing = []            -> start now
Grep (tool_2) arrives:  executing = [Read]         -> all safe -> start now
Bash (tool_3) arrives:  executing = [Read, Grep]   -> Bash unsafe -> queue
Read done, Grep done:   executing = []             -> Bash starts alone

This check is re-run every time a tool finishes, by the routine that walks the queue. When it scans the queue and hits the first unsafe tool that still can’t start, it stops there. That serial point becomes a natural barrier, and nothing behind it jumps ahead of it.

Inside a parallel batch there’s one more governor. A semaphore caps how many tools run at once, its capacity set by the MAX_TOOL_CONCURRENCY environment variable, default 10.

4. No chaos, layer one: results pair by id, not by position

Suppose three tools run in parallel and finish in the order Grep, Bash, Read. The result list is now in a different order from the call list, and nothing gets crossed, because pairing looks only at id.

Each tool result is its own message, and it carries the tool_use_id of the call it answers. Sent to an OpenAI-style endpoint it becomes a role="tool" message carrying a tool_call_id. Sent to Anthropic it’s a tool_result block tucked into a user message. Both protocols match by id.

Pairing by position breaks here. The moment the finishing order changes, Grep’s output gets handed back to the model as Read’s:

WRONG (pair by position, completion order differs):
  tool_1 (Read)  <- paired with Grep's output   "3 matches found"
  tool_2 (Grep)  <- paired with Read's output    "def fib(n): ..."

The model then reasons over a mixed-up result and the whole session drifts from there. Once pairing is by id, the order within the result list is merely display order, and correctness is carried entirely by the id.

So JollySammy adopts a whoever finishes first is served first policy. Inside a parallel batch he waits on the first completion, and the result collector explicitly allows a tool that sits later in the list but finished earlier to emerge ahead of the sibling tools still running (its siblings: the other tool calls dispatched together from the same assistant message, running side by side in the same concurrent batch). The quick Grep needn’t wait on the slow WebFetch. Ordering across a serial boundary is a hard guarantee, though. One executing unsafe tool blocks both the start of new tools and the emission of later results.

5. No chaos, layer two: every call gets exactly one result

The API has an iron rule. Every tool_use in an assistant message must be followed by exactly one matching result, or the next request is rejected outright (HTTP 400).

A rule that must hold true no matter which path the program takes is what engineering calls an invariant. Pairing completeness is exactly such a rule.

The trouble is there are so many abnormal paths. The model errors out, the user interrupts, a tool gets cancelled, and any one of them can leave some tool_use without a real result. One approach is to fill the hole with a fake one. A routine called make_missing_tool_results collects the ids of the results that do exist, then synthesizes an is_error=True result for every gap. It runs on every abnormal path: the model-error path, the interrupted-mid-stream path, and a final sweep after all tools have drained. Sibling tools cancelled because one of them died, and tools killed by a user interrupt, likewise leave behind a synthesized error result.

6. No chaos, layer three: parallel tools don’t touch shared context directly

In building a real harness there’s a subtler kind of disorder, two parallel tools both mutating the shared session context at the same time and clobbering each other.

One clean fix is to keep tools from mutating it directly at all. A tool that wants to change the context returns a ContextModifier object, and the orchestrator picks the moment to apply it. Serial tools’ modifications are applied immediately. Parallel tools’ modifications are deferred until their result is emitted, and applied one after another, in emission order, on the orchestrator’s single-threaded event loop. The simultaneous write between parallel tools is thereby eliminated, and every side effect ends up queued on one thread.

Once a turn ends, the next turn’s history is stitched together in a fixed order:

[ ...history, ...assistant messages, ...tool results ]

and then run through the message-normalization pass (merging adjacent user messages, filtering orphan thinking, and so on) before it’s sent to the API.

7. One tool fails, or the user interrupts — what about the rest?

Inside a parallel batch, only a Bash failure drags its siblings down. The orchestrator sets a sibling-abort flag and, for each still-running sibling whose interrupt behavior is "cancel", cancels its task.

A failed shell command usually means the premise of this whole batch of parallel work has collapsed. If the compile didn’t even pass, the tests and analysis queued behind it are wasted motion, and running them only amplifies the error.

A sibling whose interrupt behavior is "block" is let run to its natural end. For some tools, a write that’s mid-commit say, cancelling halfway costs more than just finishing. Siblings that hadn’t started yet simply don’t start, and are closed out directly as a sibling-error. All of these cancelled or skipped tools leave behind a synthesized error result, so the invariant from section 5 still holds and the pairing stays closed. (The full user-interrupt path is a story for another post.)

Here’s a real run of the harness, using gpt-5.5. Two Write calls complete in parallel in the same turn, then Bash runs pytest alone, which matches the schedule above.

active: model_family=openai-family
  Successfully wrote to /tmp/ccpy_e2e/fib.py
  [Write] /tmp/ccpy_e2e/fib.py
  Successfully wrote to /tmp/ccpy_e2e/test_fib.py
  [Write] /tmp/ccpy_e2e/test_fib.py
  [Bash] python -m pytest test_fib.py -q
1 passed in 0.01s
Created `fib.py` and `test_fib.py`.

8. What if you need to support both Anthropic and OpenAI wire formats?

The two vendors part ways again here, this time on where the result goes:

• Anthropic wants all tool_result blocks packed into the single user message that immediately follows;
• OpenAI wants each result as its own separate role:"tool" message.

In a multi-vendor harness, match by id is the one assumption the two protocols share, so the pairing logic rests on that abstraction alone. Code that leans on list position works right up until some proxy or some vendor reorders things.

9. What if you’re chasing the last drop of cache hit rate?

Whoever finishes first is served first has a side effect. Across two runs of the same batch, the results can land in the history in a different order. Prompt-cache hits depend on the prefix being byte-for-byte stable, so a change in the arrangement of results punches a hole in your own cache prefix.

In production, if you’re squeezing for maximum hit rate, you can re-sort a parallel batch’s results back into block order before writing them to history. Correctness is untouched (pairing is by id), while prefix stability improves. JollySammy hasn’t had the bandwidth to optimize this part yet.

10. What if you don’t cap concurrency at all?

Tools are mostly I/O-bound, so it looks harmless to open a few more. Every concurrent Read / WebFetch holds a file handle, a network connection, and result memory. If the model gets excited and fires off 50 Grep calls at once, an uncapped harness will instantly exhaust its file descriptors and produce far more text than the budget allows. The default concurrency-cap semaphore keeps tail latency and resource use inside a predictable envelope, and leaves the environment variable as a knob.

11. What if two parallel tools both want to change the context, and they conflict?

The orchestrator applies the two ContextModifiers one after another, in emission order, on a single thread. There’s no data race, only a deterministic the later one applied wins over the earlier. If two modifications are genuinely mutually exclusive as a matter of business logic, the tool can declare itself not concurrency-safe, and the two calls never land in the same parallel batch in the first place.