Too many tools? Five ways to pick an agent's tools per turn
An agent with a handful of tools can send all of them on every turn. Past 20 or so, three things go wrong at once: tool definitions become most of the prompt, the model starts confusing look-alike tools, and multi-step tasks stall because the model doesn't know it needs a lookup first. Here is how the common fixes compare.
1. Send every tool#
Good: simple, and a provider's prompt cache keeps the repeated tool definitions cheap. Breaks down: every turn pays for every tool, even "thanks!"; accuracy drops as similar tools pile up; adding a tool makes every turn worse. Fine under about 15–20 tools.
2. A fixed short list, or sub-agents#
Hand-pick the common tools, or split one agent into several specialists with a router in front. Good: small prompts, predictable. Breaks down: rare tools never get offered (in one of our tests, a list of the 5 most-used tools covered 0 of 30 turns that needed a less common tool), the lists need constant upkeep, and every split adds a routing step that can pick the wrong specialist.
3. Ask an LLM to pick the tools#
A small model reads the request and the tool list and returns names (LangChain's LLMToolSelectorMiddleware works this way). Good: understands the request. Breaks down: an extra model round trip, often before every model call of the turn; it sees the tool list in its own prompt, so the cost grows with the catalog; it picks what the request mentions, not the lookups those tools need.
4. Provider tool search#
Anthropic and OpenAI let you mark tools as deferred: the model starts with a few tools and a search tool, and loads others when it decides it needs them. Good: native, and it keeps the prompt cache intact because loaded tools are appended rather than changing the start of the prompt. Breaks down: the model must notice a tool is missing and spend a step searching, by keywords over names and descriptions; it works only with those providers; each search adds a model step to the turn.
5. Select before the model call (Toolcue)#
A dedicated service reads the user's message and the recent conversation and returns the tools this turn needs, plus the lookups they depend on, in about 35 ms with no model call. The model starts the turn with the right tools. A find_tools tool covers the rare miss. Good: no extra model step; works with any model and framework; understands follow-ups ("ok, move them to the scale plan") and adds dependent lookups; the saving grows with the catalog (51–82% fewer prompt tokens in our benchmark). Trade-off: see prompt caching below.
At a glance#
| Extra model calls | Rare tools | Follow-ups | Dependent lookups | Any provider | |
|---|---|---|---|---|---|
| Send every tool | none | yes | yes | yes | yes |
| Fixed list or sub-agents | none (router: one) | no | depends | no | yes |
| LLM selector | one per model call | yes | last message only | rarely | yes |
| Provider tool search | one step per search | when searched | yes | when searched | Anthropic, OpenAI |
| Toolcue | none | yes | yes | yes | yes |
About prompt caching#
Providers cache the start of the prompt, and tool definitions come first. Any approach that changes the tool list between turns (a fixed list doesn't; options 3 and 5 do) means the tool part of the prompt isn't reused from the previous turn. Within one turn's tool loop the list stays fixed, so those calls still hit the cache. In practice:
- A 4.5K-token prompt instead of a 24.2K-token one (our 77-tool benchmark) usually outweighs a cache miss on the smaller prompt, especially in short conversations. Keep your always-on tools first and in a stable order so the part that does repeat stays cacheable.
- For long, cache-heavy conversations on Anthropic or OpenAI, provider tool search keeps the cache warm, and the two combine: Toolcue's
find_toolssearch can sit behind the provider's search tool.
Choosing#
- Under 20 tools: send them all.
- A growing catalog, any framework or provider: select before the model call, and keep a
find_toolstool for misses. See the integrations for LangChain, LangGraph, the Vercel AI SDK, Pydantic AI, the OpenAI Agents SDK and MCP. - Very long conversations on one provider: combine selection with that provider's tool search.