Getting started: give your agent only the tools each turn needs
Toolcue is in early access. Join the waitlist and we send you an endpoint and keys. Every example below was run against Toolcue on 2026-11-01; the runnable copies are in
examples/tool-selection/.
Your agent has 40 tools. Every turn, all 40 tool definitions go into the prompt: a greeting costs as much as a refund, the model has more ways to pick the wrong tool, and adding the 41st makes every turn slower and pricier.
Toolcue runs right before your LLM call. You send it the user's message (and the conversation so far); it answers in tens of milliseconds with the tools this turn needs:
- the tools that match the request, in any language;
- the lookups they depend on: "refund Globex's last invoice" also gets "list a customer's invoices" and "find customer", because the refund needs an invoice id and the invoice list needs a customer id;
- ids it already saw in the conversation (
customer_id: cus_8812), so the model does not look them up twice; - nothing but your always-on tools for "thanks!".
There is no LLM call inside Toolcue and no framework lock-in: you keep your agent, your model and your tool code. Toolcue only tells you which of your tools to pass this turn.
user message ─▶ your backend ──▶ POST /v1/context ──▶ ["refund_invoice", "get_customer_invoices", "find_customer", ...]
│ │
└──── your LLM call with only those tools ◀┘What to expect (measured on real production conversations of B2B SaaS agents, and on synthetic catalogs grown by 60 new tools):
| The tool the turn needed was attached | 99% of turns (with your always-on tools pinned), 94–97% on a catalog that just grew by 60 tools it had never seen used |
| Prompt size | 51–82% fewer prompt tokens per turn |
| Latency | ~35 ms median, ~55 ms p95 for the selection itself (measured on a laptop, Apple M3, 4 GB container); faster on repeated messages |
1. Get access#
Toolcue is hosted. When your access is ready you get a base URL and two tokens: a read token for the per-turn calls and an admin token for publishing your catalog. Keep both on your server. Need to run it in your own cloud? Say so on the waitlist form: a self-hosted image is planned.
Install the client (the HTTP API works from any language too; see the API reference):
pip install toolcue # Python 3.10+, no dependencies
npm install toolcue # Node 18+, no dependenciesBoth read three environment variables:
export TOOLCUE_URL=... # the base URL you were given
export TOOLCUE_TOKEN=... # read token: per-turn calls
export TOOLCUE_ADMIN_TOKEN=... # admin token: publishing2. Publish your tools#
An agent is one catalog of tools, under an id you choose (acme-billing). Use one agent per assistant, and one per tenant if tenants have different tools: Toolcue learns per agent and never mixes them. Publish again whenever your tools change (it replaces the whole catalog).
curl -X PUT "$TOOLCUE_URL/v1/agents/acme-billing/catalog" \
-H "Authorization: Bearer $TOOLCUE_ADMIN_TOKEN" -H "Content-Type: application/json" \
--data @examples/tool-selection/catalog.json{"agentId": "acme-billing", "tools": 9, "edges": 5, "traces": 0,
"edgesBySource": {"DECLARED": 4, "STATIC": 1, "TRACE": 0}}edges are the dependency links it found: which tool supplies an id another tool needs. Each tool in the catalog:
{
"name": "refund_invoice",
"displayName": "Refund an invoice",
"description": "Refund a paid invoice fully or partially.",
"parameters": [
{"name": "invoice_id", "description": "The invoice id, e.g. INV-1042", "dataType": "string", "required": true},
{"name": "amount", "description": "Amount to refund; omit for a full refund", "dataType": "number", "required": false}
],
"examples": ["give them their money back", "refund INV-1042"]
}| Field | What it is for |
|---|---|
name | Your tool's name. Toolcue answers with names, so use the exact name your LLM sees. |
description | What the tool does, in plain words. The single most important field for matching. |
parameters | Name, description, type, required. Required id parameters (invoice_id) are how Toolcue knows a lookup is needed. |
examples | Optional. How users actually ask for it ("give them their money back"). Helps with vocabulary the description lacks. |
produces | Optional. Ids the tool returns (["customer_id"]). Makes it the lookup for every tool that needs that id; without it Toolcue infers links from names, with lower confidence. |
core | Optional. Always attach (e.g. "ask the user a question"). |
requiredRoles | Optional. Only for users with one of these roles (sent per turn in context.user.roles). |
Already have tool definitions? Convert them instead of writing JSON by hand: tools_from_definitions() (Python) and toolsFromDefinitions() (TypeScript) accept OpenAI, Anthropic and MCP tool definitions as they are:
from toolcue import ToolcueClient, tools_from_definitions
toolcue = ToolcueClient()
toolcue.publish_catalog("acme-billing", tools_from_definitions(MY_TOOLS, extras={
"find_customer": {"produces": ["customer_id"]},
"ask_user": {"core": True},
}))3. Ask for a turn's tools#
curl -X POST "$TOOLCUE_URL/v1/context" \
-H "Authorization: Bearer $TOOLCUE_TOKEN" -H "Content-Type: application/json" \
-d '{"agentId": "acme-billing",
"message": "Refund the last invoice for Globex",
"context": {"user": {"roles": ["agent"]}},
"options": {"includePrompt": false}}'{
"agentId": "acme-billing",
"turnType": "ACTION",
"tools": [
{"name": "refund_invoice", "reason": "RETRIEVED", "detail": "matched the turn"},
{"name": "get_customer_invoices", "reason": "DEPENDENCY", "detail": "provides invoice_id for refund_invoice"},
{"name": "find_customer", "reason": "DEPENDENCY", "detail": "provides customer_id for get_customer_invoices"},
{"name": "create_invoice", "reason": "RETRIEVED", "detail": "one of the best matches for a request"},
{"name": "cancel_subscription", "reason": "RERANKED", "detail": "reranker"},
{"name": "ask_user", "reason": "CORE", "detail": "always attached"}
],
"resolvedParams": {},
"timings": {"totalMs": 46.7}
}(Scores and the detailed timings trimmed.) Pass the tools named in tools to your model, in that order. Toolcue deliberately adds a few close matches beyond the obvious one (create_invoice, cancel_subscription): a couple of hundred extra tokens is cheaper than a turn that fails because the right tool was left out.
reason | Why the tool is there |
|---|---|
RETRIEVED | It matches what the user asked |
DEPENDENCY | It supplies an id a selected tool needs |
STICKY | It was used in the last turns, or acts on the record a recent tool returned |
RERANKED | A close match kept as a safety net |
CORE | You marked it always-on |
PAGE | It is tied to the page the user is on (only when you send context.page) |
turnType tells you what kind of turn it is: SMALLTALK ("thanks!" gets only your core tools), KNOWLEDGE, ACTION or MIXED.
Send the conversation. Follow-ups ("ok, move them to the scale plan") only make sense with history. Send the recent messages, including the tool calls and their results; Toolcue reads ids out of them:
{"agentId": "acme-billing",
"message": "ok, move them to the scale plan",
"history": [
{"role": "user", "content": "what plan is Globex on?"},
{"role": "assistant", "content": "Globex (cus_8812) is on growth.",
"toolCalls": [{"name": "find_customer", "arguments": {"query": "Globex"},
"result": "{\"customer_id\": \"cus_8812\", \"plan\": \"growth\"}"}]}],
"options": {"includePrompt": false}}The answer starts with change_plan, keeps find_customer (STICKY), and returns "resolvedParams": {"customer_id": "cus_8812"}: you can put that value in the prompt so the model does not look the customer up again.
Send the user's roles from your server. context.user.roles decides who sees role-restricted tools, so set it from your authenticated session, never from values the browser sends. Call Toolcue from your backend only.
4. Use it from your code#
The pattern is the same in every framework: keep your tool definitions where they are, ask Toolcue for the names, and pass your model only those tools. Keep the tool list fixed for the whole user turn (every model call in your tool loop), and select again on the next user message.
Python, plain#
from toolcue import ToolcueClient
toolcue = ToolcueClient()
keys = toolcue.select_names("acme-billing", "Refund the last invoice for Globex", history=[], user={"roles": ["agent"]})
tools_for_this_turn = [t for t in MY_TOOLS if t["function"]["name"] in keys]TypeScript, plain#
import { ToolcueClient } from "toolcue";
const toolcue = new ToolcueClient();
const keys = await toolcue.selectNames("acme-billing", "Refund the last invoice for Globex", [], { roles: ["agent"] });
const toolsForThisTurn = MY_TOOLS.filter((t) => keys.includes(t.function.name));OpenAI SDK (Python)#
import os
from openai import OpenAI
from toolcue import ToolcueClient, tools_from_definitions
client, toolcue = OpenAI(), ToolcueClient()
TOOLS = [...] # your {"type": "function", "function": {...}} list
BY_NAME = {t["function"]["name"]: t for t in TOOLS}
toolcue.publish_catalog("acme-billing", tools_from_definitions(TOOLS)) # at deploy time, not per turn
def answer(messages, user):
keys = toolcue.select_names("acme-billing", messages[-1]["content"], history=to_toolcue_history(messages[:-1]), user=user)
return client.chat.completions.create(
model=os.environ["OPENAI_MODEL"],
messages=messages,
tools=[BY_NAME[k] for k in keys if k in BY_NAME],
)to_toolcue_history maps your stored messages to {"role", "content", "toolCalls": [{"name", "arguments", "result"}]}; the last 10 or so messages are plenty.
Anthropic SDK (Python)#
import anthropic
from toolcue import ToolcueClient, tools_from_definitions
client, toolcue = anthropic.Anthropic(), ToolcueClient()
TOOLS = [...] # your {"name", "description", "input_schema"} list
BY_NAME = {t["name"]: t for t in TOOLS}
toolcue.publish_catalog("acme-billing", tools_from_definitions(TOOLS))
def answer(user_text, history, user):
keys = toolcue.select_names("acme-billing", user_text, history=history, user=user)
tools = [BY_NAME[k] for k in keys if k in BY_NAME] # fixed for the whole turn
messages = [{"role": "user", "content": user_text}]
while True:
response = client.messages.create(model="claude-opus-5-5", max_tokens=16000, tools=tools, messages=messages)
if response.stop_reason != "tool_use":
return response
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": [
{"type": "tool_result", "tool_use_id": b.id, "content": run_tool(b.name, b.input)}
for b in response.content if b.type == "tool_use"]})LangChain / LangGraph (Python)#
pip install "toolcue[langchain]", then add one middleware to create_agent. It selects once per user message, gives every model call of that turn the same tools, adds a find_tools tool for recovery, refuses tools that weren't offered, and falls back to every tool if Toolcue fails:
from langchain.agents import create_agent
from toolcue.langchain import ToolcueMiddleware, publish_tools
publish_tools("acme-billing", TOOLS, extras={"find_customer": {"produces": ["customer_id"]}}) # at deploy time
agent = create_agent("anthropic:claude-opus-5-5", tools=TOOLS,
middleware=[ToolcueMiddleware("acme-billing", user=lambda ctx: {"roles": ctx.roles})],
context_schema=Session)In a hand-built LangGraph graph, add a select node before the model node and bind filter_tools(TOOLS, state["toolcue_tools"]). Details: LangChain and LangGraph.
Vercel AI SDK (TypeScript)#
npm install toolcue ai zod, then select once per turn and let prepareStep keep those tools active on every step:
import { generateText, stepCountIs } from "ai";
import { findToolsTool, selectTurn, toolcuePrepareStep } from "toolcue/ai-sdk";
const turn = await selectTurn("acme-billing", messages, { user: { roles: session.roles } });
const tools = { ...TOOLS, find_tools: findToolsTool(turn) };
const result = await generateText({ model, tools, messages, stopWhen: stepCountIs(10),
prepareStep: toolcuePrepareStep(turn, tools) });Details: Vercel AI SDK. Also tested and packaged: Pydantic AI (toolcue[pydantic-ai]) and the OpenAI Agents SDK (toolcue[openai-agents]).
MCP servers#
An MCP client already has every server's tools from tools/list; publish them once and filter per turn:
listed = await session.list_tools()
toolcue.publish_catalog("my-assistant", tools_from_definitions([t.model_dump() for t in listed.tools]))
keys = toolcue.select_names("my-assistant", user_text, history)
tools_for_model = [t for t in listed.tools if t.name in keys]With several servers, prefix names (github__create_issue) so keys stay unique, and publish again when a server's tool list changes.
5. When the right tool was left out: find_tools#
No selection is perfect. Give the model a way to recover in the same turn: add one small tool of your own, find_tools, backed by Toolcue's catalog search, and add whatever it finds to the tool list.
FIND_TOOLS = {"name": "find_tools",
"description": "If none of your tools can do what the user asked, describe the task here to get more tools.",
"input_schema": {"type": "object", "properties": {"task": {"type": "string"}}, "required": ["task"]}}
# in your tool loop, when the model calls find_tools:
found = toolcue.find_tools("acme-billing", block.input["task"], exclude=[t["name"] for t in tools], limit=3, user=user)
tools += [BY_NAME[t["name"]] for t in found["tools"] if t["name"] in BY_NAME]
result = "New tools available: " + ", ".join(t["name"] for t in found["tools"]) if found["tools"] else "No matching tool."The model sees the new tools on its next call in the same turn. find_tools applies the same role rules as /v1/context: a non-admin asking to "set the VAT rate" gets nothing back.
6. Make it better over time#
- Write descriptions for users, not for developers: "Refund a paid invoice fully or partially" beats "POST /invoices/{id}/refund".
- Declare lookups with
produceson the tools that return ids. Name id parameters the same everywhere (customer_id, notcustin one tool andcustomerIdin another). - Add
examplesfor tools users describe in their own words ("give them their money back"). - Mark interaction helpers
core: "ask the user", "hand off to a human". They should never depend on matching. - Send past conversations (
POST /v1/agents/{id}/traces, same message shape ashistory). Toolcue learns which tools usually follow which when a pattern repeats, e.g. "find customer, then cancel subscription". - Mind the prompt cache: a different tool list each turn means the provider can't reuse the tool part of the previous turn's cached prompt (calls within one turn still can). Keep always-on tools first and in a stable order; for very long conversations on Anthropic or OpenAI, combine Toolcue with the provider's tool search.
- Small catalogs: under about 15–20 tools, sending all of them is fine; Toolcue pays off as the catalog grows, and keeps paying off when it grows by dozens of tools a month.
Reference#
| Method | Path | Token | What it does |
|---|---|---|---|
| PUT | /v1/agents/{agentId}/catalog | admin | Publish (replace) the agent's tools |
| POST | /v1/agents/{agentId}/traces | admin | Add past conversations to learn from |
| DELETE | /v1/agents/{agentId} | admin | Delete the agent |
| POST | /v1/context | read | The tools for one turn |
| POST | /v1/find-tools | read | Search the whole catalog (query, exclude, limit ≤ 50) |
| GET | /v1/agents/{agentId} | read | Counts: tools, dependency links, conversations |
| GET | /v1/agents/{agentId}/catalog | read | The catalog as published |
| GET | /v1/agents/{agentId}/graph | read | The dependency links (provider → consumer, the id, how it was found) |
/v1/context request: agentId, message, history (optional), context.user (roles and any user attributes, optional), context.page (url, title; optional), options.maxTools (≤ 50) and options.includePrompt (set it to false when you build your own prompt).
Errors: 401 missing or wrong token (or a read token on a publishing call), 404 {"error": "unknown agent: ..."}, 400 invalid input, 413 body too large (8 MB for publishing, 256 KB otherwise), 429 rate limited (with Retry-After, on deployments that enable it).
Fail open. If Toolcue is slow or unreachable, send all your tools for that turn: set a short timeout (the clients default to 2 s) and fall back. Your agent never depends on Toolcue to answer.