Benchmark
How often the tool a turn needed was among the tools Toolcue attached, how much smaller the prompt got, and how long selection took. Measured on production conversations of two B2B SaaS in-app assistants and on a synthetic test where the catalog grows. Numbers as of October 2026.
| Result | |
|---|---|
| The needed tool was attached | 99.3% and 99.2% of held-out production turns (two assistants) |
| After the catalog grew by 60 tools | 93.8% of 564 requests; 96.4% for the 60 new tools |
| Prompt size per turn | 51–82% fewer tokens than sending every tool |
| Selection time | ~35 ms median, ~55 ms p95 on a laptop |
Real conversations#
Assistant one: 90 days of production conversations, about 2,000, with 17 tools. Toolcue was tuned on the earlier weeks and tested on the later ones: 437 turns in which the assistant ran a tool. The label is the tool the production assistant actually called; an independent judge re-checked a sample of 150 turns and agreed with the assistant on 140.
| Turns with the needed tool attached | |
|---|---|
| Toolcue | 426 / 437 (97.5%) |
Toolcue, with the two "ask the user" helpers marked always-on (core) | 434 / 437 (99.3%) |
| Rare tools only (outside the 5 most used), with the helpers always-on | 30 / 30 |
| A fixed list of the 5 most-used tools, for comparison, on the same rare turns | 0 / 30 |
The prompt went from 13.7K to 4.5K tokens per turn (−67%).
Assistant two: held-out production turns that ran a tool: 122 / 123 (99.2%) had the needed tool attached. The prompt went from 12.1K to about 5.9K tokens per turn (−51%). Its users called only 6 different tools in that period, so a fixed list of the 5 most-used tools also covers 122 / 123: here Toolcue does as well as a hand-picked list, without anyone keeping that list up to date. The growth test below is where the two part ways.
A catalog that grows#
Real traffic tests the tools people already use. To see what happens when a catalog grows, we wrote 60 realistic new tools for assistant one, 16 of them deliberate look-alikes of existing tools (bringing it to 77), and 12 differently phrased requests per tool, without retuning anything.
| Requests with the needed tool attached | |
|---|---|
| Today's 17 tools (204 varied requests) | 197 / 204 (96.6%) |
| Grown to 77 tools (564 requests) | 529 / 564 (93.8%) |
| ... requests for the 60 new tools | 347 / 360 (96.4%) |
At 77 tools, sending every tool costs 24.2K prompt tokens per turn; with Toolcue it stays at 4.5K (−82%). The saving grows with the catalog, because the selection stays about the same size.
Speed#
| Median | p95 | |
|---|---|---|
| The 9-tool sample catalog from the examples, new messages | ~35 ms | ~55 ms |
| Assistant one's 17 tools, new messages | ~45 ms | ~70 ms |
| A message seen before | under 1 ms |
Measured on a laptop (Apple M3) with the service in a 4 GB container. Selection makes no LLM call, so these times are the whole cost.
Method#
- Needed tool attached means the tool the turn needed was in the list Toolcue returned, including the lookups it added. We measure it this way because a turn fails when the right tool is missing, while a few extra tools cost a few hundred tokens.
- Prompt size counts the tool definitions plus the rest of the assistant's prompt, with every tool versus with Toolcue's selection.
- Held out means the test conversations came after the ones used for tuning.
Limits#
- Two assistants, both B2B SaaS in-app assistants. We're adding more as early-access teams join.
- The production data is our customers' and isn't published. A benchmark you can reproduce yourself, on public tool-retrieval datasets with the same scripts, is in progress; it will be on this page.