A Cosmic agent answers for 76% less
76%
lower cost per correct answer than fetching the site
83%
lower cost than browsing it
73%
fewer steps per answer than a browser
85%
lower cost than fetching, at 50,000 products
The cost of 10,000 correct answers
$411
Cosmic agent
$1,686
Fetch the site
$2,480
Browse the site
Measured cost per correct answer times 10,000, model spend only.
What it costs your site
The Cosmic agent's share was 57% of the CAP cost, about $0.023 per correct answer. The visiting agent pays the rest. Each answer was scored against the store's record, and every link had to open the right product in the right color and size.
277 frozen runs, $10.89 in model spend. Freeze 4e17533bb681, bench commit 53019ef.
Same questions, three ways
Cosmic beside the other ways an agent gets an answer
The visitor agent asks 50 questions about Fieldhouse, a running store, three times each. Through CAP it asks the Cosmic agent. Fetching and browsing read the store's realistic pages.
Cost against accuracy
Up is more accurate. Left is cheaper per correct answer.
All 150 runs
Cost per correct answer
| Cosmic, by kind of question | Cost vs fetching | Cost vs browsing | Steps vs fetching | Steps vs browsing |
|---|---|---|---|---|
| One fact | 15% less | 68% less | 24% fewer | 60% fewer |
| Multi-constraint | 77% less | 84% less | 76% fewer | 74% fewer |
| Page vs record | 78% less | 79% less | 62% fewer | 71% fewer |
| Local and delivery | 80% less | 86% less | 66% fewer | 76% fewer |
| Catalog query | 78% less | 87% less | 76% fewer | 77% fewer |
By kind of question
Cost per correct answer
- Cosmic agent
- Fetch the site
- Browse the site
One fact
Multi-constraint
Page vs record
Local and delivery
Catalog query
Catalog size
85% less at 50,000 products
The same store at 402, 5,000, and 50,000 products, with heavier, paginated pages for the agents that read the site. Three tasks, three runs each. Every Cosmic run was correct at every size.
| Cosmic, by catalog size | Cost vs fetching | Cost vs browsing | Steps vs fetching | Steps vs browsing |
|---|---|---|---|---|
| 402 products | 88% less | 89% less | 67% fewer | 77% fewer |
| 5,000 products | 85% less | 91% less | 64% fewer | 79% fewer |
| 50,000 products | 85% less | 88% less | 71% fewer | 77% fewer |
Cost by catalog size
Cost per correct answer
- Cosmic agent
- Fetch the site
- Browse the site
402 products
5,000 products
50,000 products
| Cosmic, by task | 402 products | 5,000 products | 50,000 products |
|---|---|---|---|
| A named product, in a color and size | 100%, $0.026, 11 s | 100%, $0.020, 11 s | 100%, $0.021, 11 s |
| The cheapest match for a color and size | 100%, $0.045, 26 s | 100%, $0.065, 30 s | 100%, $0.090, 55 s |
| A shipping question | 100%, $0.022, 14 s | 100%, $0.024, 14 s | 100%, $0.026, 14 s |
Share correct, cost per run, and seconds per run.
Agent model
Choose the model by budget
The same 50 questions with Sonnet, Opus, and Haiku running the Cosmic agent. Haiku cost the least per correct answer. Sonnet and Opus were equally accurate, and Sonnet cost less. Even Opus cost 68% less than fetching the site.
One pass over the 50 questions
Cost per correct answer
Method and limits
How to read these numbers
Every number on this page comes from one frozen run. The agent prompt, the backend commit, and the bench code were hashed before it started, and only runs that match the freeze count. A step is one tool call by the visitor agent.
- One store, Fieldhouse, with 50 questions asked three times each. Results on your content will differ.
- The visitor agent is the CAP benchmark's frozen visitor, Opus 5.5, in every run. The Cosmic agent runs Sonnet 5 unless the model section says otherwise.
- Fetch and browse numbers come from the frozen CAP benchmark run, with the same visitor, questions, and scorer. The Cosmic runs were frozen separately.
- Cost is list price for both sides: the visitor's spend plus the Cosmic agent's, divided by correct answers.
- The scale runs use three questions at each catalog size, three runs each. Treat them as a direction, not a precise figure.
- Reading the site was a little more accurate: 99% fetching it and 98% browsing it, against 95% through the Cosmic agent.
- A run takes about as long in seconds as fetching the site: 21.9 seconds through the Cosmic agent, 21.1 fetching, and 24.6 browsing.
How the benchmark scores answers ยท The harness and raw results
Give your agent a front door
CAP is free on every Cosmic agent. Turn it on in the dashboard and publish agent.txt.