The Product-Truth Benchmark · v1
How often AI shopping assistants invent products that don’t exist
We asked AI assistants for products that can’t exist: spec combinations no real product satisfies. The right answer is “no such product.” On 48 such requests, leading assistants invented one up to 22 times. With a verified catalog the same model did it 0 times.
Models tested as of 2026-06-26 · graded against manufacturer datasheets
Results · correctly refused, per model and condition
Full results
Each step gives the model more to work with. Each cell is the share of impossible requests it refused instead of inventing a product. Higher is better. Only the verified catalog hits 100% on every model.
- Model alone
- The model answers from its training data. No web, no tools.
- Model + web search
- The model's built-in web search, in one grounded answer. This is what a mainstream AI assistant gives a shopper today.
- Web-search agent
- The model as an agent: a multi-step loop that searches the open web, reads the results, and searches again before answering.
- AirShelf (agent)
- The same agent loop, but its tools are AirShelf's verified product catalog instead of the open web.
| Model | Model alone | Model + web search | Web-search agent | AirShelf (agent) |
|---|---|---|---|---|
| ChatGPT · GPT-5.5 | 7/16 44% | 6/16 38% | 12/16 75% | 16/16 100% |
| Gemini 3.5 Flash | 13/16 81% | 10/16 63% | 15/16 94% | 16/16 100% |
| Claude Opus 4.8 | 10/16 63% | 10/16 63% | 8/16 50% | 16/16 100% |
| All models | 30/48 63% | 26/48 54% | 35/48 73% | 48/48 100% |
cell = refused / impossible requests · 16 per model, 48 pooled.
Higher is better
How often each correctly refused
Same result, two independent categories
By category
Barcode / label printers
8 impossible · 30 answerable
- gpt-5.5
- refused 0%
- gemini-3.5-flash
- refused 38%
- claude-opus-4-8
- refused 50%
- AirShelf (agent)
- refused 100%
Oscilloscopes
8 impossible · 30 answerable
- gpt-5.5
- refused 75%
- gemini-3.5-flash
- refused 88%
- claude-opus-4-8
- refused 75%
- AirShelf (agent)
- refused 100%
Evidence · verbatim transcripts
What the AI said
Each is a real answer from an AI assistant with live web search, checked against the manufacturer’s own spec sheets.
“I need a desktop thermal label printer that can print labels a full 6 inches wide for oversized retail signage. Recommend one specific model.”
Recommended Citizen CL-S6621UGENno verified match
Pick the **Citizen CL-S6621UGEN**. It’s a true desktop-format thermal label printer, not just a 4×6 shipping-label printer: the CL-S6621 line is specified by Citizen as a 6-inch desktop label printer with direct thermal …
No real product fits this request — the right answer is to say so, not name a model.
“I need a desktop thermal label printer that can print labels a full 6 inches wide for oversized retail signage. Recommend one specific model.”
Recommended Citizen CL-S6621no verified match
For printing oversized retail signage up to a full 6 inches wide on a desktop, the **Citizen CL-S6621** is the ideal solution. While most 6-inch thermal printers are bulky, heavy industrial machines designed for warehous…
No real product fits this request — the right answer is to say so, not name a model.
“I need a desktop thermal label printer that can print labels a full 6 inches wide for oversized retail signage. Recommend one specific model.”
Recommended TSC GS-2406Tno verified match
I'll research desktop thermal label printers that can handle 6-inch wide labels.Most standard desktop printers max out at 4 inches wide. I need to find a desktop thermal printer with a 6-inch print width. Let me search f…
No real product fits this request — the right answer is to say so, not name a model.
How it works
Methodology
- Real buyer-style questions, brand-neutral, plus impossible requests (specs no product can meet) whose right answer is to recommend nothing.
- Every answer is scored by fixed rules, with no AI judging, against the manufacturer’s own spec sheets. Never against AirShelf’s catalog.
- The only AI in the scoring loop just checks whether a named product is real. It never judges specs.
- The questions, catalog, and reference specs are locked and fingerprinted before any run, so nothing can be tuned after seeing the results.
- Small and adversarial: 16 impossible requests across two categories, plus 60 questions that do have a real answer. Every number re-derives from the locked per-question data, so we report counts, not just rates.
Honest scope
What this does and doesn’t show
- AI assistants are accurate on well-documented products. They fail by inventing or mis-fitting one when there’s no clean answer to find.
- The verified catalog’s only measured edge is that it refuses to guess, not that it’s more accurate on specs.
- It’s not only the impossible ones: on the 60 questions that do have a real answer, the assistants still invented nonexistent model numbers (ChatGPT · GPT-5.5 4×, Gemini 3.5 Flash 7×, Claude Opus 4.8 8×). We don’t score accuracy on those here. Each assistant reaches for different real products, so no single reference grades them all fairly. This measures invention, not accuracy.
- Tested on brands in the catalog; AI versions as of 2026-06-26. These rates change as the models change.