When the demo lies.

Have you noticed that AI demos never fail?

Ten minutes, a clean example, a perfect answer, applause. Then the same system meets your actual Tuesday — the forwarded email chain with three PDF documents, the invoice with the handwritten note, the customer who is angry about something adjacent to what they’re describing — and the magic gets patchy.

Nobody defrauded you. Demos are built the way show homes are furnished. What you saw was real; it was also the best case, on the cleanest input, with the failure modes politely waiting outside.

The lie isn’t in the demo. It’s in the inference — it handled that, so it’ll handle ours.

So here are the questions I ask in every AI demo now, in order of how uncomfortable they make the presenter:

What happens when the input is incomplete — not wrong, just normal-messy? Show me. What does it do when it isn’t sure — does it guess confidently, or stop and say so? What did it do the last hundred times, not the one time on stage — where are those numbers? And what’s my cost per task when this runs at my volume, not demo volume?

A vendor with a good product loves these questions. They’ve got answers ready, and they’re a little relieved someone finally asked. A vendor going pale at question two has just told you where the bodies are.

You’ll notice the Lab here runs the other way around: every experiment is labelled synthetic, sample outputs say “authored — not this model’s output”, and nothing pretends to be live while it isn’t. That’s not modesty. It’s the standard I think we should all be holding demos to — and you can’t ask vendors for honesty you don’t practise.

The demo is the show home. Ask to see a Tuesday.

Try this

Try it: move from the polished example to the awkward questions and inspect the provenance and limits beside every experiment.