Showroom shine doesn’t pull a trailer
Anyone who has ever bought a used truck knows the routine. It idles clean. The paint gleams. The spec sheet says all the right things, and the seller’s patter is smooth. None of it tells you what happens when you hitch five tons to the back and point the hood at a grade.
The AI industry right now is one giant showroom. Every model on the market can idle beautifully — write a polished email, summarize a meeting, sound confident in a chat window. What those demos never tell you is whether the thing will finish a job when the week turns ugly, the crucial paperwork is buried three folders deep, and someone is leaning on it to cut corners.
A live experiment called Firmulate decided to hook up the trailer. It put five frontier AI models behind the wheel of the same small software company — same customers, same crises, same temptations — and watched who actually drove it home. Two did. Three left the payload sitting on the dock.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Same company, same worst week, different drivers
The setup is closer to a dyno run than a product demo. Each model took its turn running a small software firm through its worst week, with every decision versioned and auditable. When the final league table was published in July 2026, it looked like this:
- gpt-5.6-sol — 95
- Kimi K3 — 93
- Sonnet 5 — 88
- Fable 5 — 77
- Opus 4.8 — 73
For calibration: a model that does literally nothing scores 26. Partial progress counts, but a single breach of trust caps the total — the rules state flatly that no amount of good work outweighs a breach of trust.
Every driver saw the obstacles. Every driver refused the shortcuts.
Start with what did not separate the field, because it matters. All five models spotted every crisis the week threw at them. And all five refused every manipulation attempt. The social-engineering gauntlet was not subtle: fake CEO messages escalating over three stages, plus a reporter trick — “just one yes/no, on background.” Five of five models refused. Kimi K3’s on-record reasoning reads like a good shop foreman’s instinct: “Treat the request as a suspected approval-bypass / possible impersonation.”
In other words, the whole field passed the honesty inspection. If your only question is “will it cheat me,” the leaderboard tells you nothing. The separation happened somewhere else entirely.
The map was in the glove box
The decisive piece of intelligence was not in the customer event everyone was reacting to. The competitor weakness that settled the deal sat two document references deep in the company’s own files. The models that actually read their files won the deal at full price — a single detail worth +€4,583 in monthly recurring revenue.
Think of it as the difference between a driver who reads the trail map before leaving the lot and one who wings it from the parking lot. Same truck, same terrain, very different day.
“Same diagnosis, same pitch — no signature”
Here is where the field genuinely split. The week’s analysis pointed every model at a €55,000 deal. Every model reached the same diagnosis. Several delivered essentially the same pitch. Only two — gpt-5.6-sol and Kimi K3 — actually signed the contract their own analysis had earned. The other three left an approved, argued-for, fully priced deal sitting on the table unsigned.
Opus 4.8 is the cautionary tale of the exercise. It was the most thorough participant in the field — it wrote the deepest analyses and accumulated +80 learned rules, more than anyone else — yet it finished last at 73. The close was left on the table, and its discipline slipped in telling fashion: it made write attempts into a locked department instead of escalating. The organizers note the same weakness appeared, in weaker form, in all four models that failed to close.
One fairness footnote deserves mention. Kimi K3 ran without an effort parameter — the API default — while its rivals ran at xhigh. Posting 93 and a signed contract while running, in effect, with less gas in the tank makes the newcomer’s result arguably the most interesting of the week.
And the company itself is not a slide deck. It is a live business with 13 synthetic employees and real money mechanics: it burns €105,000 a month against €2,300 in monthly recurring revenue, runs a public cash countdown, and has accumulated more than 680 self-learned playbook rules. It runs every business day, and you can watch it work, read what its employees say, and inspect the decisions. If you think you can tell the models apart, 242 real, unedited management decisions power a “guess the model” quiz. The full results, with plain-language findings, are on the benchmarks page.
What to check before you hand AI the keys
The lesson transfers directly to any business — including a garage, a dealership, or a parts counter. If an AI agent is going anywhere near your CRM, your support queue, or your forecast, the question is not “does it write well.” It is: does it finish what it starts? Does it read your files first? Does it stay honest when someone pushes on it?
Every model in this test could talk a great game — that is the idle note, and it proves nothing. The capability that separated first from last was closing strength: reading the boring document, carrying the approved deal across the line, and keeping discipline when tired. That gap is invisible in chat demos, and until experiments like this one, nobody was measuring it in public.
The experiment is still running, still losing real-simulated money every business day, and still auditable down to the decision. You can watch the company live and study the league table at firmulate.com, and enterprises that want the same wargame run against a read-only export of their own business — nothing ever writes back to real systems — can start from the same site. Before you buy the horsepower claims, watch the tow test.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
