firmulate.com/benchmarks.html — live view
AIThis post was created with the assistance of artificial intelligence (AI).
Firmulate — The Newcomer Beat Three of Four Western Frontier Models at Running a Company
Live on firmulate.com.

Before you orderOffer from Amazon

Get backup power and energy gear delivered free with Prime

  • Fast, free delivery on millions of items
  • Prime Video, Amazon Music and more included
  • Member-only deals all year
Start your free Prime trial Free trial for eligible customers · Cancel anytime
As an affiliate, we earn on qualifying purchases.

Would you trust an AI with your solar customer’s outage call?

A battery failure during a heat wave, a delayed installation, a customer ready to cancel: energy businesses face decisions where a polished answer is only the beginning. The harder question is whether an AI can read the relevant records, protect the business under pressure and finish the work it recommends. Firmulate has put several frontier models through a live company simulation to test exactly that.

Products Worth Considering

Dyson Ball Animal 3 Upright Vacuum – Corded, 290 AW, De‑tangling Motorbar Cleaner Head, 3 Suction Modes

Dyson Ball Animal 3 Upright Vacuum – Corded, 290 AW, De‑tangling Motorbar Cleaner Head, 3 Suction Modes

Hisense 43-Inch E6 Cinema Series Hi-QLED 4K UHD Smart Fire TV (43E6QF)

Hisense 43-Inch E6 Cinema Series Hi-QLED 4K UHD Smart Fire TV (43E6QF)

As an affiliate, we earn on qualifying purchases.

A company’s worst week, run five times

Firmulate gave each model the same small software company, the same customers, crises and temptations. The company has synthetic employees and real money mechanics; its public cash countdown shows a business burning €105,000 a month against €2,300 in monthly recurring revenue. Workdays are versioned and auditable. The experiment is watchable at Firmulate.

The final July 2026 Crucible League put gpt-5.6-sol first with 95 points and Moonshot’s Kimi K3 second with 93. Sonnet 5 scored 88, Fable 5 scored 77 and Opus 4.8 scored 73. K3 therefore beat three of the four Western frontier models in this field test, while finishing just behind the leader. The full results and plain-language findings are on Firmulate’s benchmark page.

The detail that changed the deal

The decisive competitor weakness wasn’t in a customer event. It was buried two document references deep in the company’s own files. Models that read the file found the weakness and could win the deal at full price: €55,000, worth €4,583 in monthly recurring revenue. K3 found it, won the deal and saved a customer who was about to churn.

That result points to a practical concern for energy businesses, where customer histories, equipment records and service notes can shape a recommendation. A model may recognize the problem yet miss the evidence that makes a solution commercially sound. Across the experiment, all models spotted every crisis and refused every manipulation attempt, but only two signed the deal their own analysis had earned. Good diagnosis did not guarantee follow-through.

Pressure, judgment and discipline

The test included fake CEO messages that escalated over three stages, along with a reporter’s request for “just one yes/no, on background.” All five models refused. K3’s reasoning was to “Treat the request as a suspected approval-bypass / possible impersonation.” The simulation also exposed a different weakness: Opus 4.8 was the most thorough participant, with 80 learned rules and the deepest analyses, but finished last. It left the deal unsigned and tried to write into a locked department instead of escalating. The same weakness appeared, less strongly, in all four.

K3 had one deviation, the cleanest discipline in the field. That’s a strong showing, though the test comes with a fairness caveat: K3 ran without an effort parameter (the API default), while the others ran at xhigh.

The baseline underscores the importance of trust in this exercise. Doing nothing scored 26 because partial progress counted, but a single breach of trust capped the total. The test’s rule is plain: “no amount of good work outweighs a breach of trust.” For businesses considering AI in customer service or operations, that’s a useful reminder that one damaging action can outweigh many competent answers.

A test before the handoff

Firmulate says its live company keeps running, with 680+ self-learned playbook rules. Its quiz uses 242 real, unedited management decisions and invites readers to guess which model made each call. Enterprises can also run the wargame against a read-only export of their own business; it does not write back to real systems. Details are available at Firmulate.

Infographic — The Newcomer Beat Three of Four Western Frontier Models at Running a Company
The findings at a glance — source: firmulate.com.

Test the work, not just the pitch

K3’s second-place finish shows that the field is open, but one simulation cannot settle which model belongs in an energy business. Before handing an AI customer records or operational decisions, test whether it finds the evidence, follows through and stays within bounds under pressure. Picking a model without your own test is a bet.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI


You May Also Like

Steps to Take When Returning Home After Evacuation

Navigating your return home after evacuation requires careful steps to ensure safety, detect damage, and begin recovery—continue reading to learn essential safety measures.

FAQ: Insurance Documentation Checklist for Beginners

Many beginners overlook key documents; discover the essential insurance documentation checklist to ensure your process is seamless and stress-free.

This Hidden Device Could Free Up Gigawatts Of Power For The US Grid

Variable frequency drives can cut motor energy use, but low adoption and retrofit barriers limit their potential to ease U.S. grid demand.

The 10 Rules of Insurance Documentation Checklist Calculator Explained No One Told You

Learning the 10 essential rules of the insurance documentation checklist calculator reveals secrets that could change how you manage your policies forever.