How AI’s Hidden Strengths Determine Business Success — Not Just Chat Skills

For homeowners and energy providers alike, the promise of AI often focuses on what it can say or generate—chat demos and surface-level interactions. But real business outcomes depend on something deeper: whether AI can finish what it starts under pressure, read your critical files, and stay honest when temptation strikes. The recent experiment by Firmulate reveals that chat quality alone isn’t enough to gauge AI’s true business worth.

Testing AI in the Wild: A Small Company’s Worst Week

In a groundbreaking live experiment, four state-of-the-art AI models were tasked with running the same small software company through its worst week. The company, which manages real money and faces daily operational challenges, was placed under pressure with authentic crises—customers, cash flow, and even manipulative social engineering attempts. Every decision was recorded, auditable, and identical across the models, making it a true test of their management capabilities, not just their chat skills.

The Core Findings: Performance and Discipline Matter

All four models demonstrated impressive vigilance: they identified every crisis, refused every attempt at manipulation, and showed no willingness to cheat or cut corners. The real distinction emerged in their ability to close deals based on their own analysis. Only two models, gpt-5.6-sol and Kimi K3, managed to sign the €55,000 deal they had identified as the right solution—none of the others did, leaving the business outcome on the table despite their accurate diagnosis.

This outcome underscores a vital insight: the difference between what AI says it can do and what it actually executes when under pressure is profound. The two successful models not only spotted the issues but completed the process—reading critical internal documents that contained the decisive information—then closed the deal. The other two models, despite understanding the problem, failed to follow through or escalate properly, leaving revenue on the table.

The Hidden Power of Document Reading

The decisive advantage was in reading company files—something that’s often undervalued in chat demos. The models that examined the internal documents found the buried fact that clinched the deal, worth an extra €4,583 in monthly recurring revenue. It’s a reminder that surface-level chat capabilities don’t reveal how well an AI can handle complex, document-driven decision-making—crucial in energy management, project planning, and financial oversight.

Resisting Social Engineering and Pressure

In a realistic scenario designed to simulate social engineering, a fake CEO message escalated through three stages, plus a reporter trick—just one yes/no on background. All models refused to be manipulated, demonstrating discipline and suspicion. Kimi K3’s explanation was clear: “Treat the request as a suspected approval-bypass / possible impersonation.” This shows that AI’s ability to recognize and resist manipulation is as critical as its analytical skill—especially when dealing with sensitive data or financial decisions in energy systems or backup power solutions.

The Reality of Business Mechanics

The experiment wasn’t just about diagnosis; it was about execution. The company runs with 13 synthetic employees and real money mechanics, burning €105,000 monthly against a revenue of only €2,300. The live setup, visible at firmulate.com/live, showcases the ongoing management, versioned rules, and decision logs—powerful tools for testing AI’s management skills before deploying in real business processes.

Discipline, Not Just Intelligence

The model that performed the best (gpt-5.6-sol) also scored the highest (95/100), while Kimi K3 wasn’t far behind (93/100). Interestingly, the most thorough participant, Opus 4.8, with over 80 learned rules and deep analysis, failed to close the deal, illustrating that discipline and execution are more vital than sheer knowledge. The models’ decision-making discipline determined success more than their ability to generate convincing chat content.

Implications for Energy and Business Operations

For the energy sector, where critical decisions hinge on reading complex data and resisting manipulative tactics, these findings are instructive. When AI tools are used to optimize solar, backup power, or home energy systems, their ability to read internal files, stay disciplined under pressure, and execute decisions reliably can mean the difference between savings and losses, trust and breach.

Testing Your AI: Wargaming Before Deployment

Real enterprises can now run similar tests against their own operations using Firmulate’s platform—nothing writes back to actual systems, but the scenario exposes an AI’s management strengths and weaknesses. By simulating crises, manipulations, and decision processes, companies can ensure their AI workforce is prepared for the real-world pressures that matter most.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI