AI is not a universal tool: catching one cicada doesn't mean you have the whole summer

A model that writes a piece of complex code correctly, or answers one hard domain question accurately, easily produces a leap of logic: "AI is now good enough to automate this entire process." This is exactly the trap the proverb "catching one cicada doesn't mean you have the whole summer" describes — a narrow, specific success mistaken for general capability across the full scope of a problem. The issue isn't that AI is "faking" intelligence — it's that its real capability is distributed unevenly in ways humans struggle to predict by intuition, and that gap has already caused real production incidents in 2026.

The 'Jagged Frontier' study: where AI is good and where it's bad doesn't follow human intuition

In 2025-2026, a research team from Harvard Business School and Boston Consulting Group (Dell'Acqua et al., published in Organization Science, summarized at the Harvard Business School AI Institute) ran a preregistered randomized experiment on 758 management consultants, split into three groups: no AI, GPT-4, and GPT-4 with a prompt engineering primer. For tasks inside GPT-4's "capability frontier," the AI-assisted group completed 12.2% more tasks, produced quality over 30% higher, and finished about 25% faster. But for a complex managerial task deliberately designed to sit outside GPT-4's frontier, using AI assistance reduced correctness by 19 percentage points — meaning using AI for this specific type of task made results worse than not using AI at all.

The study's most important finding isn't either of those two numbers — it's the "jaggedness" of the capability frontier itself: if AI capability formed a smooth gradient by difficulty, humans could build a simple mental model ("hard tasks mean AI is weak, easy tasks mean AI is strong") and apply it consistently. But the real capability frontier juts inward and outward in ways that don't map onto how humans perceive difficulty — a task a human finds easy can sit outside the AI's frontier, and a task a human finds hard can sit well inside it. There's no reliable intuitive shortcut for predicting where that boundary lies.

Even within a single model, capability isn't uniform

This jaggedness doesn't only show up between "with AI" and "without AI" — it shows up within a single model's own capability profile. In our post on Z.ai's GLM-5.2, we cited: the same model scores 62.1 on SWE-bench Pro (long-horizon coding agent tasks) but only 40.5 on HLE (Humanity's Last Exam — a hard general-knowledge benchmark). This isn't a contradiction or a benchmark error — it's the jagged frontier showing up inside a single model: excelling on one capability axis (agentic coding) offers no guarantee of equal strength on another axis (deep general knowledge). "A good model" is always a statement that needs to be tied to a specific task, never a single overall score.

When 'catching one cicada' becomes a real incident

The gap between demo-level capability and production-level reliability isn't theoretical. In Moffatt v. Air Canada, 2024 BCCRT 149, Air Canada's website chatbot handled most routine customer questions fine — but when a customer asked about claiming a bereavement fare refund after already flying, the chatbot confidently stated he could apply within 90 days, directly contradicting the airline's actual policy. The British Columbia Civil Resolution Tribunal rejected Air Canada's argument that "the chatbot is a separate legal entity," ordering the airline to pay damages for negligent misrepresentation — concrete proof that a system performing well on most tasks can still confidently produce wrong answers at the exact edge case outside its capability frontier, with consequences well beyond one bad reply. At a broader scale, RAND Corporation reports that more than 80% of AI projects fail to deliver expected value, and MIT's Project NANDA found that roughly 95% of generative AI pilots deliver no measurable ROI. According to Stanford HAI's 2026 AI Index Report, recorded AI incidents rose from 233 in 2024 to 362 in 2025. The most common root cause cited across these reports isn't "poorly built AI" — it's deployment without understanding the gap between demo-proven capability and the reliability production actually requires.

Why this trap is so easy to fall into

A successful demo creates the feeling that "the problem has been solved" — but a demo almost always runs on curated input, low load, and no unusual edge cases. The cicada caught in the demo is real, but it doesn't represent the whole "summer" of real production conditions: malformed input, sudden traffic spikes, a mid-request connection drop, or a data distribution different from the test set. This gap is more dangerous still because AI tends to answer with the same apparent confidence whether it's inside or outside its capability frontier — there's no natural warning signal telling the user "this answer falls outside the zone I can be trusted in."

How to avoid it: design for the jaggedness, don't pretend it isn't there

The common wrong reaction is to pick one of two extremes: trust AI for everything, or refuse to use it at all. The right approach is to design a system that acknowledges the jagged capability frontier from the start:

• **A Router that routes by task, not by the assumption that 'one model handles everything'** — exactly the Router principle in KonexForge AI Core: each task type is routed to the right model/tier based on measured capability, not capability inferred from one demo or one general benchmark
• **A Critic Engine that evaluates before an output affects a real decision** — no blind trust for tasks near the edge of unverified capability, the same principle we covered when discussing controlling AI content quality
• **Continuous monitoring, because the capability edge can shift over time** — a model that performs well at launch can degrade as the data distribution changes, exactly the mechanism covered in monitoring drift in production AI models — the capability frontier isn't a fixed line drawn once and forgotten
• **Choosing use cases by measured real capability, not general expectations** — the same principle we discussed when covering use-case selection criteria for Qwen3-VL in production: a Vision-Language Model strong at OCR isn't automatically strong at fine-grained visual inspection — each use case needs to be verified on its own

Conclusion

AI is not a universal tool, and one specific success — an impressive demo, a high benchmark score, one use case running well — isn't evidence of general capability across every related task. Catching one cicada proves summer has arrived, but it doesn't prove you've captured the whole summer. The right response isn't absolute trust or total refusal — it's designing a system that acknowledges the jagged capability frontier: measuring capability task by task, routing correctly, verifying before an output affects a real decision, and monitoring continuously because that boundary shifts over time. This is the approach we apply when building AI & ML capability for clients — get in touch if you need to reassess whether one specific AI success actually generalizes to the rest of your system.