BENCHMARK
What it measures, what it does not
Not a direct test of capability level
Biased toward tool-calling behavior
Checks for lazy early writes
Case-driven and customizable
These dimensions are not scored directly
Every bonus and deduction is explainable
CASES
The mistakes it catches
Acts before it understands the situation
Knows it is on the wrong path, but keeps going
Gets stuck in ambiguity
UPDATES
Update log
Recent changes that affect how this benchmark should be read.
- Removed Grok 4.1; added Gemini 3.5 Flash and GPT 5.5.
- Added DeepSeek V4 Pro and DeepSeek V4 Flash.
- Changed the update cadence to every 3 hours and removed Claude Sonnet 4.5 and Claude Opus 4.5.
- Added Claude Opus 4.7, removed OpenAI GPT 4.1 and Claude Haiku 4.5, added the new
search_texttool, and raised the difficulty. - Added two models: Google’s Gemma 4 31B and Xiaomi’s Mimo v2 Pro.
- Raised the difficulty because too many models were hitting full score.
- Launched the live LLMSnare benchmark.