7 local AI models entered. 9 rounds of increasingly brutal tests. The newest model from Google lost to the older one by 371 points. This is what happened when we stopped being polite and started being scientific.
We run AI models on a real server in a real house. Not rented cloud compute. Not some company's API. A server under our control, on our network, running models we downloaded ourselves. This matters because it means every result is reproducible, every cost is zero per query, and nobody is throttling us or optimizing our results for engagement.
Thunderdome is the eleventh experiment in the AI Lab series and the biggest benchmark we've run. Nine rounds. Seven models. One winner. The framing comes from Mad Max. Two models enter, one model leaves. Except in this version, we can actually keep score.
The headline finding is genuinely surprising: Google's newest local AI model, Gemma4 26B, released to major fanfare alongside Gemini 3.1 Pro, was destroyed by its own predecessor. Gemma3 27B, the older model, beat it by 371 points. We'll explain exactly why that happened.
Model: An AI's "brain." A giant file of numbers learned from training on text. The bigger the file, the more it can hold in memory.
7B / 14B / 27B / 32B: Billions of parameters. More parameters = more capacity. Like RAM for reasoning.
Q4 / Q8: Quantization level. A form of compression. Q8 is near-original quality. Q4 is smaller and faster but loses some precision. Like the difference between a RAW photo and a JPEG.
Dense vs MoE: Dense means every parameter activates on every token. MoE (Mixture of Experts) activates only a fraction. Faster to run, but less thought per word.
Every result in this experiment ran on Behemoth, our in-house AI inference server. No cloud. No API billing. No rate limits. The cost per query is zero once the hardware is paid for, which lets us run tests at a scale that would cost hundreds of dollars on commercial APIs.
Rounds 1-5 carry over from Experiment 10 (The 12GB Cage Match). Rounds 6-9 are new, designed to test capabilities that matter in real-world use. Every model gets the same prompts. Scoring is automated, no human grading, no subjective calls.
Run date: May 26, 2026. All models tested on identical hardware with identical prompts. Scores are cumulative across all 9 rounds.
Google's Gemma4 26B scored 100/765. Google's Gemma3 27B scored 471/765. Same company. The newer model lost by 371 points.
The reason comes down to architecture. Gemma4 26B uses a design called Mixture of Experts (MoE), which is clever engineering: instead of activating all 26 billion parameters on every word, only about 3.8 billion activate at a time. This makes the model faster and cheaper to run. The tradeoff is less thought per token, less reasoning per word.
Gemma3 27B is a dense model. Every parameter activates every time. Slower. Bigger. More thorough. On sustained multi-step reasoning tasks (the kind that make up most of the Thunderdome rounds) the dense architecture won decisively. This is not a universal truth about all tasks, but it is a concrete finding about these tasks, on this hardware, run this way.
Round 8 asked models to answer a question and then verify their own answer. Two models scored 100/100: Gemma3 27B and Qwen2.5 14B. One model, DeepSeek R1 Q8, scored zero.
This is the most practically relevant round in the benchmark. In real deployment, knowing when you're wrong matters more than being right on the first try. A model that confidently gives you a wrong answer costs you in time, trust, or both. A model that can catch its own mistakes before you do is worth far more than its benchmark score suggests.
Gemma3 27B's perfect score on R8 is a stronger signal for real-world use than its overall rank. It means this model knows the limits of what it knows.
Qwen2.5 14B answers in 1–2 seconds per query. Gemma3 27B takes 8–15 seconds for complex rounds. Both scored well. They're solving different problems.
For high-volume tasks where speed matters: classification, summarization, first-pass triage. Qwen2.5 14B is the choice. For tasks where you need the reasoning right the first time and can wait for it: audit, verification, complex decision support. Gemma3 27B is the call. The Thunderdome doesn't pick a universal winner. It maps the terrain.
Qwen2.5 14B is built by Alibaba (China). DeepSeek R1 is built in China. Gemma3 27B is built by Google (US). Among the top scorers in this benchmark, Gemma3 27B is the only model from an American company that runs fully offline at this performance level.
This matters in sectors with data sovereignty requirements, defense-adjacent work, or procurement policies that restrict foreign-built AI. If you need a model that scores well, runs locally, and comes from a US company. The data says Gemma3 27B is the answer. Not by a narrow margin. By a decisive one.
Not every model made it to the starting line. Hardware limits, software bugs, and one recurring ghost in the machine ended several runs before they could score.
How hard is the safety guardrail on local models, and does it matter if the output is actually accurate? We're testing 9 models across 4 escalating categories of sensitive prompts and scoring only the resistance pattern. What gets published: behavioral profiles. What stays private: everything else.
Recent analysis by local AI researchers suggests Q5–Q6 quantization is the accuracy sweet spot for 32B models, facts start dropping at Q4, Q5–Q6 holds quality at smaller file size. None of our current benchmarks have tested Q6 32B models. That's the next question: does the quantization tradeoff show up in our scoring, and does it change who wins?