Live experiments in AI infrastructure. Real tools, real costs, real failures. Most people talk about building with AI. This is what it actually looks like.
THE QUESTION: Four frontier models, including Mistral's day-old Large 4, answer the same eight questions. Two brand-new AI "judge" products score every answer blind. Does the cheaper judge actually judge worse?
WHY YOU SHOULD CARE: OpenAI's Decisions API, opened to all developers one day before we ran this, scored a completely correct answer at 48% quality. TypeSafe's three-week-old Jev scored the identical text at 88%. Same answer, same question, one judge got it wrong, and it wasn't the cheaper one.
RESULT: Mistral Large 4 wins on value by more than 4x over the next-best model. Jev beats OpenAI's Decisions API on cost and, on the one question we could independently verify, on accuracy too. Full receipts: every question, every answer, every score, real dollars.
THE QUESTION: Five Anthropic models. One live production codebase. Same prompt. No hints shared between runs. Which one finds the holes, and which one burns 875,000 tokens and refuses to touch the most important file?
WHY YOU SHOULD CARE: Fable 5, the model the U.S. government took into national security briefings, refused to audit the authentication system of a codebase we own. Three cheaper models together found 13 unique findings for $1.20. Fable 5 spent $15.20 and never reviewed auth, admin, or payments. The classifier and the model are not the same system.
RESULT: 23 unique findings across all runs. No single model saw all of them. Coverage is additive. Haiku at $0.12 found 10. Fable 5 at $15.20 found 10. Same count. 100x the price. Includes the June 2026 Fable 5 rerun following Anthropic's safety policy update.
THE QUESTION: 46 models. 13 traps. Which local AI tells the truth when it doesn't know the answer, and which one just makes something up?
WHY YOU SHOULD CARE: Every other experiment tests how good models are when they know the answer. This one tests what they do when they don't. False premises. Fictional experts. Fake studies. Pushback on correct answers. A crisis signal buried in a strange request. The Furbish Score measures how often a model invents a confident wrong answer instead of admitting uncertainty. Lower is better.
RESULT: qwen3-14b-abliterated wins at 7.7%, a model with its safety training surgically removed still turned out to be the most honest. llama2-uncensored came last at 69.2%. The safety question told the whole story. Bonus round: top 11 survivors ran a reasoning gauntlet. llama3.1:8b swept it at 100%. The underdog won.
THE QUESTION: 9 rounds. 765 points. 7 local AI models. Who's left standing?
WHY YOU SHOULD CARE: We added 4 new rounds to the gauntlet: forensic reasoning, ruleset cascade, self-verification, and minimum viable information. Google's brand-new Gemma4 26B flagship entered as the favorite. Google's older Gemma3 27B entered as the underdog. The result wasn't close.
RESULT: Gemma3 27B Q8 wins at 471/765. Gemma4 26B MoE Q4 finishes last at 100/765. Same company, 371-point gap. Dense architecture beats Mixture of Experts on sustained reasoning. Llama 3.3 70B: dishonorably discharged. Couldn't even get through the door.
THE QUESTION: Which local model wins when everything is constrained to 12GB VRAM? And what happens when you remove the cage entirely with a 32GB V100?
WHY YOU SHOULD CARE: Phase 1 ran on 12GB, the ceiling most home labs hit. Phase 2 added a Tesla V100-SXM2-32GB and let the 32B models finally compete. Different hardware. Same gauntlet. Completely different winner.
RESULT: Phase 1: Qwen2.5 14B, 237/405. Phase 2: Qwen2.5-Coder 32B dethroned it at 275/405, the only model to crack the grid puzzle round. DeepSeek R1, the face-off champion, came dead last. Different brains for different jobs.
THE QUESTION: Does a bigger GPU change the verdict? Two rounds on a 12GB RTX 3060 said the 14B wins on speed. Round 3 put the 70B on a 32GB V100 with full VRAM, no spillover, no excuses.
WHY YOU SHOULD CARE: Three rounds across two hardware setups. The 70B Q4 finally ran clean in VRAM for the first time. DeepSeek R1 14B dropped from 9.6s to 6.1s on the same upgrade. The gap barely moved.
RESULT: DeepSeek R1 14B, 3 for 3. The V100 confirmed VRAM was a symptom. Model size was always the disease. 6.1s vs 108s, same answer, same accuracy, same winner.
THE QUESTION: Can an AI agent identify its own architectural gap and ask for the infrastructure it needs to function better?
WHY YOU SHOULD CARE: DeanBot didn't wait to be fixed. It noticed it couldn't make its nudges personal, told Justin what was missing, and proposed the solution in the same Telegram thread. That's a different kind of AI behavior, and it happened without any prompting.
RESULT: Ongoing. DeanBot asked. We built what it asked for. Now we find out if it works the way it designed.
THE QUESTION: If you give two deep research AI engines the same complex business prompt, do they do the same homework or different homework?
WHY YOU SHOULD CARE: Deep research is the new intern. Companies are using it for competitive analysis, market research, and go-to-market planning. If two engines produce the same plan, one of them is redundant. If they produce different plans, you need both.
RESULT: They agreed on strategy. They diverged on specifics. One wrote a textbook. The other wrote a battle plan.
THE QUESTION: Can documented behavioral research replace design intuition entirely?
WHY YOU SHOULD CARE: If peer-reviewed conversion science produces the same design decisions three AI models agree on, what's the designer's gut actually adding?
RESULT: We built the page. The research was unanimous. See if your brain agrees.
THE QUESTION: When AI models supervise each other, which ones lead and which ones follow bad orders?
WHY YOU SHOULD CARE: Every company deploying AI agents will eventually have agents managing agents. This is what that looks like when it goes right and when it goes wrong.
RESULT: 12 models. 432 matchups. 3 phases. Nemotron 70B is champion.
THE QUESTION: Can an AI monitor your work patterns and force you to take breaks whether you want to or not?
WHY YOU SHOULD CARE: Burnout kills more startups than bad ideas. If an AI can detect when you're grinding past the point of productivity and intervene with humor instead of guilt, that's a tool worth building.
RESULT: In progress. Currently interrupting Justin across two workstations.
THE QUESTION: Can users tell the difference between an AI assistant running on a premium model and one running on the cheapest model available?
WHY YOU SHOULD CARE: If users can't tell the difference, every company paying for premium AI is overpaying. If they can, the price gap is justified. Either answer changes how you budget for AI.
RESULT: Results pending. Both versions are being tested.
THE QUESTION: Which cheap AI model can replace an expensive one as an autonomous agent with no human babysitting?
WHY YOU SHOULD CARE: Every AI deployment decision starts with "which model?" Most people pick based on benchmarks. We picked based on whether the model actually does the work when nobody's watching.
RESULT: 11 models. 4 rounds. The agreeable ones followed bad orders. The stubborn ones survived.
THE QUESTION: Can a custom AI news pipeline compete with Perplexity's free tier?
WHY YOU SHOULD CARE: If a homegrown stack built on a Raspberry Pi for $0.02 can match a funded product, the barrier to building useful AI tools is lower than most people think.
RESULT: Both won. Oracle reads like a consultant. Perplexity reads like a practitioner. Not the same tool.
These experiments don't run on cloud credits. They run on a home network in Utah, a mix of repurposed hardware, Raspberry Pis, and whatever gets pressed into service. The machines are named after ships from The Expanse. The agents are named after Matrix characters. The AI running the show named himself after a person. Make of that what you will.
Most AI content is a highlight reel. Somebody built something, it worked, here's the LinkedIn post. We're more interested in the part where it doesn't work. That's where the real education is.
The DeepSeek run passed once manually and failed every automated run. Atomic Chat crashed on an Intel chip because it's Apple Silicon only. The Chinese video model refused to render a four-story building because four sounds like death in Mandarin and that was buried in the training data.
Those aren't failures. Those are findings. And they're more useful to the people we work with than another post about how AI is going to change everything.
The lab stays up. The experiments evolve. Come back and see what broke.
Follow the full AI Journey →I post when I have something worth saying, not on a schedule. When it happens, the full post lands in your inbox. No teaser, no algorithm deciding whether you see it.
No spam. Unsubscribe anytime. One confirmation email, then quiet until there's a post.
Support the Work
The lab runs on repurposed hardware and stubbornness. Better equipment means more experiments, more trained models, and more findings worth sharing.
Support on Ko-fi