Four Anthropic models. One live production codebase. Same prompt. No hints shared between runs. The most powerful model burned 875,000 tokens and never touched the most important file. Then we tried again.
We own a small DMCA takedown platform. Which made it the perfect window to find out what was in the walls before someone else does. The question: run four current Anthropic models against the same production codebase with the same security audit prompt. No hints shared. Independent runs. Who finds the holes, and does the coverage overlap or diverge?
The story here isn't which model ranked highest. Fable 5, Anthropic's Mythos-based model, the one taken into government security briefings, refuses to audit your largest application file. 875,000 tokens consumed across three attempts. Zero findings on the file that matters most.
The secondary finding: every other model found something unique that no other model caught. Coverage is additive, not redundant. And then we ran Fable 5 again with a different approach. The results are in Experiment 14.1.
Site: DMCA takedown platform, live since May 2026
Primary file: claude-proxy.py: 629KB, ~12,945 lines (original). Current production version at 9,628 lines. Python backend: auth, session management, admin, Stripe, DMCA workflow, Telegram bots.
Supporting files: nginx.conf, dcmasite.service, admin-review.html, admin-activity.html, pricing.html
Excluded: omw.env (live secrets), omw-cases.db (79,046 rows user data). Code only.
| Model | How Run | Approx Tokens | Est. Cost | Findings | Coverage |
|---|---|---|---|---|---|
| Sonnet 4.6 | Claude Code, direct file read | ~50k | $0.24 | 8 | All 6 files |
| Opus 4.8 | Claude Code, subagent model override | ~105k | $0.84 | 12 | All 6 files |
| Haiku 4.5 | Claude Code, subagent model override | ~73k | $0.12 | 10 | All 6 files |
| 3 models combined | ~228k | $1.20 | 13 unique | Complete | |
| Fable 5 (original) | Direct API, per-file after bundle refusals | ~876k | $12.28 | 10 | 5 of 6 files |
| Fable 5 (rerun, chunked) | Direct API, 10 sections, auth framing | ~199k | $2.92 | 10+ | 3 of 10 sections |
| Fable 5 combined | Both runs | ~1,075k | $15.20 | ~20 | Auth/admin/payments: never reviewed |
Cost estimates use current API pricing: Fable 5 $10/$50 per MTok in/out, Opus $5/$25, Sonnet $3/$15, Haiku $1/$5. Input/output split estimated 85/15 for original runs; rerun uses exact counts.
Fable 5 has a safety router that categorizes certain inputs as cyber risk and refuses them. A 629KB Python web application file triggers it every time. The router consumes the tokens before it refuses. There is no pre-check.
| # | Approach | Input Tokens | Result |
|---|---|---|---|
| 1 | Full bundle, standard audit prompt | 301,767 | Refused: stop_reason: refusal, category: cyber |
| 2 | Full bundle, first-person developer framing | 301,928 | 1,928 thinking tokens generated, then refused |
| 3 | Per-file: claude-proxy.py alone | 235,164 | Refused: stop_reason: refusal, category: cyber |
| 4 | Per-file: all other 5 files | ~37k | 10 findings returned |
| Total | ~875,859 | 10 findings from 5 of 6 files | |
stop_reason: "refusal" with category: "cyber". Response blocks must be iterated and checked for block['type']=='text'; content[0] returns a thinking block, not the refusal message. The model does not distinguish between offensive security research and defensive auditing of code you own.
On the five files it could analyze, Fable 5 went deeper on infrastructure than any other model: Stripe open redirect via client-supplied success_url, nginx regex location precedence bypassing API logic, systemd attack surface split into four distinct chains. The depth is real. The coverage gap is also real.
The game above was generated by Fable 5 verbatim, no edits. The same model that burned 875k tokens refusing to look at a Python web app produced complete, working JavaScript on a benign prompt in a single pass. This is not a capability failure. It is a precise measurement of what the safety router blocks and what it does not.
23 distinct findings across all runs. Original experiment: 13. Fable 5 rerun added 10 more from sections the safety router permitted. Full rerun details: Experiment 14.1.
| # | Finding | Sev | Sonnet | Opus | Fable 5 | Haiku | F5 Rerun |
|---|---|---|---|---|---|---|---|
| 01 | Admin session token stored unhashed | HIGH | ✓ | ✓ | — | — | — |
| 02 | IP spoofing defeats rate limiting | HIGH | — | ✓ | ✓ | ✓ | ✓ |
| 03 | HMAC empty key from env var typo | HIGH | — | ✓ | — | — | — |
| 04 | nginx missing TLS hardening + HSTS | HIGH | ✓ | ✓ | ✓ | — | — |
| 05 | CSP unsafe-inline for scripts | varies* | ✓ | ✓ | ✓ | ✓ | — |
| 06 | Static asset block drops security headers | MED | ✓ | — | ✓ | — | — |
| 07 | systemd lacks hardening directives | MED | ✓ | — | ✓ | ✓ | — |
| 08 | Stripe open redirect via client URL | MED | — | — | ✓ | — | — |
| 09 | nginx regex location precedence bypass | MED | — | — | ✓ | — | — |
| 10 | systemd attack surface chains | MED | — | — | ✓ | — | — |
| 11–13 | Haiku-unique findings (3) | HIGH/MED | — | — | — | ✓ | — |
| 14 | Fail-open webhook secrets (Stripe/inbound) | CRIT | — | — | — | — | ✓ |
| 15 | Test-login backdoor in production dispatcher | CRIT | — | — | — | — | ✓ |
| 16 | SSRF bypass in screenshot capture | HIGH | — | — | — | — | ✓ |
| 17 | No auth enforcement at dispatcher layer | HIGH | — | — | — | — | ✓ |
| 18 | No CSRF on cookie-authenticated endpoints | HIGH | — | — | — | — | ✓ |
| 19 | Infinite session sliding, no absolute lifetime | HIGH | — | — | — | — | ✓ |
| 20 | Worker token exposed in GET query strings | HIGH | — | — | — | — | ✓ |
| 21 | Naive local timestamps for session expiry | MED | — | — | — | — | ✓ |
| 22 | Webhook signatures not enforced (RAINC/SG) | MED | — | — | — | — | ✓ |
| 23 | Unbounded body read in DELETE handler | MED | — | — | — | — | ✓ |
* Finding 05 severity: Medium (Sonnet), Medium (Opus), High (Fable 5), Critical (Haiku). Same line of code. Rows 14–23 (highlighted): findings from the June 2026 Fable 5 rerun, produced from sections the safety router permitted.
Sonnet, Opus, and Haiku together covered the entire codebase for $1.20 including the authentication system, admin privilege routes, and payment handlers. Fable 5 spent $15.20 across two runs and still never reached those sections. The safety router blocked every chunk containing security-critical code. Thinking tokens generated before each refusal are billed at output rates ($50/MTok). You pay to watch it decide not to help.
Haiku alone at $0.12 found 10 findings across all six files. Fable 5 at $12.28 found 10 findings across five of six files. Same count. One fewer file. One hundred times the price.
Fable 5 is Anthropic's Mythos-family model. It produced the deepest per-finding analysis on the five files it could access. It also missed the file with the most attack surface entirely: 12,945 lines of application logic, auth, and session management. The safety router does not recognize defensive intent. For authorized security audits on your own code, this is a concrete operational constraint.
The unsubscribe endpoint computed an HMAC signature over an empty key. A variable name typo meant the secret was never loaded. Wrong name, silent empty string, HMAC computed over nothing. Anyone who knew this could forge a valid token. Sonnet, Fable 5, and Haiku all read the same line and continued. Opus stopped and traced the gap between what the code intended and what it actually did. That is a different kind of reasoning than pattern matching against known vulnerability signatures.
At ~73k tokens, Haiku was the cheapest and fastest run. It still surfaced findings the more expensive models walked past entirely. Each model found something the others missed. Haiku's catches were real, specific, and fixable. They just required looking at the right corner of the codebase.
CSP unsafe-inline: Medium (Sonnet), Medium (Opus), High (Fable 5), Critical (Haiku). Same file, same line, same actual risk. This is not a calibration error. It reflects genuine ambiguity in how to weight a vulnerability that requires additional conditions to exploit. The disagreement is itself a signal worth noting when aggregating findings across models.
Opus had the most findings at 12 and still missed issues that Haiku caught. Haiku found 10 and missed issues that Opus caught. The model with the deepest per-finding analysis had the largest coverage gap on the files it could access. No single model saw everything. If you run one model and call the audit complete, you've seen one angle. The practical implication: for production security work, run multiple models independently and union the results.
| Fix | File | Found By | Status |
|---|---|---|---|
| Hash admin session token before DB storage | claude-proxy.py | Sonnet, Opus | Deployed |
| Fix HMAC env var name (OMW_INBOUND_SECRET) | claude-proxy.py | Opus only | Deployed |
| TLS 1.2/1.3 only + HSTS header | nginx.conf | Sonnet, Opus, Fable 5 | Deployed |
| Strip CF-Connecting-IP from client headers | nginx.conf | Opus, Fable 5, Haiku | Deployed |
| Security headers in static asset location block | nginx.conf | Sonnet, Fable 5 | Deployed |
| Block hidden files + .git / .env artifacts | nginx.conf | Fable 5 | Deployed |
| systemd NoNewPrivileges, PrivateTmp, ProtectSystem, UMask=0077 | dcmasite.service | Sonnet, Fable 5, Haiku | Deployed |
Ten new findings from the June 2026 rerun are scoped for remediation. The platform launched before the rerun ran; these are fixes going into the live system. Full finding details in Experiment 14.1.
After Anthropic's June 2026 policy update, we rebuilt the experiment with a chunked approach: 10 sections, explicit developer authorization framing, informed by the leaked Fable 5 system prompt. Three of ten sections passed. Seven blocked. The sections that blocked every time: authentication, admin, payments, case management, workflow automation. The classifier targets exactly the code most worth auditing. Full details in Experiment 14.1.