tagged · 8 articles
evals
Articles
- 2026
- 2026
Bigger didn't break it. Knowing did.
Obfuscation dropped this decompiler's recovery from 18.8% of clean functions (28/149) to 0.0-0.9% of the functions each transform fired on (0/97, 0/63, 1/106) — gcc-lenient scorer, native 4096-token window, paired and placebo-controlled. Tripling its parameters bought no monotonic gain in the largest obfuscated cell: three points at n = 151, intervals overlapping. Fine-tuning it on the same transform family bought 12-16% back, and the headline moved from hardening to brittleness — against one modest 6.7B attacker, where every number is a floor. - 2026
Nobody published the denominator
AI-for-security benchmarks that score models on compiled code can report a number nobody can recompute. Of 37 papers reporting an obfuscation-strength number, exactly 1 conditions it on whether the transform actually fired — and that one is a disassembly paper, which scores nothing on re-executability. Our own do-nothing control passes the metric. - 2026
The compiler did it, and my obfuscation took the credit
Two arms of the same 149 functions, neither arm obfuscated — stock LLM4Decompile v1.5, gcc-lenient scorer, native 4096-token window: GCC-built binaries recover at 27.5%, clang-built at 15.4%, paired McNemar p = 0.0029. The confound is the toolchain, and the law generalizes well past binary analysis. The three-term decomposition built to separate those effects then fails to add up, and the ~5.6-point residual is printed open. - 2026
Green is not evidence
An experiment can validate the wrong quantity against the wrong envelope and pass, a self-test canary can fail identically whether the gate works or not, and an audit command can print nothing and exit 0 — three gate designs that read green while checking nothing. - 2026
One earthquake pinned my error bound forever
A seismic stream, a global magnitude bound wrecked by a single mainshock, a windowed bound that heals — and a scoreboard where the headline empirical legs of two papers report zero violations. - 2026
The error analysis everyone cites is for a kernel nobody runs
SCAMP and STUMPY accumulate a mean-centred covariance, not the textbook inner product — and the centred case is structurally different floating-point mathematics, not a special case at mu = 0. - 2026
Three axioms and one disclosed seam
Four verification tools, one certificate: every machine-checked theorem mechanically reduced to three classical axioms with zero unfinished proofs — and the one place two provers cannot talk to each other, named right in the theorem statements.