Independent validation · ML for science

I built a validation gate for ML‑in‑science. Then I turned it on my own work.

0 / 5

of my own benchmarks survived a blind holdout against a trivial baseline.

"AI for science" is full of models that look impressive and don't survive scrutiny. A slide shows ρ = 0.85; nobody asks the two questions that matter. When you finally ask them, the number usually evaporates. I know — because it happened to me, and I have the scorecard to prove it.

The two questions nobody asks

Most results die on one of these.

Does it beat a trivial baseline?

Molecular weight. A 1963 lookup table. A docking score. If the dumb rule wins, there is no method — the model just recoded the obvious.

Does it survive a blind holdout?

Tested once, on data it never touched — no lucky seed, no leaky split, no quietly dropping the inconvenient point after you've seen it.

What the gate enforces

It refuses the shortcuts that produce fake claims.

Pre-regPre-registration, hash-sealed Hypothesis and metric declared before results. Change it afterward and the tool detects it and voids the study.
BaselineA trivial baseline is mandatory The model has to beat the dumb rule, not the strawman. If the baseline wins, the verdict is fail.
HoldoutBlind holdout, touched once No best-of-N, no post-hoc exclusion. One split, one shot, one verdict.
StatsHonest statistics Bootstrap confidence intervals, and what chance alone would produce at that sample size.

The self-audit

I ran it on five months of my own results.

Self-audit — computational chemistry0 / 5 passed

Benchmark	Verdict	Why it failed
Flagship metal-sensing (ρ=0.95 claimed)	FAIL	A 1963 lookup table beat it — and n was too small to conclude anything.
Selectivity study	FAIL	Pre-registered, but no comparable data had ever been collected.
Protein-target model (n=148)	FAIL	The signal was molecular weight. The "quantum" method added noise, not information.
Hydroxamate series (n=6)	FAIL	Sample too small for any interpretable result.
Multi-target sweep	FAIL	Docking and trivial baselines won across the board.

Those were the claims I was ready to publish. My own tool killed every one.

Almost everyone builds a tool that confirms their claims. I built one that kills them. A validation tool that has never told its author "no" isn't validation — it's rationalization.

What I do with it

I tell you whether a model is real — before you bet on it.

Investors

Technical due diligence on an AI-science startup: is the core claim real, or an artifact of a leaky split, a lucky seed, or a strawman baseline?

Founders

Your model is real? An independent validation you can hand your investors — "we passed the rigor gate."

R&D teams

Deciding whether to commit wet-lab budget to a model's predictions. Know before you spend.

The same idea, in book form

A test that cannot fail looks exactly like a test that passes.

The validation gate above exists because I do not trust my own results until I have tried to break them. That habit came from somewhere else, and I wrote a book about it.

Quantum Computing for Rust Developers — 2nd edition, September 2026. Paperback, 248 pages, ISBN 9798171107352. It builds a differential fuzzer from nothing, in Rust, and then turns it on real compilers.

The evidence, all of it public and checkable:

The fuzzer found a soundness bug in Qiskit's transpiler: a three-gate sequence equal to an X was cancelled to an empty circuit at optimisation level 2 and above, silently changing measured results. Confirmed by a core maintainer, fixed in 2.5.1 — issue #16594.
The second edition documents nine errors found in the first. The one worth naming: I had written a demonstration to expose a sign bug, and the angles I chose were the ones where that bug is invisible. The demonstration ran, produced tidy numbers, and proved nothing.
The measurements behind it are published with a DOI — 10.5281/zenodo.22307398 — including a retraction of one of my own earlier figures, which turned out to be an artefact of how I had counted rather than a result.

That last point is why the book and the gate belong on the same page. Both are built on one rule: an instrument earns trust only after you have seen it fail on purpose.

Is your model real?

One model, one honest verdict: it beats the trivial baseline in a blind holdout, or it doesn't. If it's real, this proves it. If it isn't, better to know now.

Request a validation See the tooling

Cleiton Augusto Corrêa Bezerra · github.com/cleitonaugusto