Independent validation · ML for science
I built a validation gate for ML‑in‑science. Then I turned it on my own work.
"AI for science" is full of models that look impressive and don't survive scrutiny. A slide shows ρ = 0.85; nobody asks the two questions that matter. When you finally ask them, the number usually evaporates. I know — because it happened to me, and I have the scorecard to prove it.
The two questions nobody asks
Most results die on one of these.
Does it beat a trivial baseline?
Molecular weight. A 1963 lookup table. A docking score. If the dumb rule wins, there is no method — the model just recoded the obvious.
Does it survive a blind holdout?
Tested once, on data it never touched — no lucky seed, no leaky split, no quietly dropping the inconvenient point after you've seen it.
What the gate enforces
It refuses the shortcuts that produce fake claims.
- Pre-regPre-registration, hash-sealed Hypothesis and metric declared before results. Change it afterward and the tool detects it and voids the study.
- BaselineA trivial baseline is mandatory The model has to beat the dumb rule, not the strawman. If the baseline wins, the verdict is fail.
- HoldoutBlind holdout, touched once No best-of-N, no post-hoc exclusion. One split, one shot, one verdict.
- StatsHonest statistics Bootstrap confidence intervals, and what chance alone would produce at that sample size.
The self-audit
I ran it on five months of my own results.
| Benchmark | Verdict | Why it failed |
|---|---|---|
| Flagship metal-sensing (ρ=0.95 claimed) | FAIL | A 1963 lookup table beat it — and n was too small to conclude anything. |
| Selectivity study | FAIL | Pre-registered, but no comparable data had ever been collected. |
| Protein-target model (n=148) | FAIL | The signal was molecular weight. The "quantum" method added noise, not information. |
| Hydroxamate series (n=6) | FAIL | Sample too small for any interpretable result. |
| Multi-target sweep | FAIL | Docking and trivial baselines won across the board. |
Those were the claims I was ready to publish. My own tool killed every one.
Almost everyone builds a tool that confirms their claims. I built one that kills them. A validation tool that has never told its author "no" isn't validation — it's rationalization.
What I do with it
I tell you whether a model is real — before you bet on it.
Technical due diligence on an AI-science startup: is the core claim real, or an artifact of a leaky split, a lucky seed, or a strawman baseline?
Your model is real? An independent validation you can hand your investors — "we passed the rigor gate."
Deciding whether to commit wet-lab budget to a model's predictions. Know before you spend.
Is your model real?
One model, one honest verdict: it beats the trivial baseline in a blind holdout, or it doesn't. If it's real, this proves it. If it isn't, better to know now.
Cleiton Augusto Corrêa Bezerra · github.com/cleitonaugusto