MagSolutionsAIHow often we are wrong
Every security scanner produces false positives. Almost none publish how many. This page is ours, and it is derived from the regression tests — never written by hand.
Why nobody else does this. A company with 17,000 customers cannot start saying “15% of our comments are noise”, even when an independent audit already measured it. Their customers would ask, and so would their investors. It is a position only a company with nothing to lose can hold — and it stops being holdable the day it has something. So we are building it now, while we still can.
The numbers
| What | Value | Where it comes from |
|---|---|---|
| Corrections pinned by a regression test | 83 | Counted from the test file. Delete a test and the number drops by itself |
| Public repositories cited as the source of a correction | 21 | Each test carries a owner/repo#PR comment you can open |
| Pull requests analysed in the weekly field sweeps | 1246 | 5 sweeps, 2026-09-04 to 2026-09-14 |
| Dependencies resolved against the live registries | 1583 | PyPI and npm, at the moment the pull request was opened |
| Pull requests where we said anything at all | 6.4% | The other 93.6% got a clean check and no comment |
| Hallucinated packages confirmed | 0 | Out of 10 raw 404 candidates, every one turned out to be our false positive on review |
Figures frozen at the last deploy. This page tries to refresh them live from our public endpoint; if that fails you are reading the frozen copy.
The first pass was terrible, and that is the point
The first time the detector ran against 249 real public pull requests it produced 150 findings, 79 of them marked critical. We reviewed every one by hand. Almost none were real.
On the same population, after the corrections: 39 findings, zero to one critical. The difference is the 83 tests below, and every one of them exists because we got something wrong on somebody else's code.
A recent one, so you can see the shape of it. On 17 September 2026
an audit of 6,000 files from already-published libraries — google-auth, numba,
torch, kubernetes — found four findings in our install-time scan. All four were
false. Three were the same thing: a comment documenting the PEM format,
# "-----BEGIN PRIVATE KEY-----...", reported as a private key.
On the same corpus today: zero. And a genuinely commented-out key
still fires — that regression test exists too.
Check it yourself
Not “trust us”. These open without an account:
- The corpus — all 83 tests, each with the exact line and the pull request it came from.
- The instrument — how we measure the noise against real public pull requests.
- How to verify — including why none of the credential-shaped strings in the corpus is real.
- The raw record and the live field figures.
Three ways to check, in rising order of effort: open a cited pull request
and find the line; count the def test_ functions and compare
with the number above; or install the App on a throwaway repository, paste
those lines into a pull request, and see whether it behaves as the test says.
What this page does not claim
- It is not a false-positive rate in the statistical sense. It is a count of corrections and the population each was found in. We do not claim a percentage we cannot defend.
- The corpus is ours. We chose what to look at. An independent audit would be worth more, and we would publish it too.
- Zero confirmed hallucinated packages is a real result, not a good one. It means the base rate in ordinary pull requests is very low. We publish it because it partly undercuts one of our own arguments.
- We have made mistakes in this very record. On 16 September it reported 10 hallucinated packages when the truth was 0 — the raw 404 count had been labelled as confirmed. It was found by reading the live output, fixed the same day, and pinned by a test. It is in the git history.
We publish this because a security vendor asking for access to your repositories should not be believed on its word. If you find something on this page that does not hold up, tell us and it gets corrected here, with the date.