MagSolutionsAIField report: 6949 real pull requests scanned, 6.1% flagged, 46 of our own mistakes fixed in public
6949 real public pull requests scanned and 6619 dependencies resolved against the live registries; the check flagged something in 6.1% of them. 46 corrections to our own detector, each pinned by a test. Measured, not estimated, and re-checked weekly.
We run a security check on public pull requests and publish what it finds — including what it gets wrong. This is the weekly number.
What the field looks like this week
| Pull requests analysed | 6949 |
| Pull requests where the check flagged something | 6.1% |
| Dependencies resolved against live PyPI and npm | 6619 |
| Distinct package names seen | 6926 |
| Hallucinated packages confirmed | 0 |
| Candidates discarded on review | 21 |
| Candidates still pending review | 2 |
| Sweeps, since 2026-09-04 | 28 |
A 404 means a name is not published. It does not mean anyone invented it — an unpublished sibling inside a monorepo returns exactly the same 404, and no HTTP request can tell them apart. So a name is counted only once a person has confirmed it, and the discarded and pending counts are published beside the total.
2 candidate(s) have not been looked at yet and are not in the count above. Uncounted is the default: we would rather say nothing than say something we have not checked.
None confirmed — and that is not good news for us.
A lot has been written about AI assistants inventing package names. On 6949 real pull requests we have confirmed none, and discarded 21 candidate(s) on review — sibling packages of a monorepo, and configuration keys read as package names.
We publish this because it partly undercuts one of our own arguments. The laboratory studies that report high hallucination rates measure what a model produces when asked to solve a task. That is a different population from what survives review and reaches a pull request, and the difference matters if you are deciding where to spend attention.
Check any of this yourself
- The raw record: https://api.magsolutionsai.com/quality
- The live field figures: https://api.magsolutionsai.com/measurement
- The corpus behind both, with every cited pull request: https://github.com/MagSolutionsAI/MagSolutionsAI.github.io/tree/main/evidence
- Written up in full: https://magsolutionsai.com/quality.html
Generated on 2026-10-05 from measurements taken on public repositories. Every figure above is resolved against the endpoints linked here before publishing, and re-checked weekly afterwards — if one stops being true, this article gets a dated correction at the top.
This report was generated from measurements, not written by hand: every figure above is resolved against our public endpoints before publishing and re-checked once a week afterwards. If one stops being true, a dated correction appears at the top of this page. How often we are wrong →