Research
We read all 162 automatic fixes. 95 were wrong.
In October 2026 we ran Railo's automatic fixes on 11 public repositories and read every fix it proposed. Here is what we found, why the fixes were wrong, and what we changed. First published on dev.to on 16 October 2026.
Updated · Railo
Key findings
Across 11 public repositories, Railo's scan reported 923 findings and proposed 162 automatic fixes. We read all 162 by hand: 95 should not have been proposed, mostly because they edited vendored libraries, test data or seed data. None of the 162 broke a file's syntax. Wrong is not the same as broken, and wrong was the problem. After the changes below, the same scan proposes 67 fixes.
| Measure | Result |
|---|---|
| Repositories | 11 (6 deliberately vulnerable, 5 well maintained) |
| Findings reported | 923 |
| Automatic fixes proposed | 162, all read by hand |
| Fixes that should not have been proposed | 95 |
| Fixes that broke a file's syntax | 0 |
| Fixes proposed after the changes | 67 |
Method
- A shallow clone of each repository, scanned in full rather than one pull request at a time.
- The same code path as the GitHub Action, with Semgrep pinned at 1.102.0.
- Every changed file parse-checked before and after the fix with the language's own tooling: Python compile, the TypeScript parser, gofmt, php -l and a javac parse step. A new parse error counted as a broken file.
- Every proposed fix read by a person and judged in context. “Wrong” means it should not have been proposed.
- The projects' tests were not run, and nothing from the repositories was executed.
Results by repository
| Repository | Type | Findings | Fixes before | Fixes after |
|---|---|---|---|---|
| WebGoat | Deliberately vulnerable | 381 | 95 | 15 |
| Juice Shop | Deliberately vulnerable | 216 | 11 | 6 |
| PyGoat | Deliberately vulnerable | 132 | 16 | 16 |
| DVWA | Deliberately vulnerable | 94 | 27 | 24 |
| NodeGoat | Deliberately vulnerable | 24 | 10 | 6 |
| govwa | Deliberately vulnerable | 8 | 0 | 0 |
| Flask | Well maintained | 22 | 1 | 0 |
| Gin | Well maintained | 20 | 1 | 0 |
| Express | Well maintained | 14 | 0 | 0 |
| Spring Petclinic | Well maintained | 9 | 1 | 0 |
| Slim | Well maintained | 3 | 0 | 0 |
| Total | 923 | 162 | 67 |
WebGoat alone went from 95 proposed fixes to 15; 80 of the 95 removed fixes came from it.
Why fixes were wrong
- Edits to vendored third-party code, such as the Ace editor files inside WebGoat (about 77 fixes) and three.js in Juice Shop.
- Test data rewritten, such as Gin's TLS test certificate and key in testdata/.
- Seed data changed, such as demo users given one shared password variable.
- Environment placeholders written into files where nothing substitutes them.
- Server-side environment variables written into browser code.
- Names that are not secrets flagged as secrets, such as token types and addresses.
- innerHTML changed to textContent where the HTML was intended, and Flask's Markup double-escaped.
What we changed
- Vendored, minified and bundled files, and JavaScript files over 5,000 lines, are treated as third-party code.
- Test and mock folders are skipped.
- Several different values for one key in a file are treated as data, not a secret.
- Environment placeholders are only written into Docker Compose and Spring Boot configuration.
- Browser code is detected and not rewritten for server-side settings.
- Generic token values are skipped unless they match a known provider's key format.
- Cases that need judgement, such as JSONP callbacks, are sent to a person.
Limits
- Railo's founders ran the test and judged the fixes. It is not an independent study.
- Six of the 11 repositories are deliberately vulnerable training apps, which are not typical code.
- The projects' test suites were not run, so a fix that parses can still change behaviour.
- Some findings remain that we consider noise, and some detection is partial: in one DVWA file the if branch is fixed but an identical else branch is not.
What this means if you are choosing an autofix tool
Ask any vendor for a failure rate measured on code they did not write, and read the fixes yourself. A fix that compiles can still be wrong, and our data shows that wrong fixes cluster where code is not really yours: vendored libraries, fixtures and seed data.
Railo's fixes are written by rules, not a language model, and every one is rescanned before it is suggested. Try the free GitHub Action, or have us clear an existing backlog in a Fix Sprint.