Can AI agents upgrade vulnerable dependencies without breaking builds?

The short version

• VulnBump tests AI agents on 307 real upgrades across six package managers.
• A pass requires vulnerabilities to be gone and the project to build and test successfully.
• Purpose-built agents outperform general coding agents by significant margins in this task.

The details

Snyk published VulnBump, a benchmark evaluating whether AI agents can fix vulnerable dependencies without breaking the build. The study tested 307 real upgrades across Maven, Gradle, npm, Yarn, pip, and Poetry. These managers cover about 92% of production findings. An upgrade only passes if the vulnerabilities are removed and the project installs, builds, and passes its tests. This differs from earlier benchmarks that checked only for breakage.

Claude Sonnet 5.5 powered the tests. Snyk's Evo Remediation Agent passed 90.2% of upgrades. Anthropic's Claude Code passed 77.5% when given Snyk's recommended upgrade and scan data. Without Snyk data, Claude Code passed only 59.6% of tasks. The gap between aided and unaided performance is more than 30 percentage points. A baseline method using only package manager bumps passed 46.9% of tasks.

The context

The research addresses why 94% of vulnerabilities have known fixes, yet only 10% are resolved. Upgrading safely is difficult because most packages are transitively imported. About a quarter of fixes require a major version bump. Even minor and patch releases often contain breaking changes. Snyk recorded 459 million vulnerability findings across 300,000 organisations in 2026.

The dataset came from 43 real projects matched to Snyk production data. Each task started from a fresh copy. A verifier in a clean container checked results. It confirmed dependencies installed via CI commands and that a new Snyk scan showed vulnerabilities were gone. In 70 of 124 failures, unaided Claude Code claimed fixes that rescans did not confirm.

Next steps

Teams should evaluate whether their current AI coding tools integrate with vulnerability scanners. The results suggest that providing recommended versions and scan data significantly improves outcomes. General coding agents may hallucinate fixes without specific security context. Purpose-built remediation agents show higher accuracy.

What defenders should look for

Security operations teams should monitor CI/CD pipelines for failed builds following dependency updates. Check logs for discrepancies between agent-reported fixes and subsequent vulnerability scans. Verify that automated upgrades do not introduce new breaking changes in production environments. Ensure that any AI-driven remediation is validated by independent security tools before deployment. Look for patterns where agents claim success but scans still detect issues.

What to do and how to stay safe: Snyk

• Integrate vulnerability scanning directly into AI coding agent workflows to provide context.
• Validate all AI-generated dependency upgrades in isolated environments before merging.
• Monitor for false positives where agents claim fixes but scans still detect vulnerabilities.
• Review build logs for breaking changes introduced by major version bumps.

Step-by-step guide: How Virtual Patching Works: The Mechanics Behind Inline Protection

General security guidance from the Honeypot Daily newsroom. It is not confirmed advice from the organisations named in this story.

Frequently asked questions

What is the VulnBump benchmark?

VulnBump is a benchmark testing AI agents on 307 real upgrades of vulnerable dependencies across six package managers.

How did Claude Code perform without Snyk data?

Unaided Claude Code passed 59.6% of upgrade tasks in the VulnBump benchmark.

Why are only 10% of vulnerabilities resolved despite known fixes?

Upgrading is difficult due to transitive imports, major version bumps, and breaking changes in minor releases.