Tools · Offensive & testing
AI red teaming and LLM security testing tools
These tools attack a company's own models, LLM applications and agents the way an adversary would, sending prompt injection, jailbreak, data leakage and insecure tool use attempts and reporting what got through. Several are open source. A team needs one before an agent that reads untrusted content or calls tools goes to production, and again after each change to its prompts, model or tools.
Offensive & testing 4
-
garak (vendor site, opens in a new tab)
generally availableNVIDIA
Open-source LLM vulnerability scanner that probes models and dialog systems for hallucination, data leakage, prompt injection, toxic output and jailbreaks.
-
Mindgard (vendor site, opens in a new tab)
generally availableMindgard
Runs attacker-style reconnaissance and automated red teaming against AI models, agents and applications, then validates findings and supports fixes.
-
Promptfoo (vendor site, opens in a new tab)
generally availablePromptfoo
Open-source CLI and library for evaluating and red-teaming LLM apps and agents, generating attacks such as prompt injection, jailbreaks, PII leaks and insecure tool use. It runs in CI/CD and can review pull requests for LLM security issues.
-
PyRIT (vendor site, opens in a new tab)
generally availableOpen-source Python framework for security teams to probe generative AI systems for risks through automated red teaming.
The briefing
New AI agent governance tools, by email
The briefing's New Tools section covers the latest governance launches, each linked to the vendor's announcement, next to the news that decides which ones matter. At least once a week, five-minute read, free.
You confirm by email. Unsubscribe in one click.
Questions to ask vendors
Neutral questions for a request for proposal (RFP) or a security questionnaire. No product here is named as the answer.
-
Does testing cover agents that call tools and act over several steps, or only single prompts and responses?
-
Are attacks generated for our application, or replayed from a fixed library, and how often is that library updated?
-
Can tests run in our build pipeline on every change, and can a failed test stop a release?
-
When a test agent probes our systems, what can it reach, and how is it kept away from production data?
-
How are findings scored, and does the report show how each score was reached?
More questions, grouped by topic: Questions to ask AI agent security vendors.
Related incidents
- Atlassian Rovo could be prompt-injected into sending Jira and Confluence data to attackers
- A poisoned web page rewrote AWS Kiro's config and ran code
- An OpenClaw agent exploited a gym booking system to move up a waitlist
- A GitHub issue could escalate from a low-privilege Google ADK agent to a privileged one
Every record in the AI agent incident database links the sources that reported it.
Missing a product, or found an entry that is out of date? Write to [email protected]. A submission is read, never paid for, and does not guarantee a listing. The glossary defines the terms used here.