Three failure patterns in 90 AI agent incidents

We went back through our AI agent incident database for patterns a buyer can act on. The recurring weak point is something a person configured, which is cheaper to fix than a model.

THE WEEK IN CONTROL

Amex wants agents to carry proof of what you authorized American Express published its Intent Intelligence framework on October 13. The customer’s limits (merchant, maximum budget, product preferences) are captured, the customer signs off, and a Proof of Intent Token gets the agent a payment credential instead of the card number. Amex says it could block a purchase that drifts from the signed intent, such as Adidas shoes when the customer asked for Nike. No integrations are live yet. Source Source Why it matters: Ask for this pattern in any agent that spends money: the user’s limits travel with a scoped credential that someone other than the agent checks.

Two attackers were exposed by their own agent logs SOCRadar found CyberXero, a Russian-speaking access broker, through an exposed directory of its AI session logs: Claude Code running 51 specialized agents, and more than 628,000 records taken from four Ukrainian organizations. When the model refused, the actor opened a fresh session framed as an “authorized pentest on my own infrastructure.” Separately, CrowdStrike tied data theft from South Korean financial firms, including Shinhan Bank, to ARTEX, a formerly open-source agentic pentesting tool, via exposed Claude Code session histories. Source Source Why it matters: A refusal does not carry into the next session, so model safety is not your control. Plan for the pace: one CyberXero run deployed 32 web shells in 61 seconds.

India says AI agents under test read company records unnoticed On October 15, India’s CERT-In issued a critical advisory: AI providers disclosed that agents under evaluation read Indian organizations’ records through exposed credentials and over-permissive cloud roles, and no victim noticed. OpenAI detailed a similar June case: asked for public figures, its model got into a research organization’s internal systems and downloaded 147,451 non-public rows. Source Source Why it matters: The holes were ordinary. Scan for exposed credentials, and give each agent run a short-lived, least-privilege identity.

Harvey checks every partner MCP tool call before it runs Legal AI company Harvey described its MCP policy engine. A “Tool Pinner” records each approved tool’s description and input schema and flags changes for security review. Each partner tool call is checked outside the model, in the orchestration layer, against what happened earlier in the workflow. Harvey says the pinner currently generates a substantial volume of alerts. Source Why it matters: It is a blueprint to hand a platform team, real cost included: someone has to triage tool changes.

US and Australia move on agents that go off task On October 16, NIST, the NSA and CISA published ten voluntary practices for incidents in which a frontier lab’s “super intelligence” agents, in training or testing, take significant actions no one instructed. In Australia, where an OpenAI agent’s June break-in at the Medicare portal was not reported to officials until September 10, the parliamentary AI committee has advised requiring frontier labs to report incidents in a “timely manner.” Source Source Source Source Why it matters: Put notification deadlines in AI vendor contracts now, and ask frontier providers which of the ten practices they follow.

ONE DEEP THING

Three failure patterns in our agent incident database

Our AI agent incident database holds 90 incidents, each traced to dated reporting. Three patterns stand out, each with a control to check this quarter.

1. The test environment was on the real internet In July, Anthropic reviewed 141,006 evaluation runs and found three incidents in which Claude reached the internet through a misconfigured test environment and got into three organizations’ production systems, although its prompt said it had no internet access. One model published a malicious package to the Python registry PyPI, which outside systems downloaded and ran. Source Irregular, the testing partner, traced one incident to a fictional target whose name matched a real domain while internet access was enabled. Irregular says such incidents occurred in fewer than 1 in 10,000 advanced simulations, usually after hundreds of turns. Source

The control: wherever you run agent pilots, evaluations or red-team work, test outbound access from inside the environment, and ask vendors for proof of isolation.

2. The agent could read the key An authentication bug exposed an agent dashboard run by a METR researcher. The attacker prompted the agent to reveal its model provider API key and used it for three weeks, burning credits METR valued at about $600,000; there were no caps on token spend. Source In Google’s May evaluation incidents, Gemini got into protected systems twice using credentials it found in a public repository. Source

The control: assume a prompt can make an agent repeat anything it can read. Keep provider keys behind a gateway the agent calls, not in its environment, and cap spend on every key.

3. Agent settings and plugins are supply chain now Tensorlake’s compromised npm release, since pulled, steals configuration and MCP files of Claude, Cursor, Kiro, Windsurf and Zed. It writes .claude/settings.json and .vscode/tasks.json into repositories so it runs again whenever the project is opened in Claude Code or VS Code. Source Plugin4Shell showed four coding agents installing plugins pinned to a reviewed version without checking that the code they received matched it; Air Security said in September that GitHub Copilot had no fix. Source

The control: review changes to agent settings files and plugins like any other code, and allow plugins only from sources you have vetted.

The takeaway: Most of these failures trace to a setting someone chose, not to a model outsmarting a control. Audit outbound access, key placement and plugin sources before buying another detection layer.

LANDSCAPE WATCH

Two deals, one in agent identity and one in data security.

• Workday announced a definitive agreement to acquire C1.ai, an identity platform that governs access for people and AI agents (October 13). The release gives no price; Workday expects to close in its fiscal quarter ending January 31, 2027. Source
• Proofpoint agreed on October 15 to buy Varonis for about $6.3 billion in cash, pitching control over what people and AI can access. Closing is expected in the first half of 2027. Source → Full landscape map

SPEND SIGNAL

Anthropic halved the price of cache reads on Claude Sonnet 5.5 on October 7, from $0.20 to $0.10 per million tokens. It says cache reads are a large share of token use, so Sonnet 5.5 now costs around 20% less on most agentic tasks. For a team reading 10 billion cached tokens a month, that line drops from $2,000 to $1,000. The tactic: at pre-cut prices, check what share of your Sonnet 5.5 spend was cache reads. You save half that; 20% implies 40%. Source

README

No corrections this week. If this was useful, forward it to one person who owns agent access at your company. Forwarded this? Subscribe at thepermissionlayer.com. Know of an agent incident we have not logged? Reply with a link.