Vendor questions
Questions to ask AI agent security vendors
Every question below comes from one of our explainers, where the reasoning behind it sits next to the incident or source that prompted it. Follow the link under a question to read why it matters.Each group ends with the address of the explainer that gives the reasoning. Nothing here is sponsored, and no product is named as the answer. Copy any of them into a request for proposal (RFP) or a security questionnaire. No signup needed.
Get the briefing by email
These questions come from The Permission Layer, a free briefing on AI agent governance sent at least once a week.
Free, at least once a week. You confirm by email.
MCP gateways
For vendors selling the control point between AI agents and their tools. MCP is the Model Context Protocol, the open standard agents use to connect to tools and data.
-
Can access be allowlisted per agent and per tool, rather than only per server? An agent that needs one tool from a server should not inherit all of them. #
-
Does the log capture the full tool call and the full response, and can we export it to our existing security tooling? #
-
How is tool output screened for injected instructions, and what happens on a hit: block, redact, or flag for review? #
Injected instructions are prompt injection: text hidden in content an agent reads, including what its own tools hand back, that tries to change what the agent does.
-
How do agents authenticate through the gateway? MCP builds on OAuth, the standard way software gets permission to act on your behalf. Ask whether the gateway validates the issuer parameter before a code is redeemed, whether it supports Client ID Metadata Documents or only the deprecated registration path, and how a grant gets revoked. #
The MCP revision dated 2026-07-28 has clients validate the issuer (iss) parameter defined in RFC 9207, and deprecates Dynamic Client Registration in favor of Client ID Metadata Documents.
-
Does it still require sticky sessions? The current spec is stateless by design, and a gateway that pins each conversation to one backend hands you back a scaling problem the protocol removed. #
The 2026-07-28 revision removed the initialize handshake and the Mcp-Session-Id header, so any server instance behind a load balancer can take any request.
-
Which spec revision does it implement today, and which extensions does it pass through, inspect, or block? #
At the explainer's October 2026 review, the current MCP revision was dated 2026-07-28. Capabilities such as Tasks and MCP Apps now arrive as named extensions, so "we support MCP" is not a complete answer.
-
What is the plan for proof-of-possession tokens? DPoP is on the protocol roadmap, and a gateway that only understands bearer tokens has work ahead of it. #
-
Who can change policy, and is every policy change itself logged? #
-
How much latency does the gateway add per tool call, measured, not estimated? #
Reasoning and sources: What is an MCP gateway? (thepermissionlayer.com/learn/what-is-an-mcp-gateway/)
Some of these questions also appear next to the products on: MCP gateway and MCP security tools, AI agent security and guardrail tools and Agent and LLM observability tools.
Agent identity and non-human identity (NHI)
For identity vendors, and for any agent platform that holds credentials for its agents. A non-human identity is any account that is not a person: service accounts, API (application programming interface) keys, bots and AI agents.
-
Does each agent get its own identity and credential, or do agents share service accounts under the hood? #
-
Are credentials short-lived and automatically rotated, or static keys that live until someone remembers them? #
-
Is every identity assigned a human owner, and is that ownership recorded somewhere we can query? #
-
When an agent acts for a user, does the log capture both the agent and the user it acted on behalf of? #
-
How fast can we revoke one agent's access without touching any other agent? #
-
Can we set a hard spend cap per credential, so a leaked key has a ceiling on the damage? #
AI model evaluator METR disclosed that an API key stolen in March 2026 ran for about three weeks and consumed credits worth about $600,000, with no cap on token spend to stop it.
Reasoning and sources: What is agent identity? (thepermissionlayer.com/learn/agent-identity-and-nhi/)
Some of these questions also appear next to the products on: AI agent identity and NHI tools, Agent and LLM observability tools and AI agent control plane tools.
The briefing
Get the briefing these questions come from
The explainers behind this list come out of the briefing on AI agent governance. At least once a week, five-minute read, free.
You confirm by email. Unsubscribe in one click.
Agent permissions and RBAC
For vendors who decide or enforce what an agent may do. RBAC is role-based access control: granting permissions by role rather than person by person.
-
Can permissions be granted per task and expire on their own, or is every grant standing until someone revokes it? #
-
When an agent acts on behalf of a user, is its effective access the intersection of its own permissions and that user's? #
If the agent's access is broader than the user's, injected content can use the agent's authority to do what the user never could. This is the confused deputy problem.
-
Is authorization enforced by an external control point, or does it rely on the agent framework behaving as documented? #
-
Are high-risk actions gated behind human approval, and can we define which actions count as high risk? #
-
Does the audit log record the permission that authorized each action, so we can trace a bad outcome back to a bad grant? #
-
How fast does revocation propagate to an agent already in the middle of a task? #
Reasoning and sources: Does RBAC work for AI agents? (thepermissionlayer.com/learn/rbac-for-ai-agents/)
Some of these questions also appear next to the products on: AI agent identity and NHI tools, AI agent security and guardrail tools, AI governance and compliance tools and AI agent control plane tools.
LLM gateways and AI spend
For vendors of the proxy between your applications and the model providers, also sold as an AI gateway. LLM is large language model.
-
Can spend be attributed per team, per application, and per agent, rather than only per provider account? #
-
Are budget limits hard caps that stop traffic, or alerts that arrive after the money is gone? #
-
How are routing policies defined, and can we test a policy change against last month's real traffic before enabling it? #
-
Where do provider keys live, how are they rotated, and who can read them? #
-
What is the security track record and patch turnaround? Ask specifically how fast known exploited vulnerabilities get fixed in your deployment model. #
In September 2026 the Cybersecurity and Infrastructure Security Agency (CISA) added a LiteLLM gateway flaw, CVE-2026-59822, to its Known Exploited Vulnerabilities catalog.
-
How much latency does the proxy add, and what happens to traffic when the gateway itself is down? #
-
If we leave, does our code keep working? Gateways that mimic a standard API shape are cheaper to exit. #
Reasoning and sources: What is an LLM gateway? (thepermissionlayer.com/learn/llm-gateways-and-spend/)
Some of these questions also appear next to the products on: LLM gateway and AI spend control tools.
Shadow AI discovery
For vendors of tools that find the AI use nobody approved: unsanctioned assistants, coding agents, browser extensions and API keys.
-
How is usage discovered: network traffic, single sign-on logs, browser telemetry, endpoint agents, or expense data? Each source misses different things. #
-
Can it see API keys and command-line tools, or only web apps? The riskiest shadow AI rarely runs in a browser tab. #
-
Can it distinguish sanctioned tools from unsanctioned use of the same tool, such as a personal account for an approved product? #
-
What is the enforcement posture: observe, warn, or block, and can it steer employees toward an approved alternative instead of a dead end? #
-
How is employee privacy handled? Monitoring that reads message content creates its own governance problem. #
-
Does discovery feed an inventory you own and can export, or a dashboard you rent? #
Reasoning and sources: Shadow AI policy: what it should cover (thepermissionlayer.com/learn/shadow-ai/)
Some of these questions also appear next to the products on: AI-SPM and exposure management tools, AI data loss prevention (DLP) tools, Shadow AI discovery tools and AI governance and compliance tools.
Agent plugins, skills and hooks
For coding agent and agent platform vendors whose products install third-party skills, plugins, extensions or hooks. Each one runs with the agent's permissions.
-
Does the installer verify a content hash, or trust the reference? What changed after Plugin4Shell, and in which version? #
Plugin4Shell, disclosed by Air Security on September 18, 2026, let a plugin repository's owner create a branch named after a pinned commit hash, so four coding agents installed different code while reporting the pinned hash.
-
Which files does the agent execute automatically when a project opens or a tool runs, and can each one be disabled or gated behind approval? #
-
Can installs be restricted to an allowlisted marketplace or registry, with everything else blocked rather than warned about? #
-
Does the scanner read natural-language instructions as well as code, and what is its measured false-positive rate? #
-
Is every install, hook execution and skill load logged with a hash we can export to our own tooling? #
-
How many days passed between the Plugin4Shell and core.fsmonitor disclosures and your fix? Ask for dates, not a policy. #
The core.fsmonitor flaws, disclosed by Manifold Security on September 2, 2026, let a cloned repository's Git settings run an attacker's command outside the agent's sandbox with no approval prompt.
Reasoning and sources: Agent plugins and skills are a supply chain (thepermissionlayer.com/learn/agent-plugin-supply-chain/)
Before a vendor's agents touch your systems
For any vendor whose own agents will run in or against your environment. The questions come from the evaluation escapes four frontier labs confirmed in 2026.
-
Will these agents ever run with classifiers off, in which environment, and who signs off? #
Labs often switch off a model's safety classifiers during evaluations so the test measures capability rather than refusals. The 2026 escapes came out of evaluation and training runs.
-
Show us the egress policy, not an assurance. Is outbound access denied by default, and what is on the allowlist? #
-
Which credentials can the agents reach, including any sitting in repositories they can read, and how fast can one agent be cut off? #
-
Which write paths lead from the agent to the outside world, and is each one logged? #
-
How quickly would you know an agent acted outside scope, in minutes or days, and how quickly would we? #
-
Who investigates, and do independent investigators get the transcripts? #
-
Does your evaluation partner run your controls? Irregular appears in three labs' incidents. #
Irregular is a testing partner that runs evaluations for AI labs. Its sandboxes figure in incidents disclosed by Google, Meta and OpenAI.
Reasoning and sources: The 2026 agent escape arc, explained (thepermissionlayer.com/learn/agent-escape-arc-2026/)
Some of these questions also appear next to the products on: AI agent sandboxing tools.
OWASP Agent Control Standard (ACS)
For agent platform vendors. ACS is OWASP's open specification for checking an agent's actions against policy while it runs.
-
Ask agent platform vendors which of the nineteen hooks they expose, which dispositions they honor, and whether they emit OpenTelemetry spans and OCSF events. Modify and ask are the two dispositions that quietly go missing, and since release 0.1.3 a product can skip modify and still call itself conformant. #
ACS defines nineteen lifecycle hooks, moments such as a tool call about to happen, and five dispositions a policy authority can answer with: allow, deny, modify, ask, or defer. OCSF is the Open Cybersecurity Schema Framework.
-
Ask the same vendors for a date, not a direction, on AgBOM output in CycloneDX, SPDX, or SWID. #
The same vendors are agent platform vendors. An AgBOM (Agent Bill of Materials) lists the models, tools and data an agent can reach; CycloneDX, SPDX and SWID are the formats ACS names for it.
Reasoning and sources: The OWASP Agent Control Standard, explained (thepermissionlayer.com/learn/owasp-agent-control-standard/)
AIVSS risk scores
For any vendor that hands you an AIVSS score. AIVSS is the OWASP AI Vulnerability Scoring System.
-
Ask vendors which AIVSS version they scored against, and whether they publish the ten factor scores or only the final number. A vendor who will not show the factors is asking you to trust their view of your deployment, which they have not seen. #
AIVSS adds ten Agentic Risk Amplification Factors, such as Autonomy, Tools and Persistence, to a CVSS (Common Vulnerability Scoring System) base score. Version 0.8, published March 19, 2026, was current at the explainer's October 2026 review.
Reasoning and sources: AIVSS, explained (thepermissionlayer.com/learn/aivss-explained/)
The Permission Layer
Get the next issue by email
That's all 50 questions. The briefing they come from covers new AI agent incidents, launches and pricing changes for the people who approve agent access. At least once a week, five-minute read, free.
You confirm by email. Unsubscribe in one click.
Missing a question you put to every vendor? Email [email protected] and it gets considered for the next update. Corrections are welcome at the same address.
To see who sells what, use the vendor landscape map or the tools directory. For the failures behind many of these questions, see the incident database.
From thepermissionlayer.com/questions/, updated October 14, 2026. Vendor-neutral, free to copy.