Tool · Security review question bank
AI Agent Security Review: 20 Questions Your Security Team Should Ask, and the Answers That Disqualify a Vendor
Vendor guides explain why their agent is safe; this AI agent security review gives the reviewer 20 literal questions, each paired with the answer that should end the evaluation.
10 min read · Last reviewed 28 Oct 2026
Review the agent’s boundaries, not its model — IBM found 92% of organisations with an AI-related incident lacked proper AI access controls, and Microsoft now argues every agent needs its own identity and task-scoped permissions.
- Dated anchors: Microsoft Security Blog (19 Oct 2026); OpenCode breach, disclosed 17 Oct and reported by BleepingComputer (19 Oct 2026); Asymptote Labs funding, reported by Unite.AI (20 Oct 2026).
- Breach data: IBM Cost of a Data Breach Report 2026 (Ponemon, 600+ breached organisations), as reported by Help Net Security.
- Vendor guides read 28 Oct 2026: Ada (10 Aug 2026) and Intercom (11 Jun 2026).
- Aissist’s self-audit uses only its public security page, read 28 Oct 2026.
- Disclosure: this is Aissist’s site, and Aissist sells AI agents. It appears in its own audit, weak answers included.
Why now
Why does an AI agent need its own security review?
Because an agent holds access and takes actions, its failure looks like an insider’s, not a vendor outage. Three dated events in one week made that concrete. On 19 October, Microsoft’s Nadim Abdo published “Agents are your newest insiders”, arguing that each agent needs an identity distinct from its sponsoring user, should start with no authority, and should receive task-scoped, time-bound permissions that shrink as risk rises.
“The model cannot be its own security boundary.”
The same weekend showed the cost of skipping that. OpenCode, an AI coding-agent platform, told users that provider API keys stored in its Console were exposed; 1.4% of Console users were affected, and some reported more than $1,700 in unauthorised charges overnight, BleepingComputer reported. On 20 October, Asymptote Labs raised $5.5 million for Beacon, an MIT-licensed telemetry layer that records agent activity in a common format, according to Unite.AI.
IBM’s 2026 breach study found roughly one in five organisations had an incident involving an AI model or application, and 92% of that group lacked role-based access, MFA and similar controls on it; only two in five apply access controls to AI at all, Help Net Security reported.
“AI is making attacks faster and cheaper, while breaches keep getting more expensive.”
The question bank · 1–10
Which identity, access and data questions disqualify a vendor?
An agent that borrows a human’s credentials, holds standing write access, or can’t name every subprocessor (each third party that touches your data) should fail before anyone reads the model card. Questions 1–10 cover who the agent is and what it can touch. The right-hand column is the answer that ends the review.
| # | Question to ask | Disqualifying answer |
|---|---|---|
| Identity and access | ||
| 1 | Does the agent act under its own identity, separate from any human user? | It uses an admin’s login or personal API token. |
| 2 | List every system it can read and every system it can write, per endpoint. | “Whatever the integration allows.” |
| 3 | Can permissions be scoped per action and per task, and expire? | One global on/off setting. |
| 4 | Where are the credentials we give you stored, and who at your company can retrieve them? | Stored in a shared workspace; no vault or access log. |
| 5 | How fast can we revoke the agent’s access without your involvement? | “Open a support ticket.” |
| Data handling and retention | ||
| 6 | What data does the agent store, by class, and for how many days? | “As long as needed.” |
| 7 | Do you or your model providers train on our conversations? | Anything short of a contractual no. |
| 8 | Name every subprocessor that sees our data, including each LLM provider and region. | “Our providers are confidential.” |
| 9 | Is PII masked before it reaches the model and the logs? | “Your agents can redact manually.” |
| 10 | Can we delete one customer’s data and receive a completion record? | Deletion by email, no receipt. |
Question 4 is the OpenCode question. A platform that holds your provider keys concentrates risk; ask where they live before you hand them over. Question 10 has a public benchmark: Ada’s API deletes up to 1,000 end users per tracked job, according to Ada.
“The teams adopting them fastest aren't the ones that skip the security review.”
The question bank · 11–20
Which injection, audit and incident questions disqualify a vendor?
An agent that treats ticket text as instructions, or whose action log you can’t export, should fail even if every certificate is current. Questions 11–20 cover what happens when the agent is fooled, and whether you could prove it afterwards.
| # | Question to ask | Disqualifying answer |
|---|---|---|
| Injection and tool-use safety | ||
| 11 | What stops text in a ticket, attachment or web page from becoming an instruction? | “Our model is trained to ignore it.” |
| 12 | Which actions need human or verified-customer approval? | “None; the agent is fully autonomous.” |
| 13 | Does customer identity come from the authenticated session or the conversation text? | From the conversation. |
| 14 | Can the agent call tools, URLs or agents outside an allowlist? | “It can reach whatever it needs.” |
| 15 | Do you replay injection tests on every prompt, model or tool change? Show the last run. | An annual pen test only. |
| Audit, logging and incident response | ||
| 16 | Is every action logged with input, tool call, actor and interface? | Conversations logged; actions not. |
| 17 | Can we export or stream those logs into our SIEM (security log platform)? | A PDF on request. |
| 18 | Can the agent alter or disable its own logging or monitoring? | It runs inside the same environment as its monitor. |
| 19 | What is your incident notification window, in hours, in the contract? | “Promptly.” |
| 20 | Can we stop one agent, queue or action class in seconds, handing live conversations to a human? | Global off, via redeploy. |
Microsoft’s expense-agent example in the same post is question 12 in practice: when risk rises, keep reads, block writes, and require human approval for the consequential step. Questions 16–18 are why Beacon exists. A log the agent can edit is a diary, not evidence. For the injection mechanics behind 11–15, see our prompt injection controls guide.
The downloadable version adds a “what a good answer looks like” column and blank cells for each vendor’s answer, evidence link and pass/fail.
Deployment
Why does the deployment model change the size of the review?
An agent that runs on your existing helpdesk inherits an access model your security team has already approved; a rip-and-replace platform asks them to approve a new one. Roles, SSO, audit trail, retention and deletion all stay in the system you already run. The new review item is one integration credential and the vendor’s processing.
A platform that replaces the helpdesk becomes the new system of record: a new user directory, a new store holding every conversation, a new admin console, new retention and deletion processes. Each is a fresh line in the questionnaire. Intercom’s guidance makes the same incremental point from the vendor side, recommending one system and a small set of fields before any write permission.
“Read-only access is usually the right starting point.”
Inherit the model, not the person. The integration credential should be its own service identity, scoped to the queues and actions in questions 2 and 3, never a copied admin account. And inheritance cuts both ways: if your helpdesk roles are already over-broad, the agent’s will be too. Fix that first; our guardrails guide covers the layers.
Self-audit
Which of these questions does Aissist answer badly?
Aissist answers the access and PII questions well, and the subprocessor, retention and log-export questions badly. We ran the bank against Aissist’s public security page on 28 Oct 2026, accepting only what a reviewer could find in writing.
| Question | Aissist’s public answer (28 Oct 2026) | Verdict |
|---|---|---|
| 1–2 · Identity and scope | Writes under credentials you provision; reads only what you connect | Strong |
| 9 · PII masking | Detected and masked automatically | Strong |
| 12 · Error control | Contracted S0 errors under 0.01%, S0–S2 under 1% (vendor-claimed) | Strong |
| Certifications | ISO 27001 certified; SOC 2 aligned, not certified | Partial |
| 16–17 · Logs | Audit logs kept and available on request; no documented SIEM export | Partial |
| 6 · Retention | No retention period published | Gap |
| 7 · Training | Not stated on the security page | Gap |
| 8 · Subprocessors | Names AWS and Azure; no LLM provider list | Gap |
| 19 · Notification window | Not published | Gap |
Four gaps out of nine is not a flattering scorecard. It is the one a reviewer would produce in an afternoon, so we published it first.
Verdict
Review the boundary, not the model
An AI agent passes a security review when its identity, scope, data and actions are bounded and provable, whatever model sits inside. Send questions 1–20 to every shortlisted vendor and write the answers to 4, 7, 8 and 19 into the contract. If the agent will take write actions, ask 3, 12 and 20 first, because they decide what a fooled agent can do. This bank is the security review; for commercial procurement use the AI agent RFP question bank, and for policy tooling see AI governance tools. The governance control gap explains why confidence is not a control, and Reliable AI shows how Aissist grades its own errors.
FAQ
Frequently asked questions
What is an AI agent security review?
An AI agent security review is the assessment a security team runs before an AI agent receives access to company systems. It tests the agent's identity, permissions, data handling, resistance to prompt injection, and audit logging, rather than the model's capabilities.
What questions does security ask about AI agents?
Security teams ask who the agent is, what it can read and write, where its credentials live, which subprocessors see the data, what stops injected instructions, and whether every action is logged and exportable. A good vendor answers each in writing.
What data access scopes should an AI agent have?
An AI agent should have the narrowest scopes its current task needs: read access to specific endpoints, write access only to named actions, time-bound grants, and approval gates for refunds, account changes or anything irreversible. It should never inherit a human's full access.
What are the audit log requirements for an AI agent?
Every agent action should be logged with the triggering input, the tool call, the acting identity and the interface used. Logs should be exportable or streamable to your SIEM, retained for a stated period, and outside the agent's ability to alter.
Should an AI agent have its own identity?
Yes. A distinct identity lets you tell what a person did from what an agent did on their behalf, and lets you scope or revoke the agent without touching the human's access. Microsoft's October 2026 guidance makes the same recommendation.
Is SOC 2 enough to approve an AI agent?
No. SOC 2 and ISO 27001 cover the vendor's infrastructure and processes, not how the agent behaves with untrusted text or what actions it can take. Ask the injection, approval and logging questions separately.
How is a security review different from an AI agent RFP?
An RFP covers commercial terms such as resolution definitions, pricing and data export. A security review covers identity, access, data handling, injection safety and audit evidence. Most buyers need both, run by different teams.
What answer should disqualify an AI agent vendor?
Disqualify a vendor whose agent uses a human's credentials, holds standing write access with no approval gates, can't name its LLM subprocessors, or can't export an action-level log. Each one means you cannot bound or prove what the agent did.
Send us all 20 questions.
Our security team will answer every line in writing — the four gaps above included.