AI Security

A NIST‑Based Audit Checklist for Managed AI Agents with Browsing, Shell, and File‑Edit Capabilities

Published 2026-09-28 by AISecAll Editorial

TL;DR: Use the NIST AI Risk Management Framework (AI RMF) to define audit objectives, map agent capabilities (web browsing, shell execution, file editing) to specific NIST controls, collect immutable logs, run a concise checklist, and remediate gaps before the agent goes live. Small teams can complete the audit in a day with existing tooling and a short review from a specialist like AISecAll.

What audit goals should a small business set for a managed AI agent?

Before you start testing, clarify the purpose of the audit. Typical goals for a managed agent that can browse, run shell commands, or edit files include:

• Confirm that the agent only accesses approved external domains.
• Verify that shell commands are limited to a predefined whitelist.
• Ensure file‑system changes are confined to a sandboxed directory.
• Detect any unexpected data exfiltration or privilege escalation attempts.
• Document evidence for compliance (e.g., GDPR, HIPAA) and internal governance.

These goals map directly to NIST AI RMF functions: Govern, Map, Measure, and Monitor.

Which NIST controls map to agent capabilities like browsing, shell, and file edits?

The AI RMF references core NIST SP 800‑53 controls. The most relevant ones for the three capabilities are:

1. AC‑20 (Use of External Information Systems): Controls which external URLs the agent may request. Apply a whitelist of domains and enforce TLS.
2. SC‑7 (Boundary Protection): Inspects outbound traffic from the agent’s sandbox, preventing unauthorized data leaks.
3. CM‑7 (Least Functionality): Limits the command set the agent can invoke. Define an allow‑list of safe shell commands (e.g., ls, cat on read‑only files).
4. SI‑7 (Software, Firmware, and Information Integrity): Requires integrity checks on any files the agent creates or modifies.
5. AU‑12 (Audit Generation): Mandates immutable logging of every API call, web request, shell execution, and file operation.

By aligning each capability with these controls, you create a traceable audit trail that satisfies both NIST and internal policy requirements.

How to collect evidence without disrupting the agent?

Managed platforms (Claude Managed Agents, OpenAI Agents, etc.) usually expose a logging endpoint or webhook. Follow these steps:

1. Enable structured logging in the platform’s dashboard. Choose JSON format for easy parsing.
2. Route logs to a write‑once storage service (e.g., Cloudflare R2, AWS S3 with Object Lock, or a self‑hosted ELK stack).
3. For web‑browsing, capture the full request URL, response status, and a hash of the response body.
4. For shell commands, log the exact command string, the user under which it ran, and the exit code.
5. For file edits, record the file path, operation type (create, modify, delete), and a SHA‑256 hash before and after the change.
6. Set a retention policy of at least 30 days to satisfy most regulatory windows.

All logs should be immutable; avoid mutable databases that could be altered by a compromised agent.

Sample audit checklist (step‑by‑step)

1. Define scope: List all endpoints the agent will call, the sandbox directory, and the allowed command list.
2. Configure controls: Apply NIST‑referenced policies in the platform (domain whitelist, command allow‑list, file‑system sandbox).
3. Enable logging: Turn on AU‑12 logging, route to write‑once storage, and verify timestamps are synchronized via NTP.
4. Run a baseline test: Execute a set of benign prompts that trigger each capability. Verify that logs contain the expected entries and no disallowed actions.
5. Perform a negative test: Attempt to request a non‑whitelisted domain, run a prohibited command, or write outside the sandbox. Confirm the platform blocks the request and logs the denial.
6. Review logs: Use a simple query (e.g., jq or Splunk) to filter for “error” or “denied” events. Ensure no false‑positives.
7. Assess integrity: Verify SI‑7 hashes match expected values for any files the agent created.
8. Document findings: Fill out a short audit report that maps each test to the corresponding NIST control.
9. Remediate: Adjust whitelist, command list, or sandbox permissions based on gaps, then repeat steps 4‑7.
10. Sign‑off: Obtain approval from the product owner and, if needed, a security lead before production deployment.

How to remediate common findings?

Typical issues and quick fixes:

• Unexpected domain access: Add the domain to the deny list or tighten the URL regex in AC‑20.
• Shell command leakage: Reduce the command allow‑list to the minimum needed and enable command‑argument sanitization.
• File‑system escape: Enforce chroot‑style isolation or use container‑based sandboxes that enforce path constraints.
• Missing log entries: Verify the platform’s logging flag is enabled for all modules; some SDKs require explicit opt‑in for shell events.

After remediation, re‑run the negative tests to confirm the issue is resolved.

When to involve AISecAll?

If your team lacks experience with NIST controls, needs a third‑party validation, or discovers a high‑severity gap (e.g., the agent can execute arbitrary commands), a short engagement with AISecAll can provide:

• Independent verification of your audit evidence.
• Tailored hardening recommendations for the specific managed‑agent platform you use.
• Documentation templates that satisfy auditors and regulators.

Even a one‑hour review can raise the security posture of your AI automation before it touches production data.