AI security controls

Introduction

Practical, action-first controls to reduce AI-specific risks across the human-facing, application, and platform layers — written as a pragmatic playbook for engineering teams.

Why this matters

Large models change the threat balance: small mistakes in prompts, retrievals, or training data can cause outsized damage (data leaks, unsafe outputs, poisoned models). This note gives concrete patterns you can adopt immediately and evolve as risk grows.

Core principles (how we think about controls)

1. Minimize trust — assume external text and connectors are hostile by default.
2. Pin and verify — make critical configuration and artifacts auditable and verifiable.
3. Fail-safe first — prefer blocking or human review for high-risk flows.
4. Measure everything — logging + metrics drive detection and tuning.
5. Iterate adversarially — continuously test with red-team prompts and automated regression.

A short workflow map

1. Input intake (human or machine) -> 2. Sanitization & classification -> 3. Retrieval / enrichment -> 4. Prompt composition (system + context + user) -> 5. Model call -> 6. Post-processing / safety checks -> 7. Action / response / human review.

Place guards at every boundary in that flow.

Controls by layer (patterns you can adopt)

Human-facing layer (interaction & UI)

Goal: prevent users or attackers from manipulating the instruction intent and collecting sensitive responses.

• Immutable intent block: Keep the behavior-defining part of prompts (intent, policy, blocking rules) in a versioned config file not editable at runtime. Deploy with CI approval only.
• User content bucketing: Force user uploads or text through a classifier that assigns a sensitivity bucket (public, internal, secret). Use the bucket to decide whether to allow retrieval or train usage.
• Client-side hygiene: Run lightweight checks in the client (length limits, forbidden patterns) to stop obvious mass-extraction attempts before they reach the server.
• Human review gate: For categories flagged as high-risk (legal advice, payroll, account termination), require human sign-off before any action or release.

Application layer (ingest, retrieval, orchestration)

Goal: make everything that feeds the model traceable and tamper-evident.

• Trusted retrieval pipeline: Only index data from approved sources. At retrieval time, attach source-id, timestamp, and sha256 to each snippet and include that metadata with the prompt.
• Canonicalisation & scrubbers: Normalize whitespace/unicode, strip hidden control characters, and remove instruction-like tokens from retrieved text.
• Context budgeting: Enforce a strict byte/character budget for context. Prefer short excerpts with strong provenance rather than verbatim long docs.
• Capability zoning: Split features into zones (read-only RAG, write-enabled automation). Enforce RBAC at the feature level and require MFA/approval for dangerous zones.

Example: if a retrieval comes from an unverified bucket, mark it quarantine and never include it in prompts until verified.

Platform & training layer (data, pipelines, artifacts)

Goal: reduce attack surface for poisoning/theft and ensure artifact integrity.

• Dataset ledger: For every dataset add an immutable record: source URI, ingest time, pre-processing hash, and operator id. Store ledger entries in an append-only store or signed log.
• Canary & honeypot records: Seed datasets with sentinel records (non-sensitive) to detect unauthorized data extraction or model memorization.
• Artifact seals: Sign model weights and manifests. Enforce deployment only from signed artifacts and keep roll-back checkpoints with cryptographic provenance.
• Training isolation: Run sensitive training jobs in ephemeral, network-reduced environments with strict write controls and monitored I/O.

Detection & telemetry (what to log and why)

Minimum useful trace per model call:

• user_id, request_id, timestamp
• system_prompt_id + commit/hash
• retrieved_snippets: [{id, url, hash, score}]
• model_name, model_version, temperature, max_tokens
• final_output + post-filter labels
• downstream actions (API calls, DB writes)

Use these for triage, replay, and forensic analysis.

Key signals to alert on:

• spikes in flagged outputs (policy violations)
• repeated long retrievals or identical context from many users (extraction probes)
• sudden changes to system_prompt_id or failed signature verifications
• anomalous grounding scores (many unsupported claims)

Practical safety patterns (concrete, original ideas)

• Challenge-response for connectors: Before the system uses a new external connector, require a two-step verification: admin approval + successful signed-challenge roundtrip proving connector control.
• Response grounding score: Produce a simple ratio: number of output claims backed by retrieved snippets / total claims. Use an automated threshold to force human review.
• Prompt sandboxes: Run risky prompts in a restricted "execution sandbox" with stricter filters and limited token budget; only escalate successful outputs to broader systems.
• Honeypot prompts: Deploy obvious extraction triggers to detect automated extraction scanners and alert on access patterns.

Incident playbook (short, prescriptive)

1. Snapshot — freeze all logging for the affected session(s); copy model call traces and retrieval metadata to immutable storage.
2. Quarantine — revoke any active API keys used; suspend the affected model deployment or route to a “safe” model.
3. Scope — run provenance queries to see which datasets or connectors were included; examine canary records and dataset ledger.
4. Remediate — rotate keys, remove or reindex poisoned data, roll-back to last signed checkpoint.
5. Post-mortem — publish timeline, root cause, and remediation steps; add new automated tests to prevent recurrence.

Tip: keep a "forensics runbook" that scripts the collection of these artifacts to avoid time lost during an incident.

Quick templates & examples

Minimal context envelope (JSON-style)

{
  "system_config_id": "sys-20250829-v3",
  "system_seal": "sha256:abc...",
  "context": [
    {
      "id": "doc-17",
      "source": "s3://approved-bucket/reports/2025-05",
      "excerpt": "First 1024 chars...",
      "hash": "sha256:..."
    }
  ],
  "user": {
    "id": "u-42",
    "input": "Summarize vendor contract..."
  }
}

Rules:

• Reject any context entry whose hash is missing or mismatched.
• Limit excerpt size and prefer annotated snippets (highlights) rather than full documents.

Safe prompt scaffold (runtime)

=== SYSTEM (immutable) ===
purpose: "assistant confined to summarisation and citation-only"
policy: "do not provide personal data, do not perform actions"
=== RETRIEVALS ===
[doc-17 | s3://... | sha256:...]
=== USER REQUEST ===
"Summarize and list action items."
=== RULES (runtime enforced) ===
- if any citation unsupported -> append "UNSUPPORTED" marker and flag for review

Short checklist for engineers

• system prompts are versioned and immutable in CI
• every retrieval includes source-id & sha256
• pre/post filters active for all model calls
• RBAC and short-lived credentials for connectors & keys
• model artifacts signed; deployment verifies signatures
• baseline adversarial prompt corpus + CI tests
• monitoring for extraction, grounding, and prompt changes

Prioritization (what to do first)

• Immediate (days): enforce prompt immutability, enable pre/post filters, add retrieval metadata.
• Near-term (weeks): add signed artifacts, scoped short-lived keys, and basic anomaly alerts.
• Mid-term (months): evolve dataset ledger, honeypot canaries, and integrate DP/synthetic pipelines for sensitive training.