Prompt Injection Is Now a Weaponized Attack Class — and Your AI Governance Framework Has No Answer for It

Executive Summary

Prompt injection has graduated from research curiosity to active, weaponized attack class — and most organizations deploying AI tools haven't acknowledged it exists, let alone built governance around it. This week's headlines (Microsoft's MCP tool-poisoning research, the Gaslight macOS malware, Anthropic's Fable 5 jailbreak) aren't anomalies. They're the pattern. The NIST AI RMF gives you the exact structure to address this: MAP identifies the attack surface, MEASURE defines how you test for it, MANAGE builds the response layer. If your organization is running AI-assisted tools with no documented threat model and no adversarial testing evidence in your risk register, this guide is where you start.

---

The Real-World Reality

Here's where I keep landing when I look at how organizations are actually deploying AI tools: the security posture for the AI layer is roughly where network security was in 2003. Something new and powerful showed up, everyone plugged it in, and the governance conversation got deferred until later.

"Later" is here.

This week gave us a convergence of prompt injection research and live exploitation that's hard to ignore. Microsoft published research on tool-poisoning attacks against Model Context Protocol (MCP) — the same protocol that lets AI agents call external tools, read files, and take actions. Researchers demonstrated that a malicious MCP server can inject hidden instructions into an AI agent's context window and redirect its behavior without the user seeing anything unusual. Separately, the Gaslight macOS malware showed up using prompt injection specifically to mislead AI-powered security analyst tools — not to bypass your SIEM, but to corrupt the AI layer interpreting what your SIEM is telling you. And Anthropic's Claude 3.5 Sonnet had a documented jailbreak (Fable 5) within days of release.

This is the specific risk that gets my attention as a security engineer: your AI-assisted tools are now attack surfaces, and most organizations have no governance layer that even acknowledges that.

Consider what this looks like in practice across environments:

Enterprise: Your security copilot ingests a phishing email as part of triage. That email contains a hidden prompt injection payload. The AI summarizes it as "routine marketing email, no action required" and closes the ticket. Your analyst trusts the summary. The attacker is now inside.

Healthcare: An AI-assisted clinical documentation tool processes patient notes. An attacker who has tampered with the input data — or a supply chain compromise in the underlying model — causes the AI to misclassify or omit critical findings. No one flagged it because the AI output looked authoritative.

OT/ICS: An AI-assisted anomaly detection tool monitors your operational network. Prompt injection through a poisoned data feed causes it to suppress alerts for a specific device class. From the Purdue model perspective, you've lost visibility at exactly the level where you need it most.

The gap isn't technical complexity. The gap is that most organizations haven't built the governance layer that would even surface this as a risk.

---

NIST AI RMF Application

The NIST AI RMF isn't a checklist. It's a reasoning structure. And it maps cleanly to prompt injection as an operational risk — if you actually use it.

GOVERN: Set the Policy Foundation Before You Deploy

GOVERN is where you establish that AI systems in your organization have an explicitly defined threat model — not just a data privacy section in the vendor's terms of service.

The specific gap I see here: most AI governance programs have a GOVERN layer that addresses data handling, bias, and fairness. Almost none have a GOVERN-layer policy that explicitly names adversarial inputs and prompt injection as in-scope risks requiring documented controls.

What needs to exist at the GOVERN layer for any AI system that:

• Ingests external, untrusted, or user-controlled input
• Has access to tools, APIs, or downstream systems (agentic workflows)
• Produces outputs that inform operational decisions

Required GOVERN-layer artifacts:

• An AI-specific threat model that names prompt injection, tool-poisoning, and indirect injection as attack vectors
• A documented risk tolerance statement: what kinds of AI failures are acceptable, which are categorically not
• Defined accountability — who owns AI security review, separate from who owns model selection
• A policy requiring adversarial testing evidence before production deployment of any AI system in scope

If your GOVERN layer doesn't name the threat, your MAP and MEASURE functions won't look for it. That's the structural failure.

MAP: Identify the Actual Attack Surface

MAP is where organizations identify and contextualize AI risks. For prompt injection, the MAP function requires a different mental model than traditional vulnerability management — and this is where the security practitioner's lens earns its keep.

The attack surface for prompt injection isn't a CVE. It's wherever untrusted data enters the context window of an AI system that has authority to take actions or produce trusted outputs.

MAP-level questions for any AI deployment:

1. What data sources does this AI system ingest? Which of those are controlled by the organization vs. external vs. user-supplied?
2. Does this AI system have tool access — APIs, file systems, external services, ticketing systems? What can it actually do?
3. What happens downstream when this AI system produces an output? Does a human review it, or does it flow directly into another system or workflow?
4. Is the AI system's output treated as authoritative by the people or systems consuming it?

The answers to these questions define your injection surface. An AI chatbot that summarizes internal documentation has a narrow surface. An AI security analyst tool that ingests external threat intel, has access to your ticketing system, and produces recommendations that analysts act on without secondary review — that's a materially different risk profile and the MAP function should reflect that.

OT-specific consideration: If you're deploying AI-assisted monitoring in an OT environment, the MAP function must account for the Purdue model. An AI tool with visibility into Level 2 (SCADA/HMI data) that can be manipulated into suppressing alerts is a safety and operational risk, not just a data security risk. The consequence model is different and the MAP documentation should say so explicitly.

Common MAP pitfall: Documenting the AI system's intended function without documenting its actual capability. The system prompt says "summarize incident tickets." The deployed tool has read access to your ticketing system, write access to close tickets, and integration with your Slack workspace. MAP those actual capabilities — not the vendor's marketing description.

MEASURE: You Have to Test for This

MEASURE is where most organizations have their biggest gap. The AI governance question isn't just "does this model perform well on our benchmark?" It's "how does this model behave when someone is actively trying to hijack it?"

Adversarial testing for prompt injection is not the same as model accuracy testing. It requires:

Red-team prompt testing — systematically crafting inputs designed to override system instructions, extract confidential context, or redirect model behavior. This is where an offensive security background is directly applicable: you're threat-modeling the input surface the same way you'd threat-model an API endpoint.

Indirect injection testing — testing whether the model can be hijacked through data it retrieves, not just through direct user input. This is the harder and more realistic attack scenario. The Gaslight malware didn't attack the user's prompt — it attacked the data the AI was analyzing.

Tool-call auditing — if the AI system has agentic capabilities, logging and reviewing what tool calls it actually made, not just what it reported to the user.

Behavioral baseline and drift monitoring — establishing what normal model behavior looks like under representative inputs, so anomalous outputs (AI refusing to flag a finding, AI producing outputs inconsistent with its system prompt) can be detected.

The MEASURE function should produce documented evidence: test cases run, results, anomalies observed, and disposition of those anomalies. That documentation goes into your risk register. If your risk register has no adversarial testing artifacts for your deployed AI systems, your MEASURE function is incomplete.

MEASURE pitfall: Treating vendor security documentation as a substitute for your own testing. A vendor's SOC 2 report tells you about their infrastructure security. It tells you nothing about how the model responds to adversarial inputs in your specific deployment context with your specific data and integrations.

MANAGE: Build the Response Layer Before You Need It

MANAGE is your operational layer — incident response, monitoring, containment, and continuous improvement. For prompt injection specifically, MANAGE answers: "What do we do when our AI system gets hijacked?"

Most organizations have no answer to that question. That's a governance failure.

MANAGE-layer controls for prompt injection risk:

• AI-specific incident response procedures: What constitutes an AI security incident? Who declares it? What's the containment procedure — can you isolate the AI system from its tool integrations without taking down the broader service?
• Output monitoring: Are there anomaly detection mechanisms on the AI system's outputs — not just its inputs? Unexpected refusals, outputs that contradict system instructions, tool calls that don't match the user's stated request — these are behavioral signals that can surface injection attacks.
• Rollback and containment: If you determine an AI system has been compromised or manipulated, can you roll back to a known-good state? What does that even mean for a deployed LLM with memory or session context?
• Human-in-the-loop checkpoints: For high-consequence decisions (security triage, clinical documentation, OT alert handling), MANAGE should define explicit review gates where a human validates AI output before it drives action — especially for novel or high-priority findings.
• Lessons-learned loop back to GOVERN and MAP: When an incident occurs or a test surfaces a vulnerability, the MANAGE function should have a defined path to update the threat model and risk register — closing the loop.

---

Real-World Implementation

Getting this from framework to practice doesn't require a 12-month governance transformation. Here's what a realistic starting sequence looks like.

Weeks 1–2: Inventory and triage Start with a simple question: what AI systems are deployed in your organization, and which of them ingest external or user-controlled input? That list is your initial attack surface inventory. Prioritize systems that have tool access or whose outputs inform operational decisions without mandatory human review.

Weeks 3–4: Threat model the top three Pick the three highest-priority AI deployments from your inventory. For each one, document: what data enters the context window, what the system can do (actual capabilities, not intended use), and what happens downstream from its outputs. This is your MAP work.

Month 2: Adversarial testing Run basic prompt injection testing against those three systems. You don't need a specialized red team for the first pass — start with known public prompt injection patterns and indirect injection scenarios relevant to the data sources the system ingests. Document what you find.

Month 3: MANAGE layer Draft AI-specific incident response procedures for the systems in scope. Define what an AI security incident looks like, who owns it, and what the containment steps are. This doesn't have to be long — it has to exist and be actionable.

Ongoing: Establish a review cadence — quarterly at minimum — to retest as models are updated, integrations change, and new attack techniques emerge.

Success metrics: Risk register has adversarial testing artifacts for all AI systems in scope. AI incident response procedure exists and has been tabletop-tested. At least one human review checkpoint is defined for each high-consequence AI workflow.

---

Quick-Start Checklist

GOVERN

• [ ] AI threat model exists and explicitly names prompt injection and adversarial inputs as in-scope risks
• [ ] Policy requires adversarial testing before production deployment of AI systems that ingest external input
• [ ] Accountability is assigned for AI security review

MAP

• [ ] AI system inventory documents actual capabilities (tool access, API integrations), not just intended use
• [ ] Data sources for each AI system are documented with trust classification (internal/external/user-supplied)
• [ ] Downstream consequence model is documented for each deployment

MEASURE

• [ ] Adversarial testing has been conducted for AI systems with external input exposure
• [ ] Risk register contains adversarial testing artifacts and documented findings
• [ ] Vendor security documentation has not been substituted for internal testing

MANAGE

• [ ] AI-specific incident response procedure exists
• [ ] Human review checkpoints are defined for high-consequence AI workflows
• [ ] Process exists to update threat model and risk register from incident findings

Red flags that require immediate attention:

• AI system with tool access and no human review gate on its outputs
• No adversarial testing conducted prior to or since deployment
• AI security incidents would be handled under a generic IT incident procedure with no AI-specific steps

---

Next Steps

Prompt injection is in the news this week. It'll be in the news next week. The organizations that get ahead of it aren't waiting for the governance conversation to become mandatory — they're doing the MAP and MEASURE work now, while it's still a deliberate choice rather than an incident response.

I'm working through this framework in the open, and the GRC platform at ai-governance-solutions.com is where I'm building and pressure-testing it. The free preview runs your organization against the NIST AI RMF and surfaces your top three risk findings — including whether your AI deployments have documented threat models and adversarial testing evidence. No credit card, instant results.

If you want to see where your organization actually stands on AI governance maturity, that's the fastest honest answer I can point you to: ai-governance-solutions.com.

---

Have a specific AI deployment you're trying to threat-model — an AI-assisted security tool, an agentic workflow, something in an OT-adjacent environment? Drop it in the comments or reach out. I'm genuinely working through these scenarios as I build the platform, and real-world use cases are exactly what makes it better.