AI Safety and Governance in the Enterprise: A Practitioner’s Framework

The standard AI safety article is a philosophical essay. Misalignment risks. Existential considerations. The trolley problem in a server rack. These pieces are not without value, but they are not what an engineer needs at 9 a.m. on a Monday when the model in production has started behaving differently from how it behaved on Friday.

What an engineer needs is a framework. An actual, mechanical, this-is-how-you-build-it framework that turns abstract safety concerns into concrete systems with names, owners, alarms, and runbooks.

This article is that framework.

It comes from working on AI systems in regulated environments — places where the question “can you prove what this system did and why” is not rhetorical. It is asked by auditors, by regulators, by general counsel, and by the engineering manager whose job is on the line if the answer is “I don’t know.” Operating in those environments forces a particular discipline. You cannot wave at safety. You have to instrument it.

The framework has four control planes:

Pre-deployment controls — what the model and the system around it must demonstrate before they can be turned on.
Runtime controls — guardrails that operate on every request and response, in real time, in production.
Observability controls — the telemetry, drift detection, and audit infrastructure that lets you know what the system actually did.
Lifecycle controls — the processes that govern how a model enters production, how it is retired, and what happens to its outputs in between.

Each plane is engineered separately. Each one is owned by someone. Each one fails in characteristic ways, and a serious AI system has explicit answers for those failures.

Why the framework looks like this

The temptation in any safety conversation is to start from threats and work backward. That ordering produces fragmented controls — a guardrail bolted on here, a logging hook there, an audit field added when an auditor asks for it. The system becomes a museum of past incidents, none of which fit together.

The framework above starts from the lifecycle of an AI system instead. A system has to be approved before it runs. It has to be governed while it runs. It has to be observable so that running can be assessed. And it has to have an end — every model, eventually, gets turned off. Each control plane is the engineering response to one phase of that lifecycle, and they compose because the lifecycle composes.

This is also why the framework treats governance as engineering. Policy documents that describe controls without naming the systems that enforce them are documentation, not governance. Governance is what is actually implemented in code, configuration, and infrastructure. Everything else is aspiration.

Plane 1: Pre-deployment controls

Before a model can be deployed into a production AI system, it has to pass through controls that establish three things: what it is, what it does, and what it is not allowed to do. The controls are not vibes. They are artifacts, gated by approvals, with named owners.

The model card, but operational

Every model in the system has a model card. This is not the academic model card from the original paper, useful as that template is. The operational model card is shorter, sharper, and explicitly mapped to the controls that depend on it. It records:

The model’s identity, version, and provenance — where it came from, who trained it, and on what.
The intended use cases and the explicitly out-of-scope use cases.
The training data lineage, including any data classes (PII, regulated content, customer data) that were present.
The known failure modes from pre-deployment evaluation, including the inputs that triggered them.
The ownership: who is the technical owner, who is the business owner, who signs off on changes.

The reason the model card is a control rather than a document is that downstream systems read from it. The runtime guardrails look up the model’s allowed use cases. The observability layer compares production traffic against the documented input distributions. The audit pipeline references the lineage. A model card that is just a Markdown file in a repo is documentation. A model card whose fields are queried by other systems is governance.

Pre-deployment evaluation

Before a model handles a single production request, it passes through evaluation. The evaluation suite is specific to the use case, but the structure is universal:

Capability evaluation — does the model do the thing it claims to do, on the data distribution it claims to handle?
Safety evaluation — does the model refuse the things it should refuse, and does it produce safe outputs in adversarial conditions?
Bias and fairness evaluation — does performance degrade on protected subgroups, and if so, by how much?
Robustness evaluation — what happens at the edges of the input distribution, on malformed inputs, on inputs designed to elicit failures?

Each evaluation produces a numeric score and a pass/fail against a documented threshold. The thresholds are set by the model’s owners and reviewed by whoever sits in the safety governance role. A model that fails any evaluation cannot be promoted out of the development environment without an explicit, signed override that becomes part of the audit record.

A reasonable structure for the evaluation harness:

@dataclass
class EvaluationResult:
    suite: str                      # "capability", "safety", "bias", "robustness"
    metric: str                     # "exact_match_f1", "refusal_rate", etc.
    value: float
    threshold: float
    passed: bool
    sample_size: int
    eval_dataset_version: str
    evaluated_at: datetime
    evaluator: str                  # who/what ran this evaluation

@dataclass
class ModelGate:
    model_id: str
    model_version: str
    results: list[EvaluationResult]
    overrides: list[Override]       # signed overrides for failures
    decision: Literal["promote", "block", "conditional"]
    decision_rationale: str
    decided_by: str
    decided_at: datetime

The structure matters because the evaluation results are themselves audit artifacts. When a model is in production six months later and someone asks why it was deployed, the answer is not a Slack thread. It is the ModelGate record, with its evaluations, overrides, and signatures.

The deployment manifest

The model itself is one part of an AI system. The deployment manifest is the rest. It records, as configuration:

The runtime guardrails that apply to this model (which I’ll cover in the next section).
The rate limits, quotas, and access controls.
The data classes the system is permitted to handle.
The downstream systems it is permitted to call.
The retention and logging policies for inputs and outputs.

The manifest is version-controlled and tied to the model version. Promoting a model into production means promoting a (model_version, manifest_version) pair. They are atomic. There is no scenario in which a model is in production with a different manifest than the one it was approved with — the deployment system enforces the pairing at the infrastructure level.

This is the part of pre-deployment that most teams skip. They version the model. They version the code. They do not version the policy that governs how the model is allowed to be used. When something goes wrong, the model and the code can be rolled back, but the policy that was in effect at the time of the incident is reconstructed from memory and Slack screenshots. That gap is where audit failures live.

Plane 2: Runtime controls

Once a model is in production, every request and every response passes through a layer of runtime controls. These are the guardrails. They are not the safety strategy — they are the last line of a safety strategy that mostly lives elsewhere — but they are the layer that turns approved policy into enforced behavior.

Input controls

Every input is inspected before it reaches the model. The inspection is not a single function. It is a pipeline of small, composable checks, each of which can pass, flag, or reject the request.

@dataclass
class InputCheck:
    name: str
    severity: Literal["info", "warn", "block"]
    decision: Literal["pass", "flag", "reject"]
    rationale: str
    metadata: dict

def check_input(request: Request, manifest: Manifest) -> list[InputCheck]:
    checks = []

    # Schema validation — is this even a well-formed request?
    checks.append(validate_schema(request, manifest.input_schema))

    # Authorization — does this caller have permission to invoke this model?
    checks.append(check_authorization(request, manifest.access_policy))

    # Data class — does the input contain regulated content this model
    # is not approved to handle?
    checks.append(scan_data_classes(request, manifest.allowed_data_classes))

    # Prompt injection / jailbreak — is the input attempting to subvert
    # the model's instructions?
    checks.append(scan_for_injection(request))

    # Rate and quota — is this caller within their budget?
    checks.append(check_quota(request))

    return checks

The checks are independent and pluggable. Each one is owned by a specific team — the data-class scanner is owned by the data-governance team, the injection scanner is typically owned by the security team, the quota check is owned by the platform team. When a check fires in production, the alert routes to the team that owns it, not to a generic on-call rotation.

A request that fails a block-severity check does not reach the model. A request that produces warn-severity flags reaches the model but is logged with elevated priority. A request that produces only info flags is processed normally and the flags become metadata in the audit log.

Output controls

The model’s response is then inspected before it returns to the caller. The same composable structure applies, with different checks:

Hallucination and grounding — does the response cite sources that exist? When the model claims a fact, is the fact supported by the retrieved context?
Data exfiltration — does the response contain data classes the caller is not authorized to receive?
Toxicity and content safety — does the response contain content that violates the system’s content policy?
Schema and contract — does the response match the structure the caller expects?

The grounding check deserves emphasis because it is the one most relevant to the architectures from earlier in this series. If the model is supposed to answer from retrieved context, the output should be checkable against that context. A grounding check that flags responses making claims the context does not support is one of the highest-leverage runtime controls a knowledge system can have. It does not catch every hallucination, but it catches the hallucinations that matter most: the ones that confidently assert things the system has no basis for.

Fallback and degradation

Runtime controls fail. Sometimes a check times out. Sometimes the underlying model is unavailable. Sometimes a guardrail itself has a bug. The system has to keep functioning, or fail in a way that is itself safe.

The right pattern is explicit degradation modes, declared in advance:

Strict mode — every check must pass. If any check is unavailable, the request is rejected. Used for high-stakes operations.
Standard mode — block-severity checks must pass, warnings degrade gracefully, info checks can fail silently. The default for most production traffic.
Available mode — only schema and authorization checks are enforced; everything else is best-effort. Used during incidents, with explicit operator approval and time-bounded.

Switching between modes is itself an audited operation. There is no scenario in which the system silently falls back to a less safe mode. An on-call engineer can declare a degradation, but the declaration is recorded, time-bounded, and reviewed afterward.

Plane 3: Observability controls

A safety system you cannot observe is a safety system you cannot trust. Observability is the third plane and the one that ties the others together — it is the layer that lets you tell, after the fact and in real time, what the system actually did.

Three categories matter: telemetry, drift detection, and audit.

Telemetry that is actually useful

Most production AI systems emit metrics, but the metrics emitted are usually the wrong ones. Latency and error rate are operational metrics. They tell you whether the system is up. They do not tell you whether it is behaving.

Useful telemetry covers:

Per-request decisions — which input checks fired, which output checks fired, what was the model’s confidence, was the response grounded, was it served from cache.
Distribution metrics — what does the input distribution look like today versus last week, what does the output length distribution look like, what does the topic distribution look like.
Guardrail effectiveness — what fraction of requests are flagged, blocked, or warned, and how does that fraction change over time.
Cost and budget — token consumption per caller, per use case, per model, with explicit budget tracking.

This data is logged structured, not as free text. Every log line is queryable. Every log line carries the request ID, the model version, the manifest version, and the user or system that originated the request. When something goes wrong, the investigation does not start with “go grep the logs.” It starts with a query against a structured store.

@dataclass
class RequestTrace:
    request_id: str
    timestamp: datetime
    caller_id: str
    model_id: str
    model_version: str
    manifest_version: str
    input_checks: list[InputCheck]
    output_checks: list[OutputCheck]
    retrieval_trace: Optional[RetrievalTrace]   # for RAG systems
    memory_trace: Optional[MemoryTrace]          # for agentic systems
    response_metadata: dict
    cost_units: float
    latency_ms: float
    degradation_mode: str

Notice the retrieval and memory traces. For systems built on the architectures in earlier articles in this series, the audit record includes which chunks were retrieved and which memories were surfaced. When a response is questioned, the investigation can replay the exact context the model saw. This is what it means for grounding to be auditable rather than aspirational.

Drift detection

Models drift. The world changes around them, the input distribution shifts, the population of users evolves. A model that was performing well in March may be performing poorly in September without anyone noticing, because aggregate metrics like accuracy hold up while the distribution of failures shifts underneath them.

Drift detection is its own discipline, and the controls operate on three layers:

Input drift. Is the distribution of incoming requests today statistically different from the distribution the model was evaluated on? Embedding-based drift detectors are cheap to run: encode a sample of recent requests, compare the centroid and variance against the reference set, raise an alert when divergence exceeds a threshold.

Output drift. Is the model producing different kinds of responses than it used to? A jump in average response length, a shift in refusal rate, a change in the distribution of confidence scores — all of these are signals that something has changed, even if no individual response is obviously wrong.

Performance drift. Where you have ground truth, are the metrics holding? For tasks with eventual labels (a classification that is later reviewed by a human, a recommendation that is later acted on or ignored), maintain a rolling evaluation against the ground-truth stream and alert when the metric degrades.

Drift alerts are not pages. They are findings, surfaced to the model’s owners, with enough detail that the owner can decide whether to investigate, retrain, or roll back. The most common mistake in drift detection is making the alerts too sensitive — drift detectors that fire constantly get ignored, and a drift detector that is ignored is worse than no drift detector, because it creates the illusion of coverage.

Audit trail

The audit trail is the long-term record of what the system did. It is not the same as the telemetry log, although it is built from the same events. The audit trail is what you hand to a regulator, a court, or an internal investigator. It has different retention, different access controls, and different integrity guarantees.

Three properties matter:

Completeness. Every production decision the system made — every model invocation, every guardrail firing, every degradation, every override — is in the audit trail. Not 99% of them. All of them.
Immutability. Audit records cannot be modified after the fact. Storage is append-only or write-once-read-many. Operators do not have permission to delete or alter audit records, and any attempt to do so is itself an audit event.
Reconstructability. Given an audit record, you can reconstruct what happened. The model version, the manifest version, the input, the retrieved context, the output, the check results, the caller. Anything that was decision-relevant is in the record.

The integrity guarantee is the part that takes work. Audit logs that live in the same database as operational data, with the same access controls, do not satisfy the requirement. A serious audit infrastructure uses separate storage with separate access controls, hash-chained records for tamper-evidence, and a documented chain of custody. The bar is whether you would be comfortable producing the records under subpoena. If the answer is no, the audit infrastructure is not done.

Plane 4: Lifecycle controls

The fourth plane is the one that organizations skip until they cannot. Models enter production. Models live in production. Models leave production. Each transition needs a process, and the absence of process is what produces the long tail of stale models running quietly somewhere because nobody remembered to turn them off.

Promotion

A model moves from development to staging to production through promotion gates. Each gate has explicit criteria. A model in development can be experimented on freely. To reach staging, it must have a complete model card, pass its evaluations, and have a documented manifest. To reach production, it must additionally pass an integration review with the runtime guardrails and observability layer, and it must have a named on-call owner.

Promotion is not a button. It is a workflow with named approvers, recorded in a system that is itself part of the audit infrastructure. “We promoted the model” should be a query you can run, not a memory.

Change management

A model in production is not static. The prompt template gets updated. The retrieval system gets tuned. The model itself gets re-trained or replaced with a new version. Each of these is a change, and changes need to be governed.

The right pattern is shadow deployments and progressive rollouts. A new model version runs alongside the old, receiving the same traffic but not affecting the response. Telemetry from both is compared. If the new version is at least as good across the relevant metrics, traffic shifts gradually — 1%, 10%, 50%, 100% — with each step gated by metrics. If anything regresses, the rollout pauses and the old version continues serving.

Manifest changes follow the same logic. Updating an allowed-data-class list, changing a rate limit, adjusting a guardrail threshold — each is a change, each goes through review, each is rolled out with monitoring.

Incident response

When something goes wrong, the response process is documented and rehearsed. The same incident management practices that apply to any production system apply here, with a few additions specific to AI:

Rollback paths exist for both the model and the manifest. Rolling back to “last known good” means rolling back the pair, atomically.
Output recall is a defined operation. When a model produces a bad output that has been delivered downstream — into a database, an email, a customer-facing UI — there is a documented process for identifying and, where possible, recalling those outputs.
Post-incident review covers safety-relevant questions. Was the failure mode in the pre-deployment evaluation suite, and if not, why? Did the runtime guardrails fire, and if not, why? Did observability surface the problem, and if not, why?

The post-incident review feeds back into the pre-deployment evaluations. The failure mode that was not in the test suite gets added. The check that did not fire gets fixed. The metric that did not detect the drift gets adjusted. This is how the framework learns over time. Every incident is a free piece of research about where the system is weakest.

Retirement

Every model eventually retires. The retirement process matters, and it is the one most often handled badly.

A model that is decommissioned without a process leaves a trail of orphans: callers still pointing at the now-defunct endpoint, audit records that reference a model nobody can identify, downstream systems with cached responses that nobody knows are stale. A retirement process closes those loose ends:

Deprecation is announced and time-bounded. Callers get notice, with a date.
Traffic migration is monitored. During the deprecation period, callers are moved to the replacement model and the old model’s traffic is tracked down to zero.
Audit records are preserved. The model is gone, but the audit trail of what it did is retained for the documented retention period.
Model artifacts are archived. The model weights, training data manifest, evaluation results, and deployment manifest are preserved in cold storage. If, three years later, someone needs to reproduce a decision the model made, they can.

This is mundane work. It is also the work that separates AI systems that are operated from AI systems that are merely run.

Where humans fit

Every plane in this framework has human decision points, and the framework is silent on a question that matters: how much of this can be automated, and how much should not be?

The honest answer is that the framework is designed around automation as the default, with humans inserted at decision points where the cost of being wrong is asymmetric. Pre-deployment evaluation runs automatically; the decision to override a failed evaluation is a human one. Runtime guardrails fire automatically; the decision to switch into a degraded mode is a human one. Drift detectors run automatically; the decision to retrain or roll back is a human one. Audit records are written automatically; the decision to act on what they show is a human one.

This is not a compromise. It is the right shape. The volume of decisions in a production AI system is too high for humans to be in every loop, and the consequence of high-stakes mistakes is too large for humans to be in none of them. The framework’s job is to ensure that the humans are in the loops where their judgment is most valuable, and that they have the information they need when they get there.

What this looks like in practice

The framework as described is a target state, not a starting state. Most teams begin with one plane partially built, usually runtime guardrails, and discover the others under pressure — when an audit hits, when a regulator asks, when an incident surfaces a gap.

The order I have seen work is approximately:

Stand up observability first. You cannot govern what you cannot see, and standing up observability creates the data substrate the other planes depend on.
Then build pre-deployment controls. Once you can measure the system in production, you can define what “good enough to deploy” means in measurable terms.
Then formalize runtime controls. Most teams have ad-hoc guardrails by this point; the work is to make them composable, owned, and integrated with observability.
Then build lifecycle controls. By the time the first three are in place, the gaps in promotion, change management, and retirement are obvious.

Trying to build all four planes in parallel produces fragmentation. Building them in this sequence produces a system in which each plane reinforces the next.

Closing

AI safety in the enterprise is not a philosophical posture. It is an engineering discipline with named systems, named owners, and audit trails that survive the people who built them. The four-plane framework — pre-deployment, runtime, observability, lifecycle — is the structure that lets safety scale with the AI system rather than chasing it.

The piece that often gets missed is that this framework is not in tension with the architectures from earlier in this series. It is the substrate on which they run. The RAG system from Article 3 needs grounding checks at runtime and lineage in audit. The agentic system from Article 4 needs guardrails on tool calls and replayable memory traces. Without the safety framework, those systems are not deployable in any environment that takes regulation, audit, or accountability seriously. With it, they become systems an organization can actually run.

What’s next

The next article picks up at the boundary between an AI system and the rest of the enterprise — the API layer where models meet ERPs, CRMs, and content systems. Article 6, Serving AI at Scale: API Design and Integration Patterns for Enterprise Knowledge Systems, covers the practical engineering of that boundary: FastAPI patterns for classification and entity APIs, caching strategies that respect data classes, rate limiting that enforces the manifest from this article, versioning that doesn’t break callers, and circuit breakers that contain the blast radius when a model behind the API misbehaves. The last mile that most AI architecture articles skip.

Brian James Curry is an Enterprise AI Strategist, Data and Marketing Scientist, and Applied Researcher. He is the founder of Vector1 Research, an independent research lab focused on marketing economics, causal systems, cognitive AI architecture, and the economic implications of intelligent systems. He also founded KC AI Lab, a data science community and incubator for applied machine learning education.