What Breaks When AI Agents Meet the Enterprise

By Brian Curry | Vector1 Research

Five lessons from building production agent systems, and where to find the full engineering playbook.

Most AI agent demos look great for about two weeks. Then someone from security asks which credentials the agent uses. Someone from audit asks who approved last Tuesday’s refund. Someone from finance asks why the model bill tripled. The demo has no answer to any of these, because it was never designed to.

That distance between a working prototype and a system a bank or hospital network will trust is the subject of my latest long-form guide on Vector1 Research, Engineering Agentic Systems in the Enterprise. It covers the architecture, the code and the operating model for running agents inside a large organization. It is the second part of a series that started with The Engineer’s Guide to Generative AI, a full-stack walkthrough from prompting to serving.

The full guide is long and code-heavy. Here are five ideas from it that you can put to work this week.

1. Set autonomy per action, not per agent

Teams often argue about whether an agent should be “autonomous.” That framing is too coarse. A single agent might safely read records on its own, update ticket fields within limits, and only propose refunds for a human to approve.

A simple ladder helps:

1. Assist. The agent gathers information and drafts. A human does every action.
2. Recommend. The agent proposes a specific action and a human approves it.
3. Act within limits. The agent acts inside defined thresholds and escalates the rest.
4. Act and audit. The agent acts on its own and humans review a sample afterward.

Start new action types at level 1 or 2. Move them up only when approval data shows humans are accepting the agent’s proposals almost unchanged.

2. Every action needs two identities

If your agent calls every API with one powerful service account, anyone who can talk to the agent can indirectly do anything that account can do. Your audit log will also show only the robot.

Every action should carry two identities: the person or process the agent acts for, and the agent itself. The permission the agent actually gets should be the intersection of what that person may do, what the agent is allowed to do, and what the current task needs. OAuth 2.0 token exchange is the standard way to mint short-lived credentials that carry both.

3. Keep approval rules out of the prompt

A system prompt that says “never issue refunds over $500” is a suggestion. A check in your code is a control. Whether an action needs a human should be decided by the platform, using the tool’s risk level, the amount involved and the agent’s configured limits.

def needs_approval(tool_risk: str, amount: float, autonomous_limit: float) -> bool:
    if tool_risk == "read":
        return False
    if tool_risk == "irreversible":
        return True
    return amount > autonomous_limit

The full guide expands this into a complete policy function with kill switches, principal authority checks and per-tool autonomy settings.

4. Grade the end state, and grade it more than once

An agent that writes a perfect summary while leaving your ERP in the wrong state has failed. Evaluate agents by checking the actual system state in a sandbox, then check the path they took: required steps in order, no forbidden actions, no policy bypasses.

Then run every test case several times. Agents are non-deterministic, and an agent that succeeds four times out of five looks fine on a dashboard. On a process that runs ten thousand times a month, that is two thousand failures. Track whether an agent passes all k attempts on a case, not only its average pass rate.

5. Treat everything the agent reads as a possible attacker

Agents read emails, tickets, vendor messages and web pages. Any of those can carry hidden instructions. Delimiters in the prompt help a little. Architecture helps much more.

One practical control: once an agent has read content from an untrusted source, require approval for any outbound action (sending email, messaging a vendor, sharing a file) for the rest of that task. It adds some friction, and it cuts off the most dangerous pattern, where untrusted input, private data and a way to send data out all meet in one agent.

What the full guide adds

These five ideas are the surface. The complete guide on Vector1 Research goes much further, with working Python for each piece:

• A platform architecture that separates control, execution, tool and knowledge planes, with a full reference diagram
• Declarative agent definitions that make every agent reviewable and every governance question answerable with a query
• A governed MCP server example showing how system owners publish tools as business operations
• Durable agent workflows on Temporal that survive crashes, wait days for human approval and never double-issue a refund
• The complete approval policy engine behind lesson 3
• An agent evaluation harness that grades outcomes, trajectories and consistency
• A tamper-evident, hash-chained audit log that satisfies auditors
• A threat model for agents, from goal hijacking to memory poisoning and denial of wallet
• A governance-as-code release gate that turns compliance into a pipeline step

If you are an engineer, data scientist or technical leader moving agents from pilot to production, it is written for you.

Read the full guide, Engineering Agentic Systems in the Enterprise guide, on Vector1 Research: https://brianjamescurry.substack.com/p/engineering-agentic-systems-in-the

New to building with LLMs? Start with part one, The Engineer’s Guide to Generative AI, which covers prompting, tool use, retrieval, evaluation, fine-tuning, serving and security end to end: [insert Substack link]

About the Author

Brian Curry is a Kansas City-based machine learning and AI strategist, marketing economist, and applied researcher. He leads Vector1, an independent research studio working on cognitive AI, causal inference, marketing economics, and knowledge engineering, with a focus on simulation-driven intelligence systems.

Over more than twenty years, Brian has led data science and analytics initiatives at enterprises ranging from $4B to $125B in revenue, including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, and AT&T. His work applies economics and causal inference to connect marketing investment to structural outcomes, operational decisions to economic value, and AI adoption to durable competitive advantage.

He is also the founder of KC AI Lab, a 1,500-member AI and ML education community in the Midwest. He lives in Kansas City and is an active part of the Kansas City community.

Research and Open-Source Projects

The modeling approach in this paper draws on PyCausalSim and Papilon.

Causal Inference and Economics

• PyCausalSim: Simulation-driven causal discovery and evaluation, using counterfactual scenarios to isolate structural drivers.
• Papilon: Complex-system simulation and optimization, with applications in media mix modeling and adaptive architectures.
• Daedalus: Causal analytics for money-flow mapping and macroeconomic tracking.
• The Autonomy Economy: A macroeconomic framework for the combined effects of rapid productivity growth and structural price deflation in an AI-automated economy.

Forecasting and Applied Modeling

• Regimecast: Time-series regime detection, structural breaks, and economic forecasting.
• CarryNet: Financial modeling for commodity carry trade analysis.
• Foodmesh-ml: Machine learning, optimization, and simulation for distributed food systems.

Cognitive AI and Semantic Architecture

• Memory-Node Encapsulation (MNE): A cognitive AI architecture and data structure for artificial episodic memory (NeoCortex-M).
• MeaningFlow: Semantic content modeling, embeddings, clustering, and coverage gap analysis.

Contact: brian@vector1.ai | GitHub @Bodhi8 | Substack

The text of this article may be shared with attribution under Creative Commons Attribution 4.0 International (CC BY 4.0). Code and models referenced here are available separately to paid subscribers.