Photograph: Anastassia Anufrieva / Unsplash

A pre-approval checklist for owners and C-level leaders: five questions that expose whether an AI agent project will reach production before you commit budget.

Key takeaways

• Most agent projects fail at the approval meeting, not in production - a confident demo hides a business case that was never going to reach the intended outcome.
• Force a named business metric with a known baseline; a project that cannot state today's number cannot prove tomorrow's improvement.
• Treat exception handling - detection, escalation to a named human, and containment of irreversible actions - as the core of the design, not a footnote.
• Decide deliberately what you own versus rent across data, model, infrastructure, and vendor, so a supplier's price or policy change does not become your crisis.
• Approve pilots as bounded experiments with pre-agreed success, stop, and review thresholds, removing the sunk-cost trap before it can form.

Most AI agent projects do not fail in production. They fail at approval - in the meeting where a confident deck, a plausible demo, and an enthusiastic sponsor combine to green-light work that was never going to reach the outcome anyone imagined. The demo works because a demo is a controlled environment. The business does not run in a controlled environment. By the time the gap becomes visible, the budget is committed, the team is invested, and the political cost of stopping is high.

The good news for anyone holding the approval decision is that the failure modes are predictable. Agent projects tend to die for a small number of recurring reasons, and each one can be exposed by a single question asked before money moves. You do not need to understand transformers, orchestration frameworks, or the difference between one model and another. You need to interrogate the business case with the same rigour you would apply to any capital allocation.

What follows is a pre-approval checklist built around five questions. They are deliberately blunt. Each one is framed so that an unconvincing answer is itself the finding: if the team cannot answer clearly, that is your signal, not a detail to resolve later. Treat vagueness as a red flag, not a rounding error.

1. What business metric moves - and by roughly how much?

The first question is the one most often skipped, because it feels obvious and rarely is. Ask which single number on the P&L or in operations is expected to change: cost per case handled, cycle time on a specific process, revenue captured from faster response, hours returned to a constrained team. If the answer is a capability rather than a metric - "it will handle customer queries," "it will summarise contracts" - you have a technology looking for a job, not a business case.

You are not asking for a fabricated forecast. You are asking for a named metric, a rough direction and magnitude, and a way to measure it that already exists or can be stood up cheaply. If the team cannot tell you what to measure, they cannot tell you whether the project worked. And if the honest expected impact is a marginal improvement to a metric nobody in the room cares about, the correct decision is to decline now and spend the attention elsewhere.

The baseline test

Before approval, insist on knowing the current value of the target metric. A project that cannot state today's number cannot prove tomorrow's improvement - and "we'll figure out measurement during the pilot" usually means measurement never happens.

2. Is the process stable and rule-describable enough to hand over?

Agents perform best on processes that are understood well enough to explain, yet repetitive enough to be worth automating. The trap sits at both extremes. A process that changes every quarter, depends on undocumented judgement, or lives entirely in one person's head will resist automation regardless of how capable the underlying model is. A process nobody can describe cannot be delegated to a machine, because you have nothing to check the machine against.

A useful diagnostic is to ask the team to write down the process as it actually runs - not the official version, the real one - including the exceptions. Two failure signals appear quickly. First, if the description keeps expanding with "it depends," the process is not yet stable enough to hand over. Second, if the exceptions turn out to be the majority of the volume, you are automating the easy minority and leaving the hard, expensive part untouched.

An agent cannot be more reliable than your understanding of the work you are asking it to do.

3. What happens on an exception - and who catches it?

Every serious agent deployment lives or dies on how it behaves when it is uncertain or wrong. The demo shows the happy path. Production is mostly edge cases, ambiguity, and inputs nobody anticipated. So the question that separates a real plan from a hopeful one is simple: when the agent is not confident, what does it do, and which human picks it up?

A credible answer has three parts, and you should ask for all three explicitly.

1. Detection - how the system recognises it is out of its depth, rather than confidently proceeding with a wrong answer.
2. Escalation - the named role or queue that receives the handoff, and whether that person has the time and authority to act on it.
3. Containment - what the agent is prevented from doing autonomously, so that a mistake is recoverable rather than irreversible.

If the escalation path is "a human will review everything," you have not automated anything - you have added a machine step in front of the same amount of human work. If the path is "the agent handles it end to end" for a process where errors are costly or hard to reverse, you are carrying more risk than the business case admits. The right design usually sits between the two, and knowing where is part of what you are approving.

Ask about the worst plausible mistake

Not the average error - the worst one the agent could make before a human notices. If that mistake is expensive, public, or irreversible, the containment design is the project, not a footnote to it.

4. What do we own, and what are we renting?

Agent systems are assembled from parts you own and parts you rent: your data, a model, the infrastructure it runs on, and the vendors who tie it together. Each rented component is a dependency, and dependencies concentrate over time into leverage that someone else holds over you. The question to force into the open before approval is which parts you can change later without rebuilding the whole thing.

The specific exposures worth naming are your data - where it goes, who can see it, and whether using the system trains someone else's product; the model - whether the design assumes one specific provider or can substitute another as prices, terms, and capabilities shift; and the vendor - how hard it would be to leave, and whether your process logic and data would leave with you. You do not need to insist on owning everything; renting is often the right call early. You need to know what you are renting, so that a supplier's price change or policy change does not become your operational crisis. Optionality has a cost, and it is a decision to make deliberately at approval, not to discover under duress.

5. How will we know - within one pilot - whether to scale or stop?

The final question is the discipline that makes the other four safe to get wrong. Approve the project as a bounded experiment with a decision at the end, not as an open commitment that quietly becomes permanent. Before work starts, agree the success criteria, the timebox, and - this is the part teams resist - the conditions under which you will stop.

A well-formed pilot states in advance what result would justify scaling, what result would justify stopping, and what would count as neither so you already know it needs a second look. Tie those thresholds to the metric from the first question. The purpose is not bureaucratic. It is to remove the sunk-cost trap before it forms, so that stopping is a planned, respectable outcome rather than an admission of failure. A pilot you are not genuinely willing to stop is not a pilot. It is an unbudgeted programme wearing a pilot's clothes.

The decision in front of you

These five questions are not a scoring rubric, and a project does not need a perfect answer to every one. What they are is a filter for the specific kind of confidence that gets projects approved and then quietly abandoned. A team that can name the metric, describe the process honestly, plan for exceptions, map its dependencies, and commit to a stop condition has done the thinking that predicts success. A team that answers in capabilities and enthusiasm has not - and that is worth knowing before you sign, not after.

If a proposal in front of you leaves any of the five genuinely unanswered, the most valuable thing you can do is not reject it. It is to send it back with the specific question attached. The next decision is rarely go or no-go. It is whether the case is complete enough to decide at all.

Common questions

Do we need a precise ROI forecast before approving an AI agent project?

No - and a suspiciously precise forecast should make you more cautious, not less. What you need is a named metric, its current baseline, and a rough expected direction and magnitude of change. The discipline is in knowing what to measure and being able to measure it, not in manufacturing a number to the decimal point.

What is the single most common reason these projects fail after approval?

Underestimating exceptions. Demos show the happy path, but production is dominated by ambiguity and edge cases. When there is no clear plan for how the agent detects uncertainty, who it escalates to, and what it is prevented from doing on its own, small errors accumulate into a loss of trust that quietly kills the deployment.

Should we insist on owning the model and infrastructure rather than renting?

Not necessarily. Renting is often the right choice, especially early. The point is to make the decision deliberately: know where your data goes, whether the design assumes a single model provider, and how hard it would be to change vendors later. Optionality has a cost, and you should choose to pay it or not with eyes open - not discover the dependency under pressure.

Read this on keenagents.ai - written by Petar Denev, Co-founder & CEO.

Photograph: Simon Infanger / Unsplash

Most AI agents are demos judged by the happy path; a production system is judged by what happens when something goes wrong.

Key takeaways

• A demo proves an agent can succeed once; production requires it to fail safely and predictably at the boundaries of inputs nobody rehearsed.
• Observability is non-negotiable: if you cannot see what an agent did, what it cost, and where it went wrong, you are hoping rather than operating.
• Exception handling and human escalation are the core of a production system, not an add-on - and an escalation route no one staffs is worse than none.
• Integration with systems of record, least-privilege access, and predictable per-task cost are load-bearing requirements that demos hide.
• The decisive test is organisational: a named owner must be accountable for the agent after launch, able to pause it, watch for drift, and answer for its errors.

Nearly every AI agent you will be shown looks impressive. It runs a clean scenario, returns a fluent answer, and the room nods. This is the easy part. A demo is a controlled performance: the inputs are curated, the person driving knows exactly what to type, and nothing downstream depends on the result. The moment it ends, everyone stops watching. That is the tell. Nobody has to live with it afterwards.

Production is the opposite of a demo in every way that matters. It runs on inputs nobody rehearsed, at volumes nobody watched, while the person who built it is asleep or on holiday. It touches real records, real money, real customers. And when it goes wrong - which it will - the consequences land on someone specific, at a specific desk, on an ordinary Tuesday afternoon. The gap between these two states is not a matter of polish. It is a different discipline entirely, and most organisations underestimate it because the demo felt so finished.

For a decision-maker, the useful question is not "does it work?" but "would I let this run unattended against my systems of record, and who answers for it when it doesn't?" That reframing exposes everything a demo hides. Below is what production-readiness actually requires, expressed in terms you can hold a vendor or an internal team accountable to.

Reliability Is Behaviour Under Inputs You Didn't Choose

A demo proves an agent can succeed once. Production requires it to fail safely a thousand times. Real inputs are messy: half-filled forms, ambiguous requests, the customer who pastes an entire email thread into one field, the edge case your business has quietly tolerated for a decade. A production-ready agent is defined less by how well it handles the expected path and more by how it behaves when the input falls outside it.

The signal to look for is not a higher success rate. It is predictable behaviour at the boundaries. When the agent is uncertain, does it stop and ask, or does it invent an answer with total confidence? When it cannot complete a task, does it fail loudly and hand off, or does it fail silently and leave a half-finished action in your system? A confident wrong answer is far more dangerous than an honest "I'm not sure" - because the wrong answer looks exactly like a right one until the damage surfaces. So ask to see the agent run on your ugliest real inputs, not the vendor's clean ones.

If You Can't See It, You Don't Operate It

An agent you cannot observe is not something you run; it is something you hope. Production-readiness means every meaningful action leaves a trace you can inspect after the fact - not because auditors demand it, though they will, but because you cannot improve, debug, or defend a process you cannot see. Concretely, the operating team should be able to answer a few plain questions on demand:

• What did the agent do today, and how much did it cost to do it?
• Which decisions was it confident about, and which did it escalate?
• When something went wrong, what was the exact sequence of steps that led there?
• Is its behaviour drifting over time as inputs, models, or upstream systems change?

Without this, the first time you learn an agent has been misbehaving is when a customer complains or a number looks wrong in a report. By then the cost is already incurred and the trail may be cold. Observability is what converts a black box into an operable system - it is the difference between managing an agent and merely deploying one.

Exceptions and Escalation Are the Product, Not an Afterthought

In most real operations, the routine cases were never the hard part. The value and the risk both concentrate in the exceptions: the refund that breaches policy, the contract clause that doesn't fit the template, the request that is technically valid but commercially unwise. A demo skips these; a production system is largely defined by how it handles them. Three things must be explicit. First, the agent must know its own limits - the categories of decision it is not permitted to make alone. Second, when it hits one of those limits, there must be a clean handoff to a human with enough context that the person can act in seconds, not reconstruct the situation from scratch. Third, that human path must be staffed and owned, not a theoretical queue nobody watches. An escalation route that leads nowhere is worse than none, because it creates the appearance of a safety net while removing the vigilance.

A demo is judged by what it does when everything goes right. A production system is judged by what it does when something goes wrong.

Integration, Security, and Cost: The Unglamorous Load-Bearing Walls

Three things turn a clever prototype into an operational liability if they are ignored, and none of them show up in a demo.

Integration with systems of record

An agent that reads and writes to a spreadsheet in a sandbox is a toy. An agent that acts on your CRM, your ERP, your ticketing and billing systems is a participant in your operation - and it must respect the same constraints those systems impose: validation rules, state transitions, idempotency so a retry doesn't double-charge or double-book. Real integration is where most of the engineering actually lives, and it is precisely what a demo abstracts away.

Security and access

An agent inherits whatever it can reach. Give it broad credentials and it becomes a broad liability - a single point through which sensitive data can leak or unauthorised actions can occur. Production discipline means least-privilege access, scoped to exactly what the task requires, with clear rules about what data the agent may see, retain, and send onward. This is also where control over models and vendors matters: you should know where your data goes and be able to change that decision later without rebuilding everything.

Cost predictability

Agents can consume compute in ways that are hard to predict, because a single request may trigger many internal steps. In a demo, cost is invisible. In production, an agent that retries aggressively or reasons in long loops can turn an unremarkable day into an unpleasant invoice. You need cost visibility per task and sensible ceilings, so a runaway process is caught early rather than discovered on a bill.

The quiet failure mode

Unbounded cost and over-broad access rarely announce themselves in testing. They surface at scale, under real traffic - which is exactly when they are most expensive to fix.

The Real Test: Who Owns It on a Tuesday Afternoon?

Strip away the technology and one question separates a demo from a production system: is there a named person accountable for this agent after the applause stops? Someone who watches its behaviour, owns its errors, decides when to pause it, and is measured on whether it does its job. Software that no one operates does not stay working; it decays quietly as the world around it changes - models update, upstream systems shift, inputs evolve - until one day it is confidently wrong and no one noticed the drift. This is why production-readiness is an organisational property as much as a technical one: the reliability, the observability, the escalation paths, the access controls, and the cost ceilings all exist so that a human owner can hold the system to account without heroics. An agent is production-ready when someone can be handed it on a Tuesday and run it responsibly, not when it wins a demo on a Thursday.

The Next Decision

Before you approve any agent for real work, ask to see it fail. Ask for the audit trail, the escalation path, the access scope, and the cost per task. Ask who operates it after launch and how they will know it is drifting. If those answers are crisp, you are looking at a system you can stand behind. If the only thing anyone can show you is the happy path, you are looking at a demo - and the honest next step is to define what "accountable on a Tuesday" would require before you commit anything real to it.

Common questions

What is the single fastest way to tell a demo from a production-ready agent?

Ask to see it run on your own messy, real inputs rather than the curated examples in the presentation. A demo can only show the happy path. A production-ready system behaves predictably at the boundaries - it asks when uncertain, fails loudly rather than silently, and hands off cleanly when it hits its limits.

Do we need full observability before going live, or can we add it later?

Treat it as a prerequisite, not a follow-up. Without a trace of what the agent did, what it cost, and how a failure unfolded, the first sign of trouble is usually a customer complaint or a wrong number in a report - by which point the cost is already incurred and the trail may be cold. Observability is what lets a human owner operate the system at all.

How do we keep control of data, models, and vendors when deploying agents?

Start with least-privilege access so the agent can only reach what a specific task requires, and be explicit about what data it may see, retain, and send onward. Design so that where your data goes and which model or vendor you use are decisions you can change later without rebuilding everything. Control is a property you engineer in, not one you negotiate afterwards.

Read this on keenagents.ai - written by Petar Denev, Co-founder & CEO.

Photograph: Jakub Nawrot / Unsplash

Lock-in is usually filed as an engineering concern. It belongs on the board agenda, because it quietly sets your pricing power, your continuity, and your compliance position.

Key takeaways

• Lock-in is not primarily a technical inconvenience; it determines your pricing power, continuity exposure, and compliance position, which are board-level concerns.
• The AI frontier moves fast, so the ability to switch or blend models is a source of ongoing commercial advantage, not a one-time procurement decision.
• Model-agnostic architecture means the vendor sits behind a boundary you control, so swapping a provider is a configuration change rather than a rebuild.
• The real cost of lock-in shows up at renewal, during outages, and when regulators or customers ask where data goes and which model touched it.
• A short set of pointed procurement questions exposes switching cost before you sign, when you still have leverage.

Most organisations decide how they will use AI agents long before they notice they have also decided who holds the leverage. The choice arrives dressed as a technical detail: a model provider, a cloud region, a framework, a set of convenient APIs. Each looks like a sensible engineering call made under deadline. Stacked together, they quietly determine something the board actually cares about: how much freedom the business retains to change its mind.

Lock-in is the accumulation of those choices into a position you cannot easily leave. It rarely announces itself. It shows up later, at the moments when you have the least room to manoeuvre: a renewal negotiation where the vendor knows you cannot walk, an outage that stops a revenue-generating process, a regulator asking where your data was processed, or a competitor adopting a newer model while you are still tied to last year's. By then the cost of switching is no longer a technical estimate. It is a business constraint.

The reframing that matters is simple. Lock-in is not mainly about code that is hard to move. It is about pricing power, continuity, compliance, and the pace at which you can adopt what is better. Those are commercial questions, and they belong on the executive agenda, not buried in an architecture diagram.

What lock-in actually costs the business

It helps to separate the exposure into the outcomes a board already tracks, rather than the technical mechanisms underneath them.

• Pricing power. When a vendor knows switching is impractical, your renewal is negotiated from weakness. Prices tend to move in one direction once the cost of leaving is high enough, and the increase need not be dramatic to matter across a growing volume of usage.
• Continuity. If one provider's availability, rate limits, or policy changes can halt an operation that customers or revenue depend on, you have concentrated a business risk in a party you do not control.
• Compliance and data residency. Where data is processed, which jurisdiction governs it, and whether a given model may see certain information are increasingly contractual and regulatory questions. Being unable to move workloads is being unable to answer them.
• Pace of adoption. The capability frontier moves quickly and unevenly. If you cannot adopt a stronger or cheaper model when it appears, you inherit your vendor's roadmap and their timing instead of your own.

None of these require anything to go wrong technically. They are the ordinary consequences of holding a weak hand, and they compound as the business relies on the system more heavily.

Model-agnostic architecture, explained in business terms

The technical answer to this exposure has a name that sounds like plumbing: model-agnostic, or portable, architecture. The commercial meaning is straightforward. You put a boundary between your business logic and any single provider, so the provider sits behind an interface you own rather than being woven through everything you have built.

Think of it the way a well-run company treats any critical supplier. You do not pour your operations into a single source with no alternative and no way to compare. You keep the relationship at a defined boundary, you retain your own records and processes, and you preserve a credible path to a second source. The AI equivalent is keeping your data, your orchestration logic, and your evaluation criteria in your own hands, while the model itself remains a component you can swap. When done well, changing providers becomes closer to a configuration change than a rebuild.

Portability is not a technical luxury. It is the difference between negotiating with a supplier and being administered by one.

This is not an argument against committing to a vendor. Depth of commitment is often how you earn better terms and support, and constant switching would be its own kind of waste. The point is to commit deliberately: to know the exit cost before you sign, and to ensure no single provider's pricing, outage, or policy shift can dictate terms to the business. You can lean heavily on one supplier and still hold the ability to leave. Those two things are not in tension when the architecture is designed for it from the start.

The test that cuts through it

Ask one question of any AI system you depend on: if this provider doubled its price or went dark tomorrow, how long until we are running on an alternative, and what would it cost us to get there? If nobody can answer with confidence, you do not yet know your exposure - and the vendor may know it better than you do.

The questions a CFO or CEO should ask

Switching cost is easiest to assess before you sign, while you still have leverage and the vendor still wants the deal. These questions are deliberately non-technical. You do not need to understand the implementation to hear whether the answers are evasive.

1. If we wanted to move to a different model or provider in twelve months, what specifically would we have to rebuild, and who bears that cost?
2. Where is our data processed and stored, under which jurisdiction, and can we change that without re-architecting the system?
3. What happens to our workloads, and our access to our own data, if we stop paying - during the contract and after it ends?
4. Can we run more than one model behind the same process, so we are not exposed to a single provider's outage or price change?
5. How would we adopt a newer or cheaper model when one appears, and does that decision sit with us or with the vendor?
6. What is the exit path - data export formats, notice periods, transition assistance - and is it written into the contract rather than promised in a conversation?

A confident supplier answers these plainly, because a well-built system has good answers. Evasion, or a retreat into technical language meant to end the conversation, is itself information. The vendors most worth working with tend to be comfortable telling you exactly how you would leave, precisely because they intend to keep you by being good rather than by being inescapable.

Where the bill arrives

Lock-in almost never costs you anything on the day you accept it. It costs you at renewal, during an outage, when a compliance question lands, and at the moment a better model appears and you cannot reach it. Those are the four dates to imagine before you sign, because that is when the price is paid.

The decision in front of you

As agent systems move from pilots into the operations a business actually runs on, the cost of being unable to switch rises with every process that comes to depend on them. The sensible time to protect optionality is early, while the system is still small and the boundary is cheap to draw. Retrofitting portability after the business has grown around a single provider is possible, but it is the more expensive path, and it is usually undertaken under pressure rather than by choice.

The practical next step is not to tear anything out or to swear off vendors. It is to treat switching cost as a number you are entitled to know - a line item in any material AI commitment, owned jointly by the people who build the system and the people who sign for it. Put the six questions above to your current and prospective providers, and see how readily the answers come. The quality of those answers will tell you most of what you need to know about who is holding the leverage today, and whether you are comfortable leaving it there.

Common questions

Does staying model-agnostic mean we sacrifice performance or move slower?

Not if the boundary is designed well. A model-agnostic architecture routes each task to whichever model is best suited, so you can adopt a stronger model when one appears rather than being held to a single provider's roadmap. The discipline is in keeping the integration behind an interface you own, so switching is a configuration change rather than a rewrite. The cost is modest upfront design; the return is optionality as the frontier moves.

Is avoiding lock-in the same as never committing to a vendor?

No. Depth of commitment is often how you get better pricing and support, and switching for its own sake is wasteful. The goal is to commit with your eyes open: know the exit cost before you sign, keep your data and orchestration logic portable, and make sure a single provider's outage or price change cannot halt the business. You can lean heavily on one vendor and still hold the ability to leave.

Who should own this decision internally?

It is shared. Technical leaders own the architecture that makes switching feasible, but the risk itself is commercial, so finance and the executive team should treat switching cost as a line item in any material AI commitment. Procurement, legal, and compliance all have a stake in data residency and exit terms. Treating it purely as an engineering matter is how the exposure stays invisible until renewal.

Read this on keenagents.ai - written by Petar Denev, Co-founder & CEO.