AI Production Readiness: What Must Be True Before an Enterprise Pilot
An AI production-readiness framework for product leaders: customer workflow, release evidence, permissions, operating cost and the engineering scope before a pilot.
An enterprise AI pilot should test whether the product helps a customer do useful work. It should not be the first time the engineering team discovers whether the product respects that customer's permissions.
The distinction matters when a promising prototype becomes a sales commitment. A demo can depend on its author knowing which inputs to avoid. A customer will bring incomplete records, contradictory documents and a workflow that crosses systems the demo never touched.
Before agreeing to the pilot, write down the conditions under which a customer can use the system, the evidence supporting those conditions, and who owns the work still needed. That is the starting point for a production-readiness review.
Name the workflow before choosing the next model
“An assistant for the operations team” is too broad to evaluate or price. “Draft a response to an account query using records the operator can already access, with approval before sending” describes a bounded workflow.
That example is illustrative. Its value is that it makes several decisions visible: which records are allowed, what the model produces, what remains a human decision and where the workflow ends.
Write the same description for your product. Include the user, their existing process and the action that would make the pilot useful. If the team cannot agree on that paragraph, commissioning more integration work is premature.
A model upgrade may improve the output. It cannot settle an argument about which customer problem the product is meant to solve.
Make the release decision inspectable
A production-readiness review should leave behind evidence that another engineer can run or inspect. A slide saying “guardrails implemented” does not tell a buyer which actions the system can take.
| Release question | Evidence to request | Stop condition |
|---|---|---|
| Does it complete the intended task? | A versioned evaluation set drawn from the customer workflow, with a held-out release set | Only demonstration examples have been tested |
| Can it access only permitted data? | Tests covering users, tenants and revoked access across retrieval and tools | A prompt instruction is the only access restriction |
| What can it change? | An inventory of tool actions, authorization checks and approval requirements | Consequential actions have no enforced boundary |
| What happens when a dependency fails? | Exercised timeout, retry, fallback and escalation paths | The team cannot explain whether an interrupted action already happened |
| Can the team investigate a bad result? | Traces linking the request to model, prompt, retrieval and tool activity, with suitable data handling | A complaint cannot be reconstructed |
| Can the team afford to operate it? | Measured model usage, infrastructure cost and human review effort for the workflow | The business case assumes manual review is free |
This is a starting framework, not a certification standard. Each product needs its own acceptance thresholds and accountable owner. A drafting tool and a system permitted to alter customer records require different boundaries.
Test the action, including the state it leaves behind
For an agent, a convincing response can coexist with an incorrect operation. The transcript may claim success while the target system remains unchanged, or the operation may succeed twice after a retry.
Anthropic's January 2026 guidance distinguishes an agent's transcript from the resulting state of its environment, and recommends inspecting both when designing evaluations. That distinction is useful when a release decision depends on an action actually completing. Source: Anthropic, Demystifying evals for AI agents, January 9, 2026.
For the illustrative account-query workflow, assess whether the draft is supported by permitted records and whether the approval boundary held. If the workflow later includes a write action, check the resulting record as well. Replaying a successful conversation is insufficient evidence that the write path is correct.
Keep examples used to tune the system separate from examples used to decide whether to release it. Record failures individually. An aggregate score can conceal a category of errors the customer will not accept.
Price the engineering around the model
The useful unit of scope is a workflow with acceptance criteria. “Integrate an LLM” leaves the expensive questions unanswered.
A proposal should say who owns data connectors, evaluation cases, permission checks and operational handover. It should also identify what the customer's team must supply. Missing access to a production-like environment can hold up delivery even when the application code is straightforward.
Separate engineering fees from model usage, infrastructure and software licenses. Include the time operators spend reviewing exceptions in the operating model. An inexpensive model call can still support an expensive workflow if someone must check every result.
Where a capable engineering lead already owns the roadmap, a scoped senior engineering engagement may be enough. A fractional CTO becomes relevant when the missing piece is authority across product priorities, architecture and hiring. Buying the latter title does not automatically create more delivery capacity.
Limit the first release to what you can support
Agree which customers and tasks the pilot includes. Define the fallback before the model fails, including who receives an escalation and what the customer sees. Specify how to stop new actions and how to investigate actions already taken. Rolling back software does not reverse an external side effect.
Give the internal team the deployment instructions, evaluation set and runbook. They should be able to repeat the release check without the consultant who built it sitting beside them.
If the assessment ends with a longer feature list but no clearer release decision, it has missed its purpose.
AZdev's production AI engineering engagements cover application integration, evaluation and the operating work around a release. For a focused problem with task quality or graders, see agent evaluation engineering. Bring the workflow, the current failure and the pilot commitment to an engineering scoping conversation.