AI agent production readiness checklist: 24 checks before go-live

Short answer: an AI agent is ready for production when it passes 24 checks across six areas and every one of the 14 blockers is a yes. Treat the blockers as a go/no-go gate, and give every other gap an owner and a date.

Definition: production readiness. The point where a team can depend on an AI agent's output, explain what it did, limit what it can do, afford it, and switch it off safely.

How to use this checklist

1. Run it with the business sponsor, the tech lead, and the person who owns security or data protection.
2. Answer each check with a yes or a no, and ask for the evidence in the third column. An opinion is not evidence.
3. Apply the go/no-go rule at the end. If you are still in the POC or pilot stage, use the checklist to see what is missing before you move on.

The 24 checks

Data

#	Check	Evidence to ask for	Blocker
D1	The agent was tested on real documents or messages, not only cleaned samples.	A test run on a representative sample, with the failures listed	Yes
D2	Every data source has a named owner and an access rule.	A list of sources, owners, and who may read what	Yes
D3	Personal and confidential data are identified and handled under your data protection rules.	A short data map, reviewed with the DPO	Yes
D4	Data quality is checked automatically on the live flow.	Alerts for missing, stale or malformed inputs	No

Security and permissions

#	Check	Evidence to ask for	Blocker
S1	Permissions are the smallest that do the job, and start read-only.	A permission list per tool	Yes
S2	Credentials are per person, encrypted, and out of the model's reach.	Where keys are stored and who can read them	Yes
S3	Any write needs a human confirmation for that exact action.	A demo of a write being blocked until approved	Yes
S4	Access is logged (who, when, what) without copying content, and can be revoked.	A log extract and a revocation test	Yes

Evaluation

#	Check	Evidence to ask for	Blocker
E1	A test set of real cases exists, with the expected results.	The set, and the name of its owner	Yes
E2	A go-live quality threshold is agreed in writing.	The threshold and the current score	Yes
E3	Tests re-run automatically after each change of model, prompt or tool.	A history of test runs	No
E4	A person reviews failures every week during the first months.	A review log	No

Ownership and operations

#	Check	Evidence to ask for	Blocker
O1	A named product owner is accountable after launch.	A name and a role	Yes
O2	A kill switch exists, and someone knows how to use it.	A test of switching the agent off	Yes
O3	Incidents have a path: who is alerted, who decides, who communicates.	A one-page runbook	No
O4	Versions of model, prompt and tools are tracked, and you can roll back.	A version history	No

Cost

#	Check	Evidence to ask for	Blocker
C1	Cost per task is measured on real usage.	A measured figure, not an estimate	Yes
C2	A monthly budget with alerts and a hard cap exists.	The budget and the alert settings	Yes
C3	You have estimated the cost at ten times today's usage.	A simple forecast	No
C4	A cheaper model tier was tested on the simple tasks.	A comparison on your test set	No

Adoption

#	Check	Evidence to ask for	Blocker
A1	The target users and the workflow the agent changes are named.	A short description of who and which workflow	Yes
A2	Users received short training and know what the agent must not do.	Training material and attendance	No
A3	A usage policy says what data people may give the agent.	A one-page policy	No
A4	Usage and satisfaction are tracked, with a review date.	A dashboard or a monthly review	No

The go/no-go rule

• Any blocker answered no: no-go. The agent stays in pilot, and the missing blocker becomes the next piece of work.
• Only non-blockers answered no: go is possible, if each gap has an owner and a date.
• All 24 yes: go, with a review date. Production is a stage, not the finish line.

Why these 14 blockers? Each one protects against a failure that is hard to undo: bad data reaching users, permissions that are too broad, no measure of quality, nobody accountable, no way to switch off, and costs nobody tracks.

Where the checks come from

The six areas map to the four functions of the NIST AI Risk Management Framework: govern, map, measure and manage. NIST also published a generative AI profile in July 2024 that lists risks specific to generative AI. The security checks follow the same principles as the OWASP Top 10 for LLM applications, which includes excessive agency, and the MCP specification, which puts user consent first.

In our own mailbox pilot, the checks that took the most design effort were S1 to S4. The connector was the easy part. The detail is in AI agent email access: 7 questions to ask your AI expert.

For evaluation in particular, the LangChain agent evaluation checklist is a good technical complement for your engineers.

Next step

Start with one real use case and run this checklist on it before you scale. See our services or contact Elia Studio to scope a pilot.