Your assistant handles the demo request correctly. It finds the right client, classifies the question, and prepares a useful reply. Then a request arrives without a client ID, or the CRM stops responding halfway through.
That is when the launch requirements become visible.
Before releasing an agent into an operating workflow, I want the team to agree on what it may do, what counts as success, and how it should behave when it cannot finish. Put those requirements into cases somebody can run and inspect.
Give the agent a bounded job
In this illustrative example, a service agency uses an assistant to classify incoming client requests and draft responses. A person approves every external reply. The assistant cannot issue refunds, change commercial terms, or create client commitments.
Write those boundaries as requirements. “Handle support” leaves too much interpretation. “Classify a request, retrieve the permitted account context, and save a draft for review” lets the team inspect a concrete result.
Anthropic's Building effective agents distinguishes predefined workflows from systems in which the model directs its own actions. Choose the amount of autonomy the job requires. A predictable routing step may be enough for part of the process.
Also define who owns the workflow, which records it may access, and where a reviewer can see what happened. These become part of the launch check, alongside the quality of the draft.
Turn requirements into examples
Build the first cases from the requests your team expects to receive, including the ones that could cause trouble. Specify the input, the expected behavior, and the evidence needed to judge it.
Anthropic's evaluation guidance recommends clear tasks and reference solutions, including cases where a behavior should occur and where it should not. That is a useful test of your requirements: could two reviewers independently agree whether the case passed?
Test these scenarios against your workflow and record the results in the launch checklist.
Inspect the action as well as the answer
A good-looking reply can hide an incorrect action. Check the account selected, the records read or changed, the item created, and whether the request reached the right reviewer.
For duplicates, inspect the work queue after the second event. For a failure during a write, inspect the destination before retrying. The system should have a reliable way to identify the original request so a retry does not create a second commitment.
Run each case more than once. Keep a record of every run rather than replacing a failure with a later success. The checklist starts with three trials per case. Adjust the number and variety of tests to the risk of failure and the inputs your team receives; three successful runs alone do not establish reliability.
Any prohibited action should stop the release review until the cause is understood and the relevant cases are rerun. Aggregate success rates should not make that failure disappear.
Prepare for a failure during the working day
Your launch checklist should name the person who receives an alert and explain how they can pause the automation. Test the pause control and the manual fallback before the pilot starts.
Set a processing limit so a stuck request cannot run indefinitely. Record the system version used for each test, including changes to the model, prompt, and tools. A later change needs the affected cases checked again.
The downloadable checklist includes twelve starter cases and space to record your results and evidence. The calculated status keeps incomplete checks visible and treats a recorded prohibited action as a reason to stop.
Start the release review with one ordinary request and one request the assistant must escalate. Agree on the acceptable outcome, then have the team demonstrate both in a test environment before expanding the pilot.
Check test evidence, permissions, pause controls, and ownership before approving a limited pilot.
Resources
Your assistant handles the demo request correctly. It finds the right client, classifies the question, and prepares a useful reply. Then a request arrives without a client ID, or the CRM stops responding halfway through.
That is when the launch requirements become visible.
Before releasing an agent into an operating workflow, I want the team to agree on what it may do, what counts as success, and how it should behave when it cannot finish. Put those requirements into cases somebody can run and inspect.
Give the agent a bounded job
In this illustrative example, a service agency uses an assistant to classify incoming client requests and draft responses. A person approves every external reply. The assistant cannot issue refunds, change commercial terms, or create client commitments.
Write those boundaries as requirements. “Handle support” leaves too much interpretation. “Classify a request, retrieve the permitted account context, and save a draft for review” lets the team inspect a concrete result.
Anthropic's Building effective agents distinguishes predefined workflows from systems in which the model directs its own actions. Choose the amount of autonomy the job requires. A predictable routing step may be enough for part of the process.
Also define who owns the workflow, which records it may access, and where a reviewer can see what happened. These become part of the launch check, alongside the quality of the draft.
Turn requirements into examples
Build the first cases from the requests your team expects to receive, including the ones that could cause trouble. Specify the input, the expected behavior, and the evidence needed to judge it.
Anthropic's evaluation guidance recommends clear tasks and reference solutions, including cases where a behavior should occur and where it should not. That is a useful test of your requirements: could two reviewers independently agree whether the case passed?
Test these scenarios against your workflow and record the results in the launch checklist.
Inspect the action as well as the answer
A good-looking reply can hide an incorrect action. Check the account selected, the records read or changed, the item created, and whether the request reached the right reviewer.
For duplicates, inspect the work queue after the second event. For a failure during a write, inspect the destination before retrying. The system should have a reliable way to identify the original request so a retry does not create a second commitment.
Run each case more than once. Keep a record of every run rather than replacing a failure with a later success. The checklist starts with three trials per case. Adjust the number and variety of tests to the risk of failure and the inputs your team receives; three successful runs alone do not establish reliability.
Any prohibited action should stop the release review until the cause is understood and the relevant cases are rerun. Aggregate success rates should not make that failure disappear.
Prepare for a failure during the working day
Your launch checklist should name the person who receives an alert and explain how they can pause the automation. Test the pause control and the manual fallback before the pilot starts.
Set a processing limit so a stuck request cannot run indefinitely. Record the system version used for each test, including changes to the model, prompt, and tools. A later change needs the affected cases checked again.
The downloadable checklist includes twelve starter cases and space to record your results and evidence. The calculated status keeps incomplete checks visible and treats a recorded prohibited action as a reason to stop.
Start the release review with one ordinary request and one request the assistant must escalate. Agree on the acceptable outcome, then have the team demonstrate both in a test environment before expanding the pilot.
Check test evidence, permissions, pause controls, and ownership before approving a limited pilot.