Describe the job in observable terms
Specify the input, expected output and the person who uses the result. “Find the relevant policy with its source” is easier to evaluate than “make staff more productive”. Identify mistakes that are tolerable, mistakes that need review and actions that the system must never take without approval.
Evaluate a document assistant before adding actions
Suppose an internal assistant answers questions about leave policies. Create an answer key with the policy owner, then keep some questions out of development so the final test includes unseen examples.
| Test case | Expected evidence |
|---|---|
| Question answered by the current policy | Answer cites the correct passage and does not add an invented condition |
| Two versions of a policy disagree | Uses the approved current version or flags unresolved version ownership |
| Employee asks about another person’s record | Access boundary holds before retrieval, not only in the final wording |
Choose the information path
Some tasks need only the text in the current request; others need controlled access to documents or live records. Plan how material is retrieved, updated and restricted by user role. Check the provider’s data-handling arrangement before using confidential inputs, and include deletion and logging requirements in the design.
Evaluate the running workflow
Test representative cases with expected results and review failures with the business owner. Track latency, usage cost and the time people spend checking outputs. A pilot is ready to expand only when the operational support, permissions and quality thresholds are understood, not simply because a demonstration produces a plausible answer.
Before you proceed
- Prepare examples and acceptance criteria before a model comparison.
- Confirm data access and provider handling for the proposed use.
- Include evaluation and ongoing usage in the budget.
