AI evaluation
Evaluation cases for claims, citations, extracted data, access boundaries and human handoff.
Knowledge retrieval diagnostics
Trace missing bundles, useful positions, repeated document chunks, rewrites and packed context.
Document layout evidence
Retain grouped headings, column order, selection marks, footnote anchors and continuation identity.
Text grounding and alignment
Check quote attribution, scoped glossary use, reference candidates, spoken digits and bilingual segment links.
Draft output constraints
Use literal-name, quotation-scope, exact-extract, wrapper, repair-bound and source-order fixtures.
Generated tool plan checks
Inspect tool names, preceding-result arguments, status channels, parallel references, field scope and partial results.
Local grader correctness
Test duplicates, exact tolerances, parsed equality, citation coordinates and candidate coverage before scoring.
Repeated observation reporting
Separate stable cases, repeated-attempt weighting, early-stop coverage and completion-only timing.
Evaluation data controls
Check sentinel keys, declared-link components, controlled perturbations and conditional prevalence assumptions.
Local output adapters
Parse escaped answer boundaries, choose payload channels, finalize streams, enforce versions and prepare result rows.
Structured staff grading
Retain reviewer matrices, disputed keys, required dimensions, order checks and bounded reason codes.
Judge supported, missing and contradicted AI claims
Separate each claim in a mixed answer, preserve planned versus completed states, and refuse only the unsupported part.
Check whether an AI citation supports its claim
Review each claimed fact against the cited passage, including citations that name a real document but point to the wrong detail.
Evaluate answers from superseded documents
Check explicit effective dates and document status instead of treating the newest upload as the applicable rule.
Test table extraction without losing units
Use quantities, unit prices and totals to catch row shifts, unit confusion and unsupported conversions.
Test corrections across English and Traditional Chinese
Check that a user correction updates the relevant fact while preserving unchanged details across language switches.
Evaluate an AI-to-staff handoff summary
Prepare a handoff answer key that preserves the customer’s unresolved request, known facts and completed actions.
Test repeated requests for the same action
Use a lost response and a repeated user message to define when an assistant should check action state instead of creating another write.
Test date interpretation before an action
Check ambiguous numeric dates, missing years and date-only values without silently choosing a booking date.
Test mixed-currency answers without invented rates
Use original price extracts to test currency labels, billing periods and totals that cannot be compared from the supplied data.
Evaluate answers across permission boundaries
Define which synthetic records each role may read and check direct, indirect and follow-up disclosure attempts.
Define AI recovery after an interrupted action
Use saved action states to distinguish a draft, an uncertain submission and a confirmed result before resuming.
Build a complete AI test evidence record
Connect each review decision to its input, permitted sources, actual output and action evidence so another reviewer can check it.
Test instructions embedded in source material
Use a synthetic document containing an unauthorised instruction to define the boundary between evidence and action authority.
Evaluate negated rules and policy exceptions
Check a general rule together with an explicit exception so a summary does not reverse eligibility or widen a permission.
Test totals with duplicate and excluded rows
Define row identity and inclusion rules before judging counts, totals or grouped summaries.
Evaluate answers when approved sources disagree
Use unresolved contradictory extracts to test whether an assistant shows the conflict and follows the supplied ownership rule.
Measure answerable cases and correct abstentions separately
Use a small labelled set to report unsupported answers and unnecessary refusals without hiding either behind a single pass rate.
Keep an AI review set separate from development examples
Track which synthetic questions were used to adjust an assistant and which remain reserved for a later independent review.
Review AI behaviour changes after a revision
Compare case-level outcomes before and after an instruction or source change, retaining improvements and newly introduced failures.
Test structured AI outputs and missing fields
Define a small booking-request schema and separate valid structure from a factually correct, authorised action.
No matching resource.