Check AI search updates against real staff questions

What this means for your practice

Keep a representative set of questions, expected source context and unacceptable outcomes. Review changes against the set before expanding use or replacing the existing workflow.

A practical conversation with your supplier

Ask to run the same examples before and after an update. Record the approved source, acceptable answer behaviour and outcomes that should fail review. Include questions that previously exposed mistakes. A change that improves one example but worsens another needs a visible decision before wider use.

Plan the first improvement

Choose questions from the work people actually do and remove unnecessary personal details when turning them into test examples. Include common requests, unusual wording, missing answers and questions that should remain outside the permitted scope. For each example, record the relevant source and what would make the response unacceptable. That may include invented detail, an omitted qualification or information from a restricted document. Keep the collection small enough that reviewers can use it consistently, then add examples when real failures reveal a gap.

Ask the supplier to compare a proposed update with the current version using the same questions. A summary should show improvements, newly introduced problems and remaining uncertainty, not only a favourable demonstration. The accountable owner can then decide whether to accept the change or continue refining it. Periodically review the test questions themselves when services change, because a once-correct expected answer may become outdated along with the underlying guidance.

How to check the result

Record answer problems introduced by an update and questions still failing the agreed review criteria.