AI makes it remarkably easy to generate things.
Questions. Cases. Feedback. Simulated conversations. Marking guidance. Training materials.
But in high-stakes assessment, generating something is rarely the difficult part.
The more important questions are whether it is valid, fair and useful; whether it improves the experience for candidates or those delivering the assessment; and whether it can operate safely and reliably at scale.
That's why a better starting point isn't:
Our experience across AI-supported clinical practice, case writing and item development suggests that asking that question can lead in some unexpected directions.
Use AI where it changes what's possible
Clinical skills are developed through practice.
Ideally, learners need repeated opportunities to conduct realistic consultations, timely and specific feedback from experienced clinicians, time to reflect and refine their approach, exposure to a wide range of cases and a safe environment in which to practise before applying those skills with real patients.
The challenge is providing all of that at scale.
Simulated consultations require people. Experienced educators have limited time. Role-player and faculty capacity is finite. The amount of practice available to individual learners can therefore be constrained, and the quality and availability of feedback can vary.
That gives us a much more useful problem for AI to solve:
How can we give learners more opportunities for realistic clinical practice, meaningful feedback and reflection when the people needed to provide that experience are a scarce resource?
One approach we've explored combines an AI patient with an AI coach. A learner conducts a simulated clinical consultation and then receives personalised feedback on their performance, with the opportunity to explore that feedback further with the coach.
The value isn't simply replacing a role player with an AI patient.
It's the potential to create a repeatable learning cycle:
That could give learners access to a much broader range of scenarios and more opportunities to develop confidence and clinical judgement in a low-stakes environment — whether preparing for an assessment or before working with real patients.
AI isn't being added because simulated patients are interesting technology. It is being used because it may make a valuable form of learning available more frequently and at greater scale.
That's a very different starting point.
AI can expose what you haven't clearly defined
Case writing produced a different lesson.
AI could generate clinically plausible ideas remarkably easily. But clinically plausible doesn't necessarily mean suitable for a high-stakes assessment.
In one exercise, 50 AI-generated case ideas were independently reviewed. Only seven received a clear “yes” for potential progression; 15 were considered “maybe” and 28 were rejected.
The reasons were revealing. Cases could depend too heavily on physical examination findings, become overly complex, offer little meaningful management beyond referral, rely on hidden information, be difficult for role players to portray consistently or provide insufficient opportunity to discriminate between different levels of candidate performance.
Understanding those failures highlighted something more fundamental.
Experienced clinicians and examiners carry a considerable amount of assessment knowledge in their heads. An experienced case writer can take a relatively simple case idea and use their clinical and assessment experience to fill in what isn't explicitly stated.
AI can't safely rely on the same shared understanding.
That became particularly interesting around the passing standard.
If you want AI to develop a case capable of distinguishing between weaker, just-passing and stronger candidates, you need to be able to articulate what those different levels of performance actually look like.
What does a candidate at the required standard need to demonstrate? What distinguishes that performance from one below the standard? What is the case actually intended to discriminate between?
If much of that understanding resides in experienced examiners' judgement rather than being explicitly defined, it becomes difficult to tell AI what to design for.
The solution therefore isn't simply a better prompt.
The pilot moved towards a much more explicit case specification — defining the assessment challenge, candidate task, management opportunity, role-player feasibility, discrimination value and other requirements before asking AI to develop the case.
That potentially improves more than the AI output. Making assessment intent explicit can also strengthen human case writing, review, examiner calibration and standardisation.
Sometimes experimenting with AI teaches you something important about the process you're trying to improve.
Don't automate a process you should be redesigning
Item writing produced another lesson.
Here, AI worked well in supporting the development of assessment items. The more interesting question became what to do with that capability.
If the existing process relies heavily on items being developed and reviewed by committee, simply inserting AI into the same workflow may miss much of the opportunity.
Instead, AI creates a reason to reconsider the workflow itself.
Could item writers work within a more structured process, with AI supporting development against agreed requirements? Could routine items progress without requiring the same level of committee involvement? Could scarce expert time be concentrated on variances, exceptions and items requiring particular judgement?
That is no longer simply automation.
It's process redesign.
And solving that problem creates another one.
If the quality and consistency of the AI-generated items depends on carefully developed prompts, giving every item writer direct access to those prompts may create issues around security, intellectual property, consistency and control.
Scaling the approach may therefore require a separate interface through which writers can use the capability without accessing or changing what sits behind it.
Suddenly an AI assessment pilot has become an IT and operating-model project involving access, governance, ownership, support and ongoing development.
There's an important lesson in that transition:
Start with the assessment
AI will continue to create new possibilities for assessment.
Sometimes it can enable something that wasn't previously practical. Sometimes it can augment expert work. Sometimes a successful experiment reveals that the workflow, technology or operating model needs to change before the opportunity can be realised.
And sometimes it exposes something more fundamental about the assessment itself.
That's why assessment expertise still needs to come first.
Understand the problem. Define what good looks like. Decide where AI can genuinely add value. Design the workflow and human decision points around it. Then work out what technology and governance are needed to make it sustainable.
And ultimately, the question isn't whether the AI worked. It's whether the assessment, the Candidate Experience or the service is better because of it.
Start with the assessment problem, not the technology.