Overall pass rates can tell you whether an assessment is performing broadly as expected. But they can also hide important differences between groups of candidates.
Looking at outcomes by characteristics such as gender, ethnicity and age — and, where relevant, where candidates qualified, trained or are working — can reveal patterns that deserve closer attention.
But identifying differential attainment is only the beginning.
Start with the outcomes — but don't stop there
The first step is to understand where differences are occurring.
That means going beyond a single overall pass rate. Are differences consistent over time? Do they appear across the whole assessment, or are they concentrated in particular papers, stations, cases, domains or types of task?
In a clinical assessment made up of multiple cases, for example, an overall difference between candidate groups may conceal considerable variation. Some cases might show a much larger difference, while others show relatively little.
That variation is potentially valuable.
Don't just measure the size of the gap. Look at where the gap gets bigger — and where it gets smaller.
The cases showing the biggest differences may provide a useful place to start a deeper investigation. What is different about them? Is there anything about the assessment content, context, interaction or judgement that warrants closer examination?
Cases where the difference is much smaller can be equally interesting. What is different about those?
The purpose isn't to conclude that a particular case is causing differential attainment. It is to use the data to decide where to investigate more deeply.
Look for patterns, not just individual cases
There may be a different explanation for each case.
But there may also be common themes.
Once you've identified where differences widen and narrow, you can compare those parts of the assessment. Do cases showing larger differences share any characteristics? Do cases showing smaller differences have something in common?
That might lead you to explore assessment design and content, the nature of the task, candidate interpretation, examiner judgement, role-player interaction or other features of the assessment.
These are questions to investigate, not explanations to assume.
The important progression is:
The biggest differences may tell you where to investigate first. The patterns across those cases may tell you what to investigate next.
Be careful what the data tells you
Finding a pattern still doesn't explain its cause.
Candidates arrive at an assessment with different educational experiences, opportunities, preparation and support. Aspects of the assessment and Candidate Experience may also affect different candidates in different ways.
The important thing is not to jump too quickly to either explanation.
Don't assume the explanation sits with the candidate. Don't assume the assessment is the explanation either. Investigate.
Quantitative analysis can help identify where something interesting is happening and generate better questions. Understanding why usually requires getting closer to the assessment and bringing in other evidence.
Get closer to the experience
Once you know where to look, the investigation can become much more focused.
That might mean reviewing assessment content and design, examiner judgement and standardisation, role-player training and behaviour, candidate preparation, accessibility, technology or delivery conditions.
In a clinical assessment, it might also mean looking at what actually happened during consultations. Are candidates interpreting the task differently? Are interactions developing differently? Are examiners responding differently? Does what you observe support the themes suggested by the assessment data — or point you somewhere else?
Candidate research adds another perspective. Feedback, interviews or other qualitative research can help explore how different candidates prepare for and experience the assessment.
Complaints, incidents and operational data may provide further pieces of the picture.
The objective is to combine the evidence to move from observing a difference towards a better understanding of what may be contributing to it.
Then ask what you can change
Understanding differential attainment shouldn't end with a report.
If the evidence identifies something within the assessment or Candidate Experience that could be improved, make the change and evaluate what happens.
That might mean changes to assessment content, candidate preparation, accessibility, examiner or role-player training, standardisation, technology, delivery processes or support.
Then return to the evidence.
Did the change make a difference?
Fair assessment doesn't mean every group achieving the same outcome. Different outcomes don't, by themselves, demonstrate that an assessment is unfair.
But meaningful differences shouldn't simply be observed and accepted.
The aim is to investigate them systematically, understand what may be contributing to them and identify whether anything within the assessment or wider Candidate Experience is creating an avoidable barrier to candidates having a fair opportunity to demonstrate their ability.