Examine whether your test measures fairly across groups
Groups can differ in scores for many reasons. Fairness research asks whether a test measures the same thing in the same way for everyone, and what a score difference means for decisions.
Ongoing fairness checks on the platform are being built up per test, as far as lawful group data is available.
Three levels, three analyses
Bias can sit in a single item, in a whole scale or in the way scores are used. Each level has its own analysis.
Differential item functioning
At item level: do people with the same level on the construct have a different chance of answering an item a certain way, depending on their group? DIF points to items worth reviewing.
Measurement invariance
At scale level: does the factor model hold across groups or language versions, with the same loadings and intercepts? Without it, comparing group means is not meaningful.
Adverse impact
At decision level: do selection rates differ between groups at a given cut-off or in a given procedure? Rules of thumb such as the four-fifths rule are a signal to investigate, not proof of unfairness.
Group data only where it is lawful
Characteristics such as ethnic origin or health are special categories of personal data under the GDPR. We only analyse group characteristics when there is a legal basis, a clear purpose and adequate safeguards.
- A defined purpose and a lawful basis for each characteristic analysed
- Explicit consent where required, voluntary and separate from the assessment itself
- Only the characteristics the question actually needs
- Analysis on a pseudonymised or anonymised dataset
- Results reported at group level only, with minimum group sizes
- Group data never used in an individual score or decision
How a fairness study runs
Frame the question
Which groups, which use and which decision? We check with you whether the data needed can be collected or linked lawfully.
Prepare the data
Collect group characteristics with consent where needed, or use existing data such as language version or country, and check whether groups are large enough.
Analyse
Test for DIF and measurement invariance, and where a procedure and cut-off are known, examine adverse impact.
Interpret and act
Flagged items are reviewed with content experts, then revised, removed or kept with a documented reason. The analysis is repeated after changes.
What these analyses cannot show
No analysis proves that a test is free of bias. DIF and invariance tests only look at the groups that were measured, and with small groups they have little power, so finding no DIF is not the same as there being none. A real difference between groups in the construct is not by itself bias, and a test without measurement bias can still have adverse impact in a particular procedure. Decisions stay with people, who weigh test scores alongside other information.
Frequently asked questions
Can you certify that our test is free of bias?
No, and we would be wary of anyone who does. A fairness study shows whether there is evidence of measurement differences for the groups and data examined. We report what was tested, with which sample, and what was found, including the limits.
Which groups can be compared?
Language versions and countries can often be compared using data the platform already holds. Age and gender need a lawful basis and a clear purpose. Special categories such as ethnic origin need explicit consent and strong safeguards, and are usually studied in a separate research sample rather than in live selection data.
What happens if an item shows DIF?
It is not removed automatically. Content experts look at why the item might work differently, for example an idiom that does not translate or an example that is more familiar to one group. The item is then revised, removed or kept with a documented reason, and the analysis is repeated.
Does adverse impact mean a test is unfair?
Not necessarily, but it needs a good explanation. Check that what the test measures is relevant to the role, that the cut-off is justified and whether combining it with other measures reduces the difference. Under EU equal treatment law, a practice that disadvantages a protected group must be objectively justified, so involve your legal adviser where this applies.
Related pages
Test your first candidate today
Free account, no credit card, results in minutes.