Norm groups: how to interpret a test score

A raw score needs a comparison group

A raw score is the number of items answered correctly on an ability test, or the sum of the answers on a personality scale. On its own it says little: 27 out of 40 can be a strong result on a difficult test and an ordinary one on an easy test, and a personality scale has no correct total at all.

Norms translate the raw score into a position within a reference group: people who took the same test under comparable conditions. That position is relative. It tells you how someone compares with that group, not whether the person is suitable for a role. Suitability depends on what the role requires and on evidence that the test predicts what matters (see reliability and validity).

Norm scales and how they relate

Most norm scales express the same thing: how far a score lies above or below the average of the norm group, in standard deviations (SD). They differ in mean, spread and level of detail.

• z-scores: mean 0, SD 1. The basis for the other scales, rarely shown in reports.
• T-scores: mean 50, SD 10. About two thirds of the norm group falls between 40 and 60.
• Stens: 1 to 10, mean 5.5, SD 2. Common in personality reports.
• Stanines: 1 to 9, mean 5, SD 2. Common for ability tests.
• Percentiles: the percentage of the norm group that scored lower. Intuitive, but the steps are not equal.

Handle percentiles with care

Percentiles stretch differences in the middle of the distribution and squeeze them at the ends. Moving from the 50th to the 60th percentile takes a small raw-score difference; moving from the 90th to the 99th takes a large one. So do not average percentiles. Even on coarser scales such as stens, a one-point difference between two candidates often falls within the measurement error.

Explaining a score to a hiring manager

Percentiles are usually the easiest: out of 100 people in this comparison group, about 80 scored lower. Name the group in the same sentence and mention the margin: a score is an estimate, and the confidence range in the report shows where the true score probably lies. For personality scales, add that high or low is not good or bad in itself.

Choosing the right norm group

Choose the group you actually want to compare people with.

• General or working population: useful for broad screening and development, but in selection for higher-level roles it makes almost everyone look above average.
• Educational or job-level norms: people at a similar level, such as vocational, graduate or management roles. Usually the better reference for selection.
• Applicant norms: collected in real selection procedures. On personality questionnaires, applicants tend to describe themselves more favourably than people with nothing at stake.
• Employee or organisation norms: useful for development. In selection they carry earlier hiring choices into new decisions, and they are often small.

Country and language

Norms belong to a specific test version. A translation can change how items work and response styles differ between countries, so a norm from one language version does not automatically apply to another. When candidates take different language versions, check that the manual gives a norm per version and evidence that the versions measure the same thing.

What the wrong norm group does

A group that is too broad inflates scores; a group that is too selective makes good candidates look average. Mixing norms within one procedure is worse, because candidates compared with different groups can no longer be compared with each other. Use one norm group per procedure and record which one.

How good is the norm itself?

A norm is a sample, with the weaknesses of any sample. Before you rely on it, check the following.

• Size: there is no single minimum. Several hundred people per norm group is a common starting point; stable extreme percentiles need more.
• Composition: does the group resemble the people you assess in level, sector, region, age and gender? A large but skewed sample is no better than a smaller, well-composed one.
• Recency: populations and test-taking habits change. Know when the data were collected and be cautious with norms that are many years old.
• Collection: applicants, employees, or volunteers and students? Stakes, supervision and device all affect scores.

A good technical manual reports all of this, plus descriptive statistics, how raw scores were converted and the standard error of measurement. Test review systems such as those of COTAN and EFPA rate norms as a separate criterion; the guide to test quality criteria explains how. On Match.hr, the label of each test shows its norm sample sizes and the date of the last calculation, and every report names the norm group used (more under norming).

Norms and fairness

Groups can differ in average scores. A tempting response is a separate norm per group, so that women are compared only with women, or older candidates only with older candidates. For language versions, separate norms are often necessary. For characteristics such as gender, age or ethnic origin, they are usually not the answer in selection.

• The same raw score would lead to a different norm score depending on someone's group, which can amount to unequal treatment.
• They hide a difference instead of explaining it. A difference can come from biased items, from a real difference in what is measured or from how the test is used, and each calls for a different response.
• Characteristics such as ethnic origin are special categories of personal data under the GDPR and often cannot be collected in selection at all.

The better route is to examine bias directly: differential item functioning, measurement invariance across groups and language versions, and adverse impact at the cut-offs you use. Then check that what the test measures is relevant to the role. The page on fairness and bias explains these analyses.

Checklist before you read a report

• Which norm group is named, and does it match the level, country and language of the role?
• How large is the norm group, how was it collected, and when?
• Did the candidate take the same version, in the same language and under the same supervision as the norm group?
• Are the differences between candidates larger than the confidence range?
• Are all candidates in this procedure compared with the same norm?
• For personality scales: are you reading high and low as positions, not as good and bad?
• What else do you know? A test score sits next to interviews, work samples and references, and a person makes the decision.

If the report or the manual cannot answer these questions, ask the publisher before you rely on the scores.