Are personality tests reliable? How to tell a serious one

"Reliable" can mean two different things for a personality test, and it helps to keep them apart. Reliability asks whether you get roughly the same result if you take it again. Validity asks whether the score measures what it claims to and predicts anything about real life. The Big Five questionnaires used in research pass both checks with published numbers. Many online tests publish neither, which is already half an answer.

Reliability: the same result twice

One way to measure reliability is internal consistency: do the statements meant to measure the same trait agree with each other? The usual index, Cronbach's alpha, runs from 0 to 1. For the IPIP-NEO-120, the public questionnaire our test uses, alphas for the five traits ranged from .81 for Openness to .90 for Neuroticism and Conscientiousness in a sample of 619,150 people (Johnson, 2014).

The other way is to retake the test later. A synthesis of 848 stability coefficients found average correlations between .69 (Agreeableness) and .76 (Extraversion) between two measurements of the same trait (Viswesvaran and Ones, 2000). If you land in the upper part of a scale today, you will very likely be there in a few weeks, though your exact position may move a few points.

Facets, the six narrower parts of each trait, are less precise. In the IPIP-NEO-120 each facet has four statements, and facet alphas range from .63 to .88 (Johnson, 2014). Johnson himself notes that important decisions about an individual usually call for a reliability of .95, which no short questionnaire reaches. A good Big Five test is a tool for understanding yourself, not for deciding a hire or a diagnosis.

Validity: does it measure something that matters?

A stable score could still be meaningless. The most direct check is whether traits predict real outcomes in health, work and relationships. Christopher Soto repeated 78 published trait-outcome links with preregistered hypotheses and samples of about 1,500 people each. 87% replicated in the expected direction, but the effects were weaker: on average about 77% as strong as the originals (Soto, 2019).

Read that both ways. The links are real and they replicate, which is not a given in psychology. They are also modest: they describe tendencies across large groups, not the fate of the person taking the test.

Types or traits

Many online tests assign you a type. The trouble is that normal personality does not seem to come in boxes. A review of 311 analyses covering more than half a million people, using methods built to tell categories from dimensions, found little persuasive evidence of categories in normal personality (Haslam, Holland and Kuppens, 2012). The data look like continuous scales on which most people sit near the middle.

Boxes throw away information exactly where most people are. Someone just above a cutoff and someone just below it end up in opposite types although they answer almost the same way, and a small change in answers flips them across. Our 16 archetypes are boxes too, which is why we present them as a memorable summary: the actual result is your five positions on the scales, as the methodology page explains.

The Barnum effect

In 1949 the psychologist Bertram Forer gave his students a personality sketch he said was based on a test they had just taken. Every student got the same text, and on average they rated it as highly accurate (Forer, 1949).

Statements that fit almost everyone ("at times you doubt your decisions", "you can be sociable but you also need time alone") make a test feel accurate while saying nothing. There is a simple check: would someone at the opposite end of that trait disagree with the sentence? If not, it is not describing you.

Who you are compared with

A raw score means little until you know how other people answer. A serious test says which norms it uses, meaning the sample it compares you with. Ours are from Kajonius and Johnson: 320,128 people in the United States who took the IPIP-NEO-120 online (Kajonius and Johnson, 2019). The authors themselves say the sample is probably not representative of the whole population: volunteers, young on average (28), who found the site on their own. No published norms exist for this questionnaire in Italian, and we would rather say so than invent them.

Honest answers

Every questionnaire like this rests on what you report. When something is at stake, people describe themselves more favorably: across 33 studies, job applicants scored higher than non-applicants, mostly on emotional stability and conscientiousness, by a little under half a standard deviation (Birkeland and colleagues, 2006). A test you take for yourself, with nothing to win, gives less reason to polish the picture. The result is still only as good as your honesty.

How to tell a serious test

Before trusting a result, check whether the test:

• uses a published questionnaire with reliability figures you can look up;
• gives you a position on continuous scales, not only a type;
• says which sample it compares you with;
• says what it cannot do, starting with diagnoses and decisions about jobs or health;
• describes you in a way someone at the other extreme would reject.

Our test answers these questions on its methodology page. If you want to see the five traits in practice, the test below takes about 10 minutes.