Reliability & Validity

When researchers collect data, they need assurance that their measurement tools are doing their job properly. Two fundamental properties every good measurement instrument must have are reliability and validity. Reliability refers to the consistency of a measurement — does the tool give the same result under the same conditions? Validity refers to accuracy — does the tool actually measure what it is supposed to measure? Both properties are essential: a measurement can be reliable without being valid (consistently wrong), but it cannot be valid without first being reliable. Consider a bathroom scale that always reads 5 kg too heavy. That scale is perfectly reliable — it gives consistent readings — but it is not valid for measuring your true weight. Contrast this with a scale whose needle jumps randomly; it is neither reliable nor valid. Only a scale that consistently reads your true weight is both reliable and valid. This distinction matters enormously in statistics, psychology, education, and any field that depends on data collection. Researchers use specific quantitative methods to evaluate reliability and validity. Test-retest and parallel-forms methods produce correlation coefficients that measure reliability, while content validity and criterion-related validity (both concurrent and predictive) assess how well an instrument captures what it claims to measure. Understanding these concepts helps you critically evaluate studies, surveys, and tests you encounter in everyday life and academic work.

Key takeaways
  • Reliability: What It Means
  • Test-Retest Reliability
  • Parallel-Forms Reliability
  • Validity: What It Means
  • Content Validity
  • Criterion-Related Validity

Reliability: What It Means

Reliability is the degree to which a measurement instrument produces stable, consistent results across repeated administrations or equivalent forms. It is typically quantified using a reliability coefficient, denoted r, which ranges from 0 (no reliability) to 1 (perfect reliability). In practice, social-science instruments aim for r ≥ 0.80, while high-stakes educational tests aim for r ≥ 0.90.

Test-Retest Reliability

Test-retest reliability is assessed by administering the same instrument to the same group of participants on two separate occasions and computing the Pearson correlation coefficient between the two sets of scores. If scores are stable over time, the correlation will be high. A waiting period of two to four weeks is typical — long enough that participants do not simply remember their answers, but short enough that the trait being measured has not genuinely changed. For example, if 30 students take a maths anxiety questionnaire in Week 1 and again in Week 3, a Pearson r of 0.87 between the two administrations indicates good test-retest reliability.

Parallel-Forms Reliability

Parallel-forms reliability (also called alternate-forms reliability) involves creating two equivalent versions of the same instrument — same constructs, same difficulty level, same number of items — and administering both to the same group, either simultaneously or in close succession. The correlation between scores on Form A and Form B is the parallel-forms reliability coefficient. This method avoids the memory problem of test-retest but requires the considerable effort of constructing two truly equivalent forms. Standardised achievement tests such as the SAT use this approach to ensure different test versions are interchangeable.

Validity: What It Means

Validity is the extent to which an instrument measures the specific concept or construct it claims to measure. Whereas reliability is a necessary condition for validity, it is not sufficient. Validity is always evaluated with respect to a specific purpose: a test valid for one use may be invalid for another.

Content Validity

Content validity (sometimes called face validity when judged superficially) is the degree to which an instrument's items adequately represent the full domain of the construct being measured. It is evaluated judgementally, not statistically. A panel of subject-matter experts reviews the instrument and rates each item's relevance. A Content Validity Index (CVI) is often computed: CVI = (number of items rated relevant by experts) ÷ (total number of items). A CVI ≥ 0.80 is generally considered acceptable. For example, a final exam covering only one chapter of a six-chapter course would have low content validity for measuring the whole course.

Criterion-related validity assesses how well scores on an instrument predict or correlate with an external criterion — some outcome or measure that is independently known to represent the construct. There are two subtypes. Concurrent validity is established when both the instrument and the criterion are measured at the same time; the correlation between them (the validity coefficient) tells you how well the instrument substitutes for the more expensive or time-consuming criterion. Predictive validity is established when the instrument is administered first and the criterion is measured later; a high correlation means the instrument successfully predicts future outcomes. For example, if scores on a new scholastic aptitude test correlate r = 0.72 with end-of-year GPA measured six months later, that is evidence of predictive validity.

The Key Equation

rxy=∑i=1n(xi−xˉ)(yi−yˉ)∑i=1n(xi−xˉ)2  ∑i=1n(yi−yˉ)2r_{xy} = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2 \; \sum_{i=1}^{n}(y_i - \bar{y})^2}}
  • r_{xy} — Correlation coefficient (dimensionless): Measures the strength and direction of the linear relationship between two sets of scores; used as the reliability or validity coefficient.
  • x_i — Score on instrument (occasion/form 1) (score units): The i-th participant's score on the first administration or form.
  • y_i — Score on instrument (occasion/form 2 or criterion) (score units): The i-th participant's score on the second administration, alternate form, or criterion measure.
  • \bar{x}, \bar{y} — Sample means (score units): Arithmetic averages of the x and y score sets respectively.
  • n — Sample size (persons): Number of participants in the study.

Worked Example

Problem. A researcher administers a 20-item wellbeing scale to 5 students, then re-administers the same scale three weeks later. The scores (out of 100) are: Student 1: (62, 65), Student 2: (78, 74), Student 3: (55, 57), Student 4: (90, 88), Student 5: (70, 72). Calculate the test-retest reliability coefficient and interpret it.

rxy≈0.99r_{xy} \approx 0.99

Answer. Test-retest reliability coefficient r ≈ 0.99, indicating excellent reliability.

Figure
Diagram
Reliability vs. Validity — Four ScenariosHIGH ReliabilityHIGH ValidityClustered on targetHIGH ReliabilityLOW ValidityClustered, but off targetLOW ReliabilityLOW ValidityScattered and off targetLOW ReliabilityHIGH ValidityTheoretically impossible:inconsistent ≠ accurateReliability is necessary but not sufficient for validity.
Origins · 1 minLee J. Cronbach and how this was discovered
LJ
Lee J. Cronbach
1916–2001 · American

Lee Cronbach was a psychologist and educational statistician at Stanford University best known for developing Cronbach's alpha (α) in 1951, the most widely used measure of internal consistency reliability. He also made foundational contributions to validity theory, particularly in his 1955 paper with Paul Meehl introducing construct validity, and later in his 1989 work refining the unified view of validity. His generalisability theory extended classical test theory to account for multiple sources of measurement error simultaneously. Cronbach reportedly said that his alpha coefficient had been cited so many times that he could not keep track, and he spent much of his later career cautioning researchers not to over-rely on it as the sole measure of a test's quality.

  • Pearson Correlation Coefficient — The Pearson r is the primary statistic used to compute test-retest and parallel-forms reliability coefficients as well as criterion-related validity coefficients.
  • Sampling Methods — The reliability and validity of conclusions from any study depend heavily on how the sample was drawn; biased samples compromise both properties.
  • Measurement Error — Reliability theory is built on the classical test score model X = T + E, where random error (E) reduces reliability and systematic error reduces validity.
  • Hypothesis Testing — Statistical significance tests are used to determine whether a reliability or validity coefficient is meaningfully different from zero.