What does an anxiety score actually measure? Two questionnaires carry that label, but one mostly captures general distress and the other focuses on fearful anticipation. We can interpret the number only if we know which questions produced it and how it will be used.

01

The result is a score, but the question is about a person

A study asks whether an intervention reduces depression. Participants answer a questionnaire, and their scores fall. Did depression fall as well? The questionnaire is a useful tool, but its number is not the experience it was designed to represent. Daniel McNeish begins with this gap. Before comparing groups or announcing an effect, researchers need reasons to believe that the answers capture the psychological idea they want to study.

Psychologists often study concepts that cannot be put directly on a physical scale: motivation, reading ability, extraversion or symptoms of depression. They ask several questions and combine the responses. The work of validation asks what those combined responses mean, to whom and in which setting. It is not a stamp permanently attached to a questionnaire. A scale that helps with one question or population may require new evidence when the language, participants or intended decision change.

The consequence is easy to underestimate. If a score poorly represents its target, sophisticated statistics on that score cannot repair the meaning. A result may fail to replicate because the new study measured the concept differently, not simply because the original effect was absent. The review does not tell readers to distrust all questionnaires. It asks researchers to put measurement at the beginning of the reasoning, before the dramatic claim at the end.

02

Validation was never meant to be a single stamp

The meaning of validity has changed across a century of psychology. Early approaches often asked whether a score predicted an outside result. Later standards added questions about the content of the items, how people understand them, the internal pattern of their answers and the consequences of using the score. The shift matters because a scale can predict something and still be misunderstood, or form a neat statistical pattern and still omit an important part of the experience.

The standards described in the review increasingly treat validity as an argument supported by several sources of evidence for a particular use of a score. That view helps explain why borrowing a familiar questionnaire does not settle the matter. If researchers shorten it, translate it or give it to a different group, they should examine whether the answers still carry the meaning needed for this study.

McNeish also reports reviews in which many studies offered little or no validity evidence for the scales they used, including measures created for a single paper. This is a reporting and research problem, not proof that every unreported scale is worthless. It makes the reader's job harder: without knowing how the score was justified, one cannot tell whether a later association concerns the psychological construct or an artifact of the questions.

03

The model behind a group of questions

Suppose a short questionnaire has four questions intended to measure extraversion. A researcher expects that people who endorse one question will tend to endorse the others because an underlying trait contributes to all four answers. Confirmatory factor analysis compares this proposed pattern with the pattern actually seen in the responses. If they are close, the proposal is plausible. It has not been proved to be the only possible explanation.

The method works by estimating how strongly each item relates to the proposed factor and how the items should relate to one another if the model is right. The crucial issue is the discrepancy between expected and observed relationships. Some discrepancy is inevitable in real samples. Too much may suggest that an item also measures another idea, that a model has omitted a connection or that a question works differently than intended. A single summary of the discrepancy is called a fit index.

Fit is only one kind of validity evidence. A tidy arrangement of answers does not establish that the questions really mean ‘extraversion’ to respondents or that the score predicts relevant behavior. Conversely, a model can be judged too harshly by a numerical rule even when the items work well for a particular purpose. The review's central concern is what happens when a statistical check of internal structure is treated as the entire case for a psychological measure.

04

How a useful experiment became a universal rule

In 1999, Hu and Bentler used simulations to ask how well different fit indices separated a correctly specified model from one with a meaningful error. Because they generated the data, they knew which was which. Under the conditions they studied, values around 0.06 for RMSEA, 0.95 for CFI and 0.08 for SRMR often made useful dividing lines. These values were results of a particular experiment, not timeless properties of every questionnaire.

The original simulation included a particular number of items and factors and a specific kind of wrong model: an item related to more than one factor, but one of those relationships was omitted. The authors themselves cautioned against broad generalization. Yet their numbers were easy to remember, cite and apply. Over time many papers treated them as gates: a value on one side meant ‘valid’ and a value on the other meant ‘invalid.’ This made reviewing convenient but shifted attention away from the question the score was meant to answer.

McNeish's criticism is not that Hu and Bentler were careless. Their simulation was a more disciplined way to propose guidelines than relying only on intuition. The trouble came when conditions unlike their simulation inherited the same thresholds. A rule can be excellent for classifying the cases it was designed to distinguish and poor for a different set of cases. Knowing where a number came from is therefore part of knowing what it can do.

05

The same number can judge different scales unfairly

A questionnaire with more items does not behave statistically like one with fewer. The review describes studies in which increasing the number of items made one popular index appear more favorable while another appeared less favorable. Stronger relationships between questions and their intended factor can also make an index more sensitive to an omitted connection. Paradoxically, a scale whose questions reflect the intended factor more strongly may be penalized by a fixed cutoff.

Other details matter too: missing answers, the size of unexplained variation and the kind of response options. The threshold is static, while the circumstances that shape the index change. That can produce both false reassurance and false rejection. If a study reports only that it passed a familiar rule, readers cannot tell which discrepancy was considered acceptable or why that level of discrepancy is appropriate for the scale's intended use.

The risk reaches beyond a technical debate. Researchers may accept a questionnaire that confuses two constructs or discard one that is helpful, and later analyses inherit that choice. A clinician or policymaker should not interpret a passed fit cutoff as proof that an individual has a disorder or that a program worked. The review treats validation as a chain of evidence, and the fit number as one link whose strength depends on the model and data.

06

Look at where a model misses, not only how much

A global fit index compresses many differences between observed and predicted answers into one number. That number may hide one large problem among otherwise good matches, or many small problems spread across the questionnaire. Local fit instead examines which pairs of questions the model predicts poorly. If two questions use nearly identical wording, for example, their extra similarity may reflect the wording rather than the intended psychological factor.

The review reports that local fit is rarely described in published studies, despite methodological recommendations to inspect it. Looking locally does not produce an automatic verdict: researchers still need to decide how large a discrepancy matters. Yet it gives a more informative question than ‘Did the overall score cross the line?’ It lets readers ask whether a problem is concentrated in a particular item, in a subgroup of items or in the model's general account of the scale.

This is where substantive knowledge returns to measurement. Someone who understands the content of the questions can investigate why two items behave together, whether an omitted factor is plausible and whether changing the wording would improve the measure. A computer can identify a mismatch; it cannot alone decide what experience a respondent understood or whether the revised questions still capture the intended construct.

07

Better tests still need a better question

McNeish discusses two other alternatives. Equivalence testing places uncertainty around fit estimates and asks whether the evidence supports a discrepancy small enough to be acceptable, rather than merely failing to reject a model. Dynamic fit cutoffs use simulations tailored to the model and data under study instead of importing the same threshold everywhere. Both approaches make the decision more explicit, although neither removes the need to choose what kind of misspecification is consequential.

There is no single replacement number that makes judgment unnecessary. A stronger validation argument combines the structure of responses with the content of the questions, the population, the purpose of the score and what happens when the score is used. If a new study changes language or setting, the researcher should ask whether the old evidence still applies. A precise method is valuable when it helps answer that question, not because its mathematics gives the appearance of certainty.

Return to the falling depression score. To interpret that fall, we need to ask what changed in the person's responses, whether those questions capture the experience of interest and whether the scale behaves suitably in this group. Only then does a comparison before and after the intervention support a claim about depression. Evidence supporting the scale allows us to interpret the number. Passing a familiar statistical cutoff does not establish what the score represents or what changed following the intervention.

What to keep

  1. Validity concerns score interpretations.
  2. Cutoffs were used beyond their original purpose.
  3. Model fit is evidence, not a verdict.
?

The nuance

What this story does not let us claim

No universal number can replace an argument about the construct, population and intended use of scores.

Go deeper: concepts and optional reading practice

Reversible cards

Test the concept

1 / 1

Topic assessment

Check whether you can explain the idea, not merely recognize it.

Every answer is grounded in the explanation above. Some questions introduce new cases to test whether you can apply and analyze the ideas, not merely recall them.

7 questions4 options per questionOne correct answer
01Which statement best captures the review’s central thesis?
02Which is the most appropriate operational definition of “Construct validity”?
03Which case correctly applies the central concept?
04If two studies obtain different results, which interpretation best preserves the logic of this guide?
05Which complementary conclusion belongs to the paper’s core argument?
06Which formulation best distinguishes “Construct validity” from a superficial label?
07Which limit should accompany any application of the paper to a new case?
Answer all seven questions.

Source & scope

How Do Psychologists Determine Whether a Measurement Scale Is Good? A Quarter-Century of Scale Validation with Hu & Bentler (1999). Daniel McNeish. Annual Review of Psychology, vol. 77 (2026).

This editorial guide is an independent explanation. Consult the original publication for the studies, references and full context.

Open the original review ↗