Different approaches to change measurement

This came out of doing some recent additions about change measurement and is a bit different from my usual glossary entries as it’s about organising ways of thinking rather than about one term.

Details #

The glossary started as a companion to the OMbook: Outcome measures and evaluation in counselling and psychotherapy so this is a pretty central item I guess. I think it’s important to distinguish the following.

Purely qualitative, narrative analyses of change by client, practitioner or other. Though these are not conventionally regarded as “measurement” they: can be rated; they are what practitioners use; and they are probably most understood by the clients and practitioners. Of course, they are idiographic: how easily we can compare ratings of them across clients is a moot question but that doesn’t make them meaningless.
Quantitative but idiographic measures. These include “client/user generated measures”, i.e. the whole range of possible “personal questionnaires”, PSYCHLOPS, repertory grids and other idiographic techniques. The scores can be compared across clients but we have to be very careful about what that means statistically and psychologically.
Measurement using nomothetic measures, i.e. measurement where every client completes (you hope) the same measure. Such scores can be analysed:
- dichotomised/trichotomised (in our field, often using the four way categorisation: “stayed ‘high'”, “stayed ‘low'”, “went from ‘high’ to ‘low'” and “went from ‘low’ to ‘high'”, or by the three level degree of change: “reliably improvement”, “reliable deterioration” and “no reliable change” or the ten category “RCSC” (Reliable and Clinically Significant Change) method.
- using continuous scores
Then it’s important to distinguish analyses purely within each individual versus across more than one, often many, clients, such analyses can again be:
- of dichotomised scores
- of continuous scores.

For analyses of nomothetic data from a number of clients we move into the issues of the statistical method, broadly of whether the analyses are purely exploratory/descriptive (often undervalued!) or involve estimation or are located in the Null Hypothesis Significance Testing (NHST) paradigm. Such analyses, whether by estimation or within the NHST paradigm can either ignore non-independence of observations or try to address it. Now that software to analyse multilevel models (MLMs) is widely available it has probably become almost unjustifiable to ignore the issue whether using NHST or estimation. Sadly, that doesn’t mean that many of the MLM analyses reported in our field are reported with full explanations and explorations of their possible problems.

Finally, analyses can be graphical as well as purely producing numbers. My own take is that you should be very, very wary of any reports that don’t give you graphical ways of seeing the complexities of the data.

No one approach is “right” though some are more obvious inappropriate than others depending on the data and on what findings are sought from the data.