Subgroup Analysis

Particularly vulnerable to error, is the post hoc analysis of trial data when a number is derived retrospectively from trial data and is then said to separate the responders from the nonresponders.

An analysis of the CARE study data³ by prominent investigators was said to indicate that treatment of LDL cholesterol to a level below 125 mg/dL was not helpful in patients with coronary disease (blocked arteries of the heart). This was later shown to be erroneous.

Similarly, a different post hoc subgroup data analysis concerning which patients with a cardiomyopathy (weak heart muscle) benefit from an implantable defibrillator led to erroneous conclusions by the Medicare* administration.

Dr. Peter Sleight and colleagues have written insightful comments in regards to the limitations of subgroup analysis.^1,2

Dr. Sleight and the ISIS-2 trial investigators performed a subgroup analysis of patients in the ISIS-2 trial by astrological sign to show the potential limitations in reliability of subgroup analysis. details
This subgroup analysis suggested that the treatment was quite effective and statistically significant for all patients except those born under the sign of Gemini or Libra.
The difference in outcome with respect to astrologic sign was naturally an artifact and would not be reproducible in subsequent studies which was the point of their analysis.

Validity of subgroup analysis:

The view of this website is that subgroup analysis can be useful, but the validity tends to be inversely related to the number of subgroups which are analyzed. A study is not immune to an incorrect subgroup analysis outcome simply because the subgroups have been prespecified. This is particularly the case if there are a large number of prespecified subgroup analyses.

(If 20 subgroup analyses are prespecified, then it is expected that one of these subgroup analyses may show a false result for a P=.05 probability relationship.) Part of the benefit of a prespecified subgroup analysis is that there are necessarily fewer such analyses than the almost unlimited number of ways to subdivide the data in a post hoc analysis after the trial results have been obtained.

What is the reliability of a finding for a small subgroup of a trial who unexpectedly have a different outcome from the rest of the group?

In particular, if a given therapy has a highly significant and strongly beneficial effect for the group as a whole, a subgroup analysis that results in an unexpected finding that certain subgroups do not have benefit is frequently incorrect.

In fact, it is more likely that the unexpected subgroup finding which runs counter to the group finding, is simply not valid. As pointed out by Dr Sleight, it is more reliable to assume that the subgroup actually had the same outcome as the overall group.

Unexpected results from a subgroup analysis tend to be more useful as a potential starting place for a subsequent clinical trial, rather than being viewed as a definitive result.

For an excellent look at potential limitations of subgroup analysis the following two articles are recommended:

1. Debate: Subgroup analyses in clinical trials: fun to look at- but don’t believe them! Peter Sleight. Current Control Trial
Cardiovasc Med. 2000 1(1): 25-27.

2. ISIS-2 (Second International Study of Infarct Survival). Lancet 1988: ii: 349-360 (pages of interest 356-357)
----------------------------------------------
---------------------------------------------

See right column of this page for additional examples of subgroup analysis leading to incorrect conclusions.

Reference cited in right column of this page:
3. Sacks FM, Moyé LA, Davis BR, Cole TG, Rouleau JL, Nash DT, Pfeffer MA, Braunwald E. Relationship between plasma LDL concentrations during treatment with pravastatin and recurrent coronary events in the Cholesterol and Recurrent Events trial.
Circulation. 1998 97:1446-52.