Misinterp Data

A study in the New England Journal of Medicine where the conclusions run exactly the opposite of what the data suggests.

The results reported in this flawed study:

Elective heart catheterization in stable patients is usually associated with a major complication in only 1 out of 1000 patients. This study reported that 3 of 192 outpatients (1 out of 64) in the study experienced a heart attack with elective outpatient cardiac catheterization. Only 1 patient of 189 patients with inpatient cardiac catheterization had a heart attack.
Because this difference was not statistically significant, the authors incorrectly said their trial demonstrated outpatient catheterization is safe. (Though outpatient catheterization is safe, the data in this study trended in the opposite direction.)

Some studies have have multiple errors. This study¹ concerned cardiac patients undergoing outpatient cardiac catheterization.In this one study, the authors miscounted their trial's primary endpoint, included patients that did not belong in the trial, and failed to appreciate the limitations of the small sample size in regards to reliably proving that no difference exists. Finally, the authors failed to be alarmed at the implications of their own miscounted data.

Analogy of incorrectly declaring a procedure safe:

An example that would parallel this study would be a trial to evaluate whether drawing blood with needles that have never been sterilized is safe. Assume that 2 out of 100 patients develop an infection and die with the use of the needles that have not been sterilized. Assume 0 out of 100 patients die using the fully sterilized needles. This difference would not be statistically significant. However, it would be a Type II statistical error to state that this study demonstrates that unsterilized needles are safe because statistical significance has not been reached in this small sample size.

A Type II statistical error can easily occur when the numbers being studied are too small to reliably determine that no significant difference exists between the groups being studied. A significant difference, at times may exist, but not reach statistical significance. Larger numbers of patients are often required to prove that no difference exists, compared to the smaller numbers that at times demonstrate that a difference is present. A Type II statistical error occurs if there is truly a difference present, but the study is incorrectly said to demonstrate that no difference exists between the groups.

This 1988 study¹ was an attempt to assess whether outpatient cardiac catheterization was a safe approach.

This study reported that 3 of 192 outpatients (1 out of 64) in the study experienced a heart attack with elective outpatient diagnostic cardiac catheterization. (This is in contrast to the usual rate of 1 out of 1000 complication rate for elective diagnostic heart catheterization.) One patient out of 189 patients with inpatient cardiac catheterization had a heart attack.

Implausibly, the authors concluded that these patient results demonstrated that outpatient heart catheterization was safe because this difference was not statistically significant.

If a cardiac catheterization lab actually experienced a rate of 3 heart attacks out of 192 previously in stable patients undergoing diagnostic heart catheterization, an investigation of that laboratory is in order rather than a commendation for procedural technique. Though outpatient catheterization is safe, the data in this study trends in the opposite direction. Further details

Subsequent letters to the editor revealed the unusual high frequency of heart attacks was due to the author's miscounting of data points and other trial protocol problems.

After this study was published, a letter³was written which was published in the New England Journal of Medicine questioning the investigators about the details of the patients with myocardial infarctions since it was such an unusual occurrence (and not reassuring) that 1 out of 64 reportedly low risk patients undergoing elective outpatient heart catheterization developed a heart attack in this study.

The authors answered (details⁴):They had mistakenly counted patients having a heart attack twice rather than once. This occurred even though having a heart attack as a complication of the procedure was the most important primary trial data endpoint. Furthermore, contrary to their description of the patients undergoing low risk outpatient diagnostic cardiac catheterization, they accidentally included at least one patient in the outpatient group that had a much higher risk procedure, an elective balloon angioplasty who went on to have a heart attack. These errors explained the high frequency of heart attacks that occurred in the outpatient group.

In this one study, the authors not only misinterpreted their data and ignored adverse trends, they also miscounted their primary data endpoints and unintentionally broke trial protocol as well.

1. Block P, Ockene I, Goldberg R, et al. A Prospective Randomized Trial of Outpatient versus Inpatient Cardiac Catheterization; N Engl J Med 1988: 219: 1251-55.

2. Davis K, Kennedy J,Kemp J, et al. Complications of Coronary Arteriography from the Collaborative Study of Coronary Artery Surgery (CASS). Circulation 1979: 59: 1105-12.

3. Letter. Outpatient versus inpatient catheterization. Roehm E. NEJM 1989; 320:938.

4. Letter. Outpatient versus inpatient catheterization; Block P. NEJM 1989; 320:938-939