1.  What did the trial actually prove and what part of the authors’ conclusions remains speculation?  (Conclusions that go beyond what the data warrants represent the most common problem in clinical trials published today.) 

2.  Are there any features of the specific treatment protocol employed that bias the results in a particular fashion?  (This is often a key question.)  

3.  What specific type of patient was studied in the trial and are there compelling reasons to think that the trial results extend beyond the particular patient population in the trial?

4.  Does a trial’s positive outcome apply only to a limited subgroup of the patient’s studied?  (This result is less secure.)

5. Additional questions if the study is a meta-analysis:

 a. How similar are the patient populations in the trials that are being combined?  (A treatment may benefit one patient population, but not benefit another patient group.  Adding divergent patient populations may obscure different outcomes that may occur for particular patient groups.)

 b. How similar are the treatment strategies being combined in the various trials?  (The closer the treatment strategies are to being identical, the more reliable the meta-analysis.)

 c. Is the meta-analysis authored by a strong proponent of one side of a medical controversy that the meta-analysis is studying?  (If this is the case, the resulting meta-analysis should be considered potentially biased until proven otherwise.)  

d. Is the meta-analysis a “network meta-analysis”?  
A network meta-analysis adds an additional dimension fo error.  Unless the treatments being studied are nearly identical, and one is familiar with the individual studies being used in the meta-analysis, it is best to be quite wary of the reliability of the results of a network meta-analysis.

Limitations of Meta-Analyses
    A meta-analysis adds similar smaller trials together in an attempt to better assess the effects of a treatment.  Often a meta-analysis is thought of as the final word concerning a medical topic.  A topic that receives relatively little attention today concerns the potential weaknesses of meta-analyses.  

back