Potential confounders in studies of complementary alternative medicine: Which study design?

The design of a scientific study should be appropriate to answer the study question but independent of the theory which generated the study question. Different study designs will be needed to answer questions related to diagnosis, therapy or prevention but may not be needed to investigate different treatments founded by different theories.

Aim

Using the example of treatment-related questions we discuss the applicability of 12 questions [Usability of Scientific Publications (USP) Questionnaire; manuscript submitted] to studies of complementary alternative medicine.

Methods

Q1 Was the aim of the study clearly stated? Q2 Was the design appropriate to answer study question? Q3 Can the design of the study be used in a confirmatory study? Q4 Were the risk profiles of the study populations similar? Q5 Was the allocation of patients to study groups concealed? Q6 Were doctors and patients continuously blinded? Q7 Was the follow up long enough to detect the defined endpoint? Q8 Were all patients included in reported results? Q9 Were adequate statistics applied? Q10 Were the results influenced by conflicting interests? Q11 Is the validity of report acceptable? Q12 Is the described effect clinically relevant?

Result

The questions Q1, Q2, Q7–Q10 and Q12 can be answered without specific instruction. Possible answers to the other questions are: Q3 The designs of some studies are ambiguous and not clear enough to be used in a confirmatory study. Q4 The risks of harm may be imbalanced among the investigated groups. Imbalanced means that most of the risk factors indicate a marginally higher risk of harm in one of the investigated groups. Q5 The answer to this question is “no” if the allocation to a particular treatment group can be predicted. Q6 In some studies doctors and/or patients can identify rather fast to which treatment a particular patient is allocated. Q11 This question should be answered by a group of advocates and skeptics of the tested treatment.

Conclusion

It is not possible to recommend a “one fits all” study design but the criteria can be specified which influence the validity of the study results. Two interesting aspects have to be discussed in more detail, the role of randomization and the threshold of validity. Increasing evidence indicates that both aspects may be more a societal than a scientific problem.