Systematic reviews—Benefits and pitfalls

Systematic reviews have considerable impact on the discussion of complementary therapies in the scientific community, the media and on health care decision making, as they are considered to be the most reliable tool to summarize and assess the available evidence on a defined question. This lecture will discuss strengths and limitations of systematic reviews, both in general and in relation to complementary and alternative medicine (CAM). Well-performed systematic reviews clearly define the questions addressed, use systematic methods and describe them carefully, assess the quality of the primary studies and summarize their findings in a transparent manner. However, review questions in CAM tend to be broad which often leads to results which leave wide room for subjective interpretation. For searching the literature comprehensively it is sometimes necessary to go far beyond conventional electronic databases, but the quality of the material identified in other sources seems often doubtful. Small changes in selection criteria can influence strongly which studies are actually included. Publication bias, insufficient reporting in primary studies and problems related to quality assessment apply to both conventional medicine and CAM. In conclusion, while systematic reviews are (at least currently) without alternative, they have to be read and interpreted with caution. And in many cases, particularly in the area of CAM, they will leave a lot of room for controversial discussions.