Objectives The aim of our study is to describe the congruence in the terminology used to describe both design and outcomes of clinical trials. Terminology affects the endorsement of commonly accepted clinical experience, and of poorly defined and undifferentiated interpretations of efficacy and effectiveness. Further, choice of terms may explain the results of our congruence analysis of clinical guidelines. A review of recommendations based on 330 scenarios from 24 international guidelines for the same scenario about treatment of malignant diseases found congruence in only 15% of the corresponding recommendations.¹

Method We analysed five corresponding pairs of terms recommended in six reviews articles on the reporting the outcomes of clinical trials.^2-7 These pairs were efficacy/effectiveness outcome, experimental/observational trial, explanatory/pragmatic attitude, randomized/non-randomized allocation of intervention, and analytical/descriptive trial. We also analysed four corresponding triplets of terms used in 100 published quality of life (QoL) studies. The references of these 100 QoL studies were reported previously.⁸ The four triplets of terms were explanatory/pragmatic/not defined attitude, randomized/non-randomized/not defined allocation of intervention, primary/secondary/not defined endpoint, and efficacy/effectiveness/not defined outcome. References in online version.

Results None of the reviews included all six pairs of terms. The pairs explanatory/pragmatic, and randomized/non-randomized were included in 5/6 reviews.^2-7 The pair efficacy/effectiveness was - with exceptions^{2 5} - included in four reviews.^{2 3 5 7} The two pairs experimental/observational and analytical/descriptive were included – with exceptions⁶ – each in two reviews ^{4 6} The review by Schwartz & Lellouch² included the pair of explanatory/pragmatic and discussed efficacy but not effectiveness. The CONSORT statement⁵ included three pairs of terms but did not discuss the meanings of efficacy/effectiveness. The remaining reviews^{3 4 7} included three but different pairs of terms. Our analysis of the studies confirmed the use of an experimental (RCT) design in 91%, the assessment of the primary endpoint in 29%, the assessment of efficacy in 14%, of effectiveness in 25%, and neither of these in 60%. In 6% the study was classified as a pragmatic trial.

Conclusions The most likely reason for the observed incongruence is a lack of a consensus on terms and methods for reporting the results of pragmatic clinical trials. All reviews expect pragmatic trials to describe effects under RWC but assess these effects under experimental but not real-world conditions (RCTs) resulting in a conflict between expected and observed outcomes. CONSORT includes two imprecise statements.⁵ The review by Schwartz & Lellouch² does not use the term ‘randomized’ and cannot justify the new name of ‘randomized pragmatic trial’.⁵ Second, the aim of 2008 CONSORT is extension of its applicability to pragmatic trials. In fact, CONSORT changed ‘pragmatic’ into ‘pragmatic randomised’ and used an inappropriate reference² to justify this change. The benefit of this maneuver is the recommendation of a valid instrument for assessment of effectiveness, but it may result in harm from the resulting confusion and incongruence of guideline recommendations.