An important problem, recognised more than quarter of a century ago

In an article published in the James Lind Library, Gordon Guyatt and Andy Oxman – pioneers in the development of methods to support evidence-based medicine – explain why those of us working in the field of healthcare owe a methodological debt to the social sciences.1 American social scientists – including one who introduced the term ‘meta-analysis’2 – pioneered assessments of the effects of educational and social interventions using systematic reviews. They made clear that reviews should be conceptualised as research projects, and so needed to use scientifically defensible methods.3

One of the most important methodological contributions made by the social scientists in the 1970s was to point out that, as systematic reviews were based on surveys of the relevant primary evidence, these surveys needed to identify the relevant studies as completely as possible. The consequence of failing to take account of evidence that had not been published in academic journal articles was made clear by Mary Lee Smith and her colleagues in articles published more than three decades ago.4 5 For example, they tabulated the results of 12 meta-analyses that had used data derived both from published journal articles and from studies that had not been formally published. In every one of the 10 meta-analyses in which the comparison could be made, the average experimental effect from studies published in journals was larger than the corresponding effect estimated from theses and dissertations: the mean effect size in journal articles was 64 compared with 48 in dissertations. Furthermore, they noted that there was no evidence that the quality of research in unpublished dissertations was inferior to that in journal articles, and in some instances, it was actually better. Unsurprisingly, they concluded that failing to include studies that had not been published as journal articles could result in misleading conclusions – a message reiterated in one of the earliest textbooks on research synthesis.5

Another relevant study was done by Kay Dickersin and Curtis Meinert, cited in Egger et al.6 They found that projects associated with thesis work (doctoral and masters level) done by graduates of the Department of Epidemiology at the Johns Hopkins University School Hygiene and Public Health were less likely than other work to be published. Fifty-five (68%) of 81 graduates who received doctorates in the years 1967 through 1987 and responded to their survey had published at least one full text report in a journal or book. Publication did not appear to be related to the year the degree was granted, current employment status, the type of degree (DrPH, PhD or ScD), or the sex of the student. Although numbers were small and differences not statistically significant, students with women advisors had better publication records than those with men advisors (87.5% vs 63.1%). Indeed, women students with men advisors had the lowest publication rate of all combinations (47.6%).

The overall publication rate (68%) observed in the Johns Hopkins study was similar to the rate (70%) found in a survey of British public health trainees.7

Reporting British doctoral research on clinical topics in journal articles

Uncertainty about the extent to which doctoral research on clinical topics remains unpublished in journal articles – thus compromising the validity of evidence to inform clinical practice – has been addressed by Woody Caan and Michael Cole in an article published in this issue of the Journal.8 They found that, in an analysis based on 82 theses associated with 39 British universities, 39 (47.6%) were apparently not associated with any journal articles. Although journal articles were found for all 11 MD theses, they were found for only 29 of the 48 PhD/DPhil theses (60%), and only 3 out of 23 (13%) doctoral theses in psychology, dentistry and health sciences.

Caan and Cole note that public funding for universities and science in the UK has been increasingly concentrated within doctoral training centres, based on the assumption that universities with more PhD students offer higher standards of training.9 However, they did not find any statistically significant difference in the proportions of unpublished clinical research in the 23 universities that had only one doctoral thesis each, and those in the other 16 universities, which had multiple theses. Indeed, in the designated doctoral training centre that generated the most doctorates, only 4 of 15 doctoral students appeared to have published journal articles based on their thesis work.

Caan and Cole estimate that, each year, 3200 person-years are invested in ‘clinical’ doctorate work. If the results of half of this research remain inaccessible, this represents a massive waste of public investment.

The output of doctoral training centres should be audited and published

Like the recently reported estimates that half of the clinical trials conducted are never reported,10 the findings reported by Caan and Cole reveal a very serious problem, which has ethical and economic as well as scientific dimensions. As they note, ‘many patients and carers may be participating in clinical studies on the understanding that their results may benefit future healthcare. Their goodwill and effort are wasted if the research is never disseminated.’

Caan and Cole observe that unpublished theses are a ‘vast untapped – and currently untappable – source of information.’ However, unlike much other unpublished research, which may not even have been properly written up,11 at least dissertations and doctoral theses can – in principle – be made widely accessible, for example through ProQuest Dissertations and Theses (http://www.proquest.co.uk/en-UK/catalogs/databases/detail/pqdt.shtml). This resource claims to be ‘the world’s most comprehensive collection of dissertations and theses, the official digital dissertations archive for the Library of Congress and the database of record for graduate research.' It currently contains 1.2 million full text dissertations downloadable in PDF format, with full text available for most of the dissertations added since 1997.

Caan and Cole report that Cambridge University has set up a voluntary scheme (http://www.dspace.cam.ac.uk/) to promote easier access to some of the doctoral theses for which it has awarded degrees. But this is not a matter that should be left to the vagaries of voluntary participation. It has been estimated that 85% of the 100 billion dollars annual public investment in medical research is being avoidably wasted.12 Some of this massive waste of investment is going to be far more difficult to tackle than the current inaccessibility of doctoral theses.

The full texts of doctoral theses should be made publicly accessible at the time the degrees are awarded, and this output of doctoral training centres should be audited to guide future public investment in research and training for research.