Article Text
Statistics from Altmetric.com
Introduction
Rationale
If important evidence is unavailable for ‘systematic’ knowledge reviews, then the conclusions they reach may be wrong.1 As research supervisors, we wondered how much scientific evidence on healthcare produced for doctoral degrees (which are examined for their substantial and original research content) was missing from the wider knowledge base. While personal contacts might lead to communication of unpublished reports, clinical research that is never published will be ‘unavailable’ to most reviewers. For example, volume 342 of the online version of the BMJ 2011 included 18 papers with ‘systematic review’ in their title and reference lists with a total of 1293 items. Only 1 out of 1293 references was to a PhD thesis, and this was a self-citation by the main author of that review. Evidence-based medicine is not unique in its concerns about missing evidence, for example in ecology and conservation research unpublished theses are a ‘vast untapped – and currently untappable – source of information’.2 In sports and exercise research, comparison of published literature with unpublished theses3 revealed significant bias where the ‘effectiveness’ of interventions was overestimated.
Health researchers in Germany seem to be more concerned about grey literature than their UK peers, for example one editor created an archive of unpublished reports such as urban myths around AIDS.4 In a recent follow-up of 1100 presentations at conferences of the German Society of Orthopaedics and Trauma Surgery,5 only 36% of this research led to any journal publications within 5 years of the presentation. In all the German-speaking countries, a long postdoctoral period for Habilitation that includes publication of multiple research papers (or sometimes a published monograph, the Habilitationsschrift) is a requirement for professorial posts. However, only about 6% of postdoctoral scientists in Germany enter that career path.6
Many patients and carers may be participating in clinical studies on the understanding that their results may benefit future healthcare. Their goodwill and effort are wasted if the research is never disseminated and failure to publish is a wider ethical concern. In the UK there are specific concerns within higher education about the standards of doctoral research training of those people who are expected, in the future, to take leading roles in developing health research.7
Previously, doctoral theses have been used to profile nursing research capacity in the UK8 and Sweden.9 At that time author WC was surprised that he could find no publications in the nursing literature for an estimated 38% of nursing theses, but did not follow up his incidental observation. Similar to traditional PhD theses, there was interest in the range of new professional doctorate programmes offered in the UK10 and the knowledge base these might represent for healthcare. Many medical schools offer the experience of MD research to develop clinical academic careers, but there is a perceived need to improve the preparation of clinicians who undertake this degree.11
The initial assumption
Our starting point was that doctoral training forms good lifetime habits in medical researchers. We assumed most research theses on clinical topics would be disseminated as publications that are easily found in online databases, but a few would remain unpublished. It was anticipated the proportion of recent clinical research, linked to higher university degrees, but inaccessible as publications would be much less than the 38%, found in nursing theses above.3
Methods
In July 2011 a search for doctoral awards for clinical research was made between 2000 and 2010, in the Index to Theses in the UK and Ireland (Expert Information Limited: contains 336 000 thesis abstracts). Different universities have different areas of expertise and produce different styles of thesis abstract. Multiple pilot searches were conducted with different terms, looking for doctoral work related to human health (ie, patients, not animals, not basic laboratory sciences and not Masters-level degrees). The aim was to cover a range of clinical conditions or interventions and gather a national picture from universities. Most potential search terms tested produced heterogeneous material unrelated to healthcare, for example an unsatisfactory search in abstracts for ‘clinical trial’ yielded large numbers of theses not related directly to human healthcare (eg, with recommendations for future clinical trials to follow the student's biological research). The most efficient search used either the term ‘clinical trial’ in the thesis title or ‘clinical research’ in title or abstract. Neither term on its own gave a satisfactory range of UK institutions and health specialities. The combined searches (21 ‘clinical trial’+61 ‘clinical research’) found doctorates in all four nations of the UK. This final core dataset of 82 theses came from 39 universities, with the author, degree, title, abstract, date and place of the research. Twenty-five of those 39 universities currently include medical schools and all but one of the others have close links with a teaching hospital in the same city.
Publication of any material from this doctoral research was sought independently from two international databases, the Web of Knowledge (WoK: Thomson Reuters, covers 23 000 journals) and SCOPUS (Sc: Elsevier B V, covers 18 000 journals) up to the present. Publication dates were investigated for all material located in each database. We presumed that a publication with some resemblance to the thesis title or abstract was related to that research, even when the publication was only a small-scale report like proceedings of a conference or correspondence.
In each publication database used, the key measures were the following:
Theses from which any publications could be traced versus theses with no published evidence;
Date of the first accessible publication;
Date of the last accessible publication, if more than one paper was found;
The total number of related publications over time;
The date of other papers published by the author of each thesis, but which did not appear related to the topic of their doctoral research.
The focus of analysis was the characterisation of doctoral research that appeared to have never been published compared with research that was published. The parameters used included date of the award, place of the award, type of degree awarded and previous publishing history. Some authors had compound (‘double-barrelled’) names and the literature search included their married name and all its permutations. Where individuals had common names (eg, Jones) their university website was searched to confirm their full name. Searching for students' local city could capture publication addresses in the health service if they had not used a university address. The Index to Theses does not give the names of doctoral supervisors, therefore it is possible that some research by these students was published solely under another person's name. However the majority of clinical research publications found were multi-author papers, presumably representing a research team. Four of the five measures above seemed to be replicable when repeating small sample searches, whereas the total number of ‘related publications’ may not show high inter-rater reliability because the appraisal included material with any possible relation to the thesis. We recorded a few authors who continued to publish papers in broad fields (eg, oncology, dental surgery) long after their degree but did not include these in our count of ‘related publications’ where their earlier degree work was not the focus of those later publications. In instances where authors' published material was quite unrelated to their doctoral research, the date of their first such ‘unrelated publication’ was analysed if past experience of authorship could be a factor in subsequent publication. From the papers available online, it was discovered that one doctoral student died during the study period (they had a posthumous publication) and another had retired from their original university. This last doctor was an exceptional ‘outlier’ in our distribution, whose related publications preceded their MD by many years: it is possible that this was a doctorate ‘by publication’ awarded retrospectively on the basis of past papers. All data were first collated in an Excel spreadsheet. Coded data were then input to Statistical Package for the Social Sciences (SPSS) statistical software. Temporal information like ‘date of award – date of first publication’ was analysed for individuals using both paired t-tests and the matched-pairs signed ranks test. Within-groups analysis (like date of award and number of publications) used tests of correlation and linear regression. Between-groups analysis used χ2 tests of association for categorical variables like ‘any publications versus no publications’ and Mann–Whitney U and t tests for continuous variables like ‘date of first publication’, grouped by student characteristics.
Results
Overall pattern of degrees and publication
The 82 degrees comprised 48 PhD/DPhils, 11 MDs and 23 other professional doctorates (professions included psychology, dentistry and health sciences). The number of doctorates awarded per university during 2000–2010 ranged from 1 to 15, with 23 universities represented by only a single award. Similar literature was found in both WoK and Sc, with no significant differences in dates or number of papers per author. However, the WoK traced related publications for more authors (43/82) than Sc (38/82). We found that no one who was ‘unpublished’ in WoK had publications only in Sc. Overall, the more comprehensive WoK data suggested 39/82 clinical projects (47.6%) were unpublished. Where authors had published material (range 1–20 publications), the median number related to each thesis was three in WoK and two in Sc. All subsequent analysis in this paper will use the fuller set of literature from WoK. One author with 20 related publications was distinct within our sample: he worked in an ongoing international research programme with many outputs. Four other authors published between 10 and 15 papers each, but the modal number was just one (14/43 authors) related to their doctoral thesis.
Timing of publication
We examined degrees awarded in the decade of the NHS Plan (2000–2010), which expanded medical school capacity and promoted more healthcare research and development in the UK. The first publications related to thesis research were found over a wide range of dates (1995–2011). Publications unrelated to theses were also found for 28 authors. The earliest dates for these authors ranged between 1982–2010.
Depending on the type of doctoral degree there was a significant association (p<0.001 using χ2 test, df2) with the likelihood of publication (table 1). Publications were found for all 11 MD theses. This contrasted with publications from only 29 of the 48 PhD/DPhils (60%) and only 3 out of 23 professional doctorates (13%). The relationship between award and publication was consistent across all dates, although the number of professional doctorates rose after 2008. The Government Minister for universities and science has increasingly concentrated public funding within Doctoral Training Centres12 which are based on the assumption that universities with more PhD students offer higher standards of training. However, there was no significant difference in the proportion of unpublished clinical research between the 23 universities that had only one award each and the other 16 in our sample with multiple awards (p=0.46, χ2 df1). In the most fertile university with 15 doctorates (and a training centre), only four students published from their thesis work.
It has been assumed that most research students only publish after they have completed their degree programme. Unexpectedly, this was not typical of those 43 students who did publish. On average, the first publication preceded award of the doctorate by 1.3 years (CI 0.41 to 2.15 years before the award, p=0.005 using a paired t-test; p=0.006 using the matched-pairs signed ranks test). Publication year preceded the award in 29 cases (mean interval 2.7 years earlier), six cases published in the same year and eight published after the award (mean interval 3.0 years later). Linear regression of date of the first publication and year of award confirmed this relationship was maintained across the time period sampled (p<0.001).
For 28 students we found publications unrelated to the topic of their doctorate. There was a bimodal distribution of the earliest date of unrelated publications. Those researchers who both published from their theses and published unrelated work did the latter before their degree was awarded (mean 2.6 years earlier). There were researchers whose thesis work was unpublished but became authors of unrelated material long afterwards (mean 6.6 years later). The first unrelated publications had complex distribution and were investigated with the non-parametric U test. There was a significant difference between students who published from their thesis and those who did not (p=0.038).
Twenty-nine students published more than one paper related to their thesis. The timing of the last publication in relation to their award date was more variable than their first publication. It is possible that some papers from the later theses in our sample have yet to be written, let alone published. From the current WoK data, there are two sharp peaks in the timing of any final publication: 1 year before the doctoral award (ie, soon after their first publication) or 3 years after their doctorate. The five most prolific authors (10 or more related papers) all had their initial publication before their award and then continued to publish into that second ‘peak’ period.
Discussion
There is a substantial resource represented by accumulated doctoral research in medicine. For example in 2005 the Higher Education Funding Council for England13 reported on progress in the subject area ‘medicine/veterinary’. For a 1-year cohort of 906 full time research students a doctorate was awarded to 76% and for 565 part time students a doctorate was awarded to 53%. The median time to complete these 988 diverse PhDs was 4 years. Looking in the Index to Theses for awards in 2005 with the word ‘clinical’ in abstracts reveals 804 theses just in that 1 year. So about 3200 person-years are invested in such ‘clinical’ doctorates awarded every year. If about 46.7% of that huge investment each year is never published, that is a tragedy for healthcare.
A major limitation of our study is that not every health-related journal is covered by the databases WoK or Sc. We could have missed some published evidence, but so would many literature reviews which rely on just such databases to scope or synthesise current knowledge. These databases also index citations and these count in the coming UK Research Excellence Framework. Universities that do not encourage publication in journals that they cite will be penalised. Any secondary research is limited by capturing relevant material with the search terms used (here 82 theses). However when we used searches that captured hundred more theses, all samples were too heterogeneous in relation to ‘knowledge about clinical questions’1 to address the original focus of concern. Another weakness in our secondary research was that we treated all ‘publication’ as equivalent, therefore the overall figure of publications arising from this UK pool of research is likely to be an overestimate of the accessible ‘evidence’. If this research is ever repeated, then a more detailed appraisal of every individual item found in WoK and Sc might reveal further insight into the authors and their development as researchers.
At present is clear that few literature reviews check through PhD theses (perhaps 80 000 words each) for clinical research evidence. However, new online repositories may make that easier in future. Many science theses are examined every year in Cambridge University and it has set up a voluntary scheme (Dspace@Cambridge) for easy access to some of these.
A doctorate awarded to a work is supposed to confirm it is substantial and original, the judgement of examiners and of journal editors is different. It is possible that some of the clinical research we identified was of too poor quality to publish. In particular the theses for some professional doctorates may be based on a small, local dataset that relates to the quality of that student's practice – but may not interest a wider scientific readership. Excepting the German archive mentioned earlier,4 we lack collections of unpublished material that was rejected, to cross-check with degree awards. What has been revealed by a recent Canadian study of 27 000 doctoral students14 was that roughly one third of all published papers in health had contributions from PhD students (any relationship with their PhD thesis was not reported). Potentially, such students represent a huge intellectual resource.
What is already known on this topic
Doctoral research training is a formative time for many health researchers. A supply of appropriately trained clinical scientists is vital to resuscitate clinical research.
What this study adds
A surprisingly high proportion of doctoral projects are never published, as recorded by the two main International databases for scientific literature. Researchers who do publish their evidence, begin publishing material before they complete their doctoral programme. Several types of doctoral award are made for clinical projects, but the degree most consistently associated with publications is the MD.
Developing improved capacity for experimental medicine, clinical trials and related research has been the subject of intense efforts since a report from the Academy of Medical Sciences in 2003.15 One problem highlighted in that report was a ‘lack of appropriately trained clinical scientists’. Government funding of PhD students looks set to decrease in at least some settings16 so professions must make the best use of the people and resources that are left. Training should include the publishing of evidence, and students in our sample who did publish doctoral research, usually began before they had completed their degree. Any earlier experience of authorship may prove helpful. Practice in different UK universities and nations did not differ, so it seems the experience of the individual trainee counts. A study in five German medical schools found the key experiences were the quality of their supervisor and support from peers.17 Good supervisors and role models may even leave a ‘legacy’ extending beyond that individual student to influence other researchers for years to come.18
Acknowledgments
Staff at four academic libraries helped this project, especially Isla Kuhn in the Cambridge University Medical Library at Addenbrooke's Hospital.
References
Footnotes
-
Competing interests None.
-
Data all three electronic databases used (Index to Theses, Web of Knowledge and SCOPUS) are in the public domain, subject to a subscription fee.