During the COVID-19 pandemic, the rush to scientific and political judgements on the merits of hydroxychloroquine was fuelled by dubious papers which may have been published because the authors were not independent from the practices of the journals in which they appeared. This example leads us to consider a new type of illegitimate publishing entity, ‘self-promotion journals’ which could be deployed to serve the instrumentalisation of productivity-based metrics, with a ripple effect on decisions about promotion, tenure and grant funding, but also on the quality of manuscripts that are disseminated to the medical community and form the foundation of evidence-based medicine.
This article is made freely available for use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.https://bmj.com/coronavirus/usage
Statistics from Altmetric.com
The hydroxychloroquine saga was perhaps the scientific controversy that received the biggest media coverage of the first 100 days of the COVID-19 pandemic. This controversy originated from Didier Raoult, a microbiologist and director of the Institut Hospitalo-Universitaire Méditerranée Infection in France, who, with his team, published a highly questionable study in the International Journal of Antimicrobial Agents.1 Despite major concerns highlighted in 17 Pubpeer comments and later in a postpublication review,2 the study and its coverage in the media and by politicians3 (1) ignited a wave of research wastage with more than 150 clinical trials across the world exploring the efficacy of chloroquine and/or hydroxychloroquine,4 (2) fostered shoddy science, including the highly mediatised withdrawal of the Surgisphere paper by The Lancet5 and (3) produced science that is highly likely to be non-reproducible. One aspect of this germinal paper is remarkable. Jean-Marc Rolain, the editor-in-chief of the International Journal of Antimicrobial Agents, works in Raoult’s institute (and reports to him) and is also a signatory of the paper. This may or may not be a problem, but without explicit mitigation it certainly gives the impression of potential conflicts of interest. The peer review of this paper was unusually fast, as it was expedited in 1 day. Such speed, even in a pandemic, is reminiscent of what one might expect from a predatory journal.6 The International Society of Antimicrobial Chemotherapy (ISAC), which owns the journal, quickly expressed its concern, stating that ‘the article (did) not meet the Society’s expected standards, especially relating to the lack of better explanations of the inclusion criteria and the triage of patients to ensure patient safety’.7 However, ISAC also stated that the peer-review process did adhere to the peer review rules in the field, highlighting that full responsibility for the peer review process of the manuscript was delegated to an associate editor.8 The journal has not implemented an Open Peer Review; we asked the authors to share the peer reviews, but to no avail. The team published four other papers (see online supplemental table 1), which were below general research standards (eg, International Council for Harmonisation (ICH) guidelines, relevant reporting guidelines), in journals where members of the team were part of the editorial board or indeed editors-in-chief. Among these, a so-called meta-analysis on the therapeutic efficacy of hydroxycholoroquine9 was published in New Microbes and New Infections (NMNI), and was at odds with all best practices in the field of meta-analyses (eg, it included a withdrawn preprint and it pooled different outcomes extracted from the same studies). It also received seven critical comments on Pubpeer. As the reporting did not respect the PRISMA statement, the methods and results were not reproducible. NMNI’s editor-in-chief also works for Raoult. A further six associate editors of the journal also work for Raoult. The scope of the journal is to serve the field ‘as a peer-reviewed, open access journal for rapid dissemination of the latest research, with a particular focus on new genomes, new microbes and new technology applied to the diagnosis of infectious/tropical diseases’,10 an unusual definition for publishing a meta-analysis on a therapeutic issue.
A highway to publication
In its 2017 report on Didier Raoult’s unit, the French ‘Haut Conseil de l’évaluation de la recherche et de l’enseignement supérieur’ (High Council for the Evaluation of Research and Higher Education), an independent authority that inspects French research units, noted that the ‘creation of this journal, which serves to publish papers rejected by other journals, is a somewhat desperate initiative’. A careful inspection of the NMNI publication output (see online supplemental table 2) showed that the journal, created in 2013, had published 728 papers up to 25 June 2020. Of these, 231 (32%) were published by at least one author on the current editorial board, 226 (31%) by one editor from Marseille, and 235 (32%) by Didier Raoult, who is not part of the editorial board.
Computing the proportion of contributions published in a journal by any single author can provide a very rough index to spot problematic journals. We explored scientific journals specialised in infectious diseases selected from the National Library of Medicine Catalogue using NMNI MeSH terms (see online supplemental methods). Among these 789 journals, 239 published at least 50 papers between 2015 and 2019.
Figure 1A shows this indicator for the most prolific author for each journal in relation to the volume of the journal’s published output. NMNI is a strikingly clear outlier, with both a large proportion of published papers by the same author (37%) and a large publication volume over the last 5 years (N=598 articles). A sensitivity analysis was computed with ‘journal articles’ only (using the NCBI publication type) in order to exclude contributions such as editorials, news items or comments. The results of this sensitivity analysis were consistent with those for all articles. However, the case of Nature Review Microbiology raises the risk of misclassification bias as this author only contributes to ‘News & Comment’ or ‘In Brief’ section articles and should not have been flagged in the sensitivity analyses. We, therefore, think that a large proportion of papers published by one author could be used as a red flag—to identify journals that are suspected of dubious editorial practice—but deserves a subsequent qualitative investigation of the journal.
We, therefore, explored the 12 journals with an Index value >10.2% corresponding to the 95th percentile threshold. The key features of these journals are shown in table 1. Figure 1B shows the distribution of the index for each author, among the five journals ranking respectively at the minimum, first quartile, median, third quartile and maximum, over the 5 years, by year. Details for the whole sample of journals are presented in online supplemental figure 1. NMNI appears consistently as an outlier over the past 5 years.
Self-promotion journals: a new type of illegitimate publishing entity?
Of course, to avoid publication bias it is expected that all researchers transparently submit all outputs of their research. However, publications are presumed successful if the research is sound enough and a rigorous, unbiased peer-review actually took place. Elsevier’s general policies explicitly state that ‘the editor must not be involved in decisions about papers which she/he has written him/herself, or which have been written by […] colleagues.’ In application of this policy, more than 40% of published papers should not have been handled by Michel Drancourt, the editor-in-chief. This is a very large proportion for the editor, supposed to be responsible for the whole journal content. It is also expected that an editor of a journal should publish editorials delineating the agenda of the journal. However, a high proportion of this type of article raises questions about the plurality of viewpoints and the independence of the journal. A similar case was described in 2008 with Elsevier’s theoretical physics journal Chaos, Solitons and Fractals, whose Editor in Chief M.S. El Nashie published 332 papers in the journal as an author.11
In contrast with the El Nashie case, NMNI appears to prioritise the productivity of a larger network of editors/authors. We suggest that there is (1) a consistently large proportion of papers published by a group of authors, (2) particularly in the presence of relationships between the editors and these authors and (3) publication of low-quality research all key characteristics of a new type of illegitimate publishing entity, that is, ‘self-promotion journals’. The first two criteria have the advantage of being simple and easy to obtain for a given journal, but they are likely to detect only the more problematic journals deserving further investigation. Coincidentally, Dorothy Bishop in a blog post reported a similar analysis for the addiction subfield of psychology12 and found that 3 of 99 journals had more than 8% by the most prolific author. These two preliminary studies—in the field of infectious diseases and psychology—are being extended in a comprehensive survey of biomedical journals to refine the description of such dubious editorial practice.13 This survey showed that prolific authors were often associated with shorter lags between submission and publication which reinforces the idea of ‘self-promotion journals’.
‘Self-promotion journals’ could be deployed to game productivity-based metrics, with a ripple effect on decisions about promotion, tenure and grant funding. COVID-19 has clearly shown the detrimental effects of such practices: authorisations issued in March 2020 for chloroquine and hydroxychloroquine for emergency use, were suspended by the Food and Drug Administration 3 months later.14 Didier Raoult implicitly acknowledged using his publication capacity as clout in his own research ecosystem, even threatening to go on strike over the signing of his own publications.15 Indeed, in France, hospitals are rewarded according to the volume of publications. Various initiatives, including the Declaration on Research Assessment (DORA),16 warn against the use of incentives based on scientific productivity, which can easily be gamed and could be related to a kind of natural selection of bad science.17 In the case of NMNI, it is not possible to ascertain the integrity or quality of the peer review process because the journal does not have an open peer-review policy.
Possible consequences of dubious editorial practice on evidence-based medicine
A cornerstone of evidence-based medicine is the use of the best available evidence that have to be obtained from trustworthy findings. By carefully managing the peer-review process, editors are responsible for the quality of manuscripts that are accepted and consequently disseminated to the medical community.18 The International Committee of Medical Journal Editors explicitly states that ‘in return for the altruism and trust that make clinical research possible, the research enterprise has an obligation to conduct research ethically and to report it honestly’.19 In our opinion, such relationships between authors and editorial board members could (1) facilitate selective publication of clinical results driven by cronyism rather than the peer-review process and (2) facilitate publication of studies with high risk of bias, studies underpowered or misreporting/selective reporting research. Once published, these low-quality positive studies disrupt evidence-based medicine at several levels (table 2).
Rewarding integrity instead of productivity
Authorship is an important component of scientific integrity, it entails responsibilities20 and any doubts on actual authorship call the trustworthiness of the science into question. Publishers such as Elsevier can easily screen their catalogue using the indicators we propose to detect outliers such as NMNI and to audit the specific processes in these journals. Independent researchers can explore and refine the index we propose on the basis of an exhaustive study across a broad range of scientific journals to explore its validity and possible variations according to the field. It is indeed time to reward scientific integrity instead of productivity, institutions, journals or publishers. Following the Declaration on Research Assessment (DORA initiative16 or the Hong Kong principles for assessing researchers21 affords a good opportunity to determine which values matter: productivity-based metrics, research quality or the societal consequences of research.
We thank Angela Swaine Verdier for revising the English and Clémence Belvèze, teaching librarian, for the research strategy on NLM Catalog.
Contributors All authors contributed to drafting this manuscript, with FN taking a lead role. All authors provided intellectual input to improve the manuscript and have read and approved the final version. CL is the guarantor of the manuscript.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.