The commentary on the trial by Marso et al1 assessing the cardiovascular (CV) safety of a new agent for treating type 2 diabetes highlights a recognised dilemma in clinical research on CV risk in assessment of new therapeutic agents. That dilemma is that the baseline risk of CV events in the population is reduced with better adherence to measures known to reduce risk like statins and blood pressure control; as a result the incremental benefit of new agents is smaller.2 Recognition of the limitations of surrogate end points has resulted in increased focus on reporting on patient-important outcomes in clinical trials.2 This means that the cost of doing superiority efficacy studies for benefit on CV outcomes is increased.3 This expense is justified for studying investigational agents that have been demonstrated to be safe with a good effect on surrogate outcomes, but is prohibitive in the preapproval phase.3

In response to safety concerns raised regarding diabetic agents approved on their ability to reduce glycated haemoglobin, the Food and Drug Administration (FDA) in the USA introduced new guidance to industry in 2008.4 As per the FDA guidance, in order for investigational agents to be approved they added the requirement that they demonstrate no excess increase in CV risk in large safety outcome studies. They suggested non-inferiority (NI) studies of the new agent compared with placebo may be used. Clinicians are familiar with randomised controlled trials (RCTs), which compare a new and a standard treatment (or placebo) to determine whether the new treatment is superior. Another approach is the NI trial. In the NI RCT the question is whether the new treatment is not worse than the control by a defined amount called the NI margin. NI RCTs have the potential to lead to invalid conclusions.2 ,5 It is important that clinicians understand the limitations of these trials and how to interpret their results.6 ,7

It is reasonable to consider the NI design if the new intervention is anticipated to have meaningful benefits aside from those related to increased effectiveness compared with standard therapy. Examples include treatments that are less costly, toxic or have fewer side effects, and where it would be unethical to use a placebo.5 ,6 NI RCTs have historically not been used if the standard therapy has not been proven to have clinically meaningful efficacy, as demonstrated with adequate precision in robust trials.5 ,6

The choice of NI margin is critically important. Statistically, usually the NI margin is set to preserve 50% of the lower bound of the CI of the magnitude of the effect of the active control versus placebo as determined in systematic review meta-analysis of historic trials.5 It is statistically reasonable to test for superiority in NI trials. From the clinicians' and patients' perspectives the choice of the NI margin has to make clinical sense. The choice of the NI margin drives the number of events required to show NI.3 This is illustrated in table 1. From a cost perspective a generous NI margin means that fewer events are required. Small numbers of events can affect the ability of the trial to detect harms in important secondary outcomes and prespecified subgroups.

The FDA considered that there is evidence that improved glycated haemoglobin is associated with improved microvascular complications of type 2 diabetes, so they stipulated the need for confidence that there is not a 30% increase in excess CV morbidity and mortality compared with placebo, determined through independent, adjudicated evaluation.4 In the preapproval phase it is sufficient to demonstrate there is not an 80% increase in excess CV morbidity and mortality.4 However, if the studies in the preapproval phase do not exclude the excess risk of 30%, then the FDA requires a separate postmarketing randomised safety trial. Clinically, this means that if a new agent achieves this threshold of not being 30% worse than placebo it is declared ‘non-inferior’.

There are excellent reviews summarising considerations of some of the different approaches for doing this work.3 ,4 ,8 The CV events required to achieve these generous FDA cut-offs for ‘NI’ are summarised in table 1.

The use of NI studies in assessing the CV safety of new agents compared with placebo is a significant departure from the usual rationale supporting their use. The use in this case is to provide a ‘quick look’ to rule out excessive CV risk of the agent compared with placebo. In order to do this expeditiously, high-risk populations with their associated higher CV event rates are selected for these preapproval studies.3 There is incentive to have the event rate accumulate quickly as there is no expectation of superiority so there is no advantage to run the trial for a long period of time in order to enhance the differences in the treatment arms.3

One wrinkle is that these are event-driven trials.8 In fact, the studies ‘may employ group sequential designs that permit interim monitoring of accumulating data with the possibility of stopping for efficacy or futility’, which is discussed in detail by Geiger et al.8 It is well known that stopping trials early for benefit can result in implausible estimates of efficacy in superiority RCTs.9 Unlike superiority RCTs where the numbers of CV death are typically low compared with myocardial infarctions and strokes, in these studies CV death rates are much higher. Consider the examples of the two CV safety NI trials that concluded superiority in the primary outcome (composite of CV death, non-fatal myocardial infarction and non-fatal stroke) compared with placebo in table 2. In both studies, all patients were encouraged to have optimal diabetes management, for example, with statins, good blood pressure control and good glycaemic control.

It is interesting to note that in EMPA-REG and LEADER, the significance of the primary composite outcome of time to event of non-fatal myocardial infarction, non-fatal stroke and CV death is being driven by CV death, with neither stroke nor myocardial infarction achieving significance. In fact, in EMPA-REG the CI on the HR for stroke extends to a 67% increase in non-fatal stroke compared with placebo at the upper bound. In the discussion of the EMPA-REG study the investigators comment, “Notably, reductions in the risk of cardiovascular death and death from any cause occurred early in the trial and the benefits persisted throughout the trial.” Is it possible that in this high-risk population with a high rate of CV death that through random chance the death events happened earlier than in other NI trials of other agents with the same composite outcome that did not demonstrate this finding? It would be interesting to see superiority RCTs of these agents to see if the result would be repeated.

This NI approach has resulted in large-scale, adequately powered, end point adjudicated trials that have demonstrated that some new agents do not reduce risks of CV events and identified adverse events and side effects, which may have previously become apparent only with postmarketing surveillance.3

Menon and Lincoff3 highlight that the downside of this NI approach is that patients with low-risk and early disease might actually have greater potential to reverse and mitigate the damage caused by diabetes mellitus but are not the focus for NI trials. This regulation may disincentivise conducting long-term trials designed to focus on benefit. This reduces rich data on the long-term effects of treatment benefit in important subgroups, and adequate power to show robust estimates of harms.

Clinically, a rational consumer might entertain a small risk that the new agent is actually slightly less effective, if it provided other tangible benefits. It is more difficult to understand the appeal of an improvement in glycaemic control (associated with improved microvascular complications) in exchange for any worsening of CV risk compared with placebo. As in other areas like HIV research, which use NI trials extensively, there are calls for follow-up superiority trials for promising agents.12

With the FDA guidance, the NI approach has resulted in a number of exciting new agents approved for use in type 2 diabetes, with confidence that they are not associated with excessive CV harms compared with placebo. Agents showing promise with this approach should be followed by appropriate superiority trials, with prespecified subgroups with sufficient power and duration to provide robust estimates of harms and benefits to inform clinical decision-making.