Uncertainty and Coverage With Evidence Development: Does Practice Meet Theory?

Highlights:

• Although crucial to ensure optimal decision-making, uncertainty was neither systematically nor completely identified in the initial assessment dossiers of the analysed coverage with evidence development cases. Additionally, the impact of the identified uncertainties on the results was not explored in the analysed dossiers.

• Although it is desirable to fully incorporate all uncertainty in value of information (VOI) analysis, especially in a time-constrained policy process, this is extremely challenging, if not impossible. Analysts should ensure that the potential impact of uncertainties that are not incorporated in VOI analyses is clearly communicated to decision makers because these can profoundly influence the optimal policy to handle uncertainty.

• Coverage with evidence development research objectives should be discussed with all involved stakeholders and should address the identified uncertainties.

Abstract

Objectives

In theory, a successful coverage with evidence development (CED) scheme is one that addresses the most important uncertainties in a given assessment. We investigated the following: (1) which uncertainties were present during the initial assessment of 3 Dutch CED cases, (2) how these uncertainties were integrated in the initial assessments, (3) whether CED research plans included the identified uncertainties, and (4) issues with managing uncertainty in CED research and ways forward from these issues.

Methods

Three CED initial assessment dossiers were analyzed and 16 stakeholders were interviewed. Uncertainties were identified in interviews and dossiers and were categorized in different causes: unavailability, indirectness, and imprecision of evidence. Identified uncertainties could be mentioned, described, and explored. Issues and ways forward to address uncertainty in CED schemes were discussed during the interviews.

Results

Forty-two uncertainties were identified. Thirteen (31%) were caused by unavailability, 17 (40%) by indirectness, and 12 (29%) by imprecision. Thirty-four uncertainties (81%) were only mentioned, 19 (45%) were described, and the impact of 3 (7%) uncertainties on the results was explored in the assessment dossiers. Seventeen uncertainties (40%) were included in the CED research plans. According to stakeholders, research did not address the identified uncertainty, but CED research should be designed to focus on these.

Conclusions

In practice, uncertainties were neither systematically nor completely identified in the analyzed CED schemes. A framework would help to systematically identify uncertainty, and this process should involve all stakeholders. Value of information analysis, and the uncertainties that are not included in this analysis should inform CED research design.