Article Text
Statistics from Altmetric.com
Introduction
Health technology assessment (HTA) has become a key part of assessing evidence to determine which treatments are funded. Early applications were within the centrally funded health systems of high-income countries. Recent years have seen increased use within low-income and middle-income countries, including India.1–6
Increased application of HTAs has required corresponding increases in the systems’ resources that generate or use them.7 8 It is not surprising, therefore, that many countries have found it challenging to develop the technical capacity to conduct and use HTA.3 4
In 2017, the Government of India established the Health Technology Assessment Unit (HTAIn) to promote value for money within the public health system.9–11 Regional health departments refer topics to HTAIn for consideration, and these are then assigned to one of 18 independent Regional Resource Centres (RRCs) that are commissioned to conduct HTA studies. The resultant HTA report forms the basis of a recommendation by the HTAIn Board, which is subsequently sent to the nominating department for implementation. Policy briefings, designed to educate and empower the public, are published on the economic evaluation HTAIn website (https://htain.dhr.gov.in/).
HTA capacity development within India has proceeded using ad hoc workshops,12 a certificate course in economic evaluation for HTA,13 and the development of a Masters Course in Health Economics and Technology Assessment.14 However, a competency-based framework for developing HTA skills and knowledge is considered to be important in moving to a more robust and sustainable capacity-building approach.
Here, we define competencies as a combination of attitudes, skills and knowledge that enable an individual to perform tasks or activities successfully for a given role within an organisation. In addition, a competency-based framework aligns organisational roles with a set of competencies and associated assessments.
In this paper, we report on the approach adopted by Indian HTA experts in collaboration with international faculty to identify the HTA-related competencies required to carry out key roles for the different positions within HTAIn. The scope of the work focused exclusively on technical staff within its RRCs and secretariat (defined as those managing the HTAIn appraisal process). The four stages adopted by our approach are summarised in figure 1, with detailed descriptions in the following sections.
Describe roles within mature HTA systems
As a starting point, we identified competencies from mature reimbursement systems. The systems in England, Scotland and Wales were chosen as they offer variations in practice yet share structural similarities with HTAIn (online supplemental figure S1, online supplemental materials).
Supplemental material
We searched for role details on the three principal organisations’ websites (ie, National Institute for Health and Care Excellence (NICE), the Scottish Medicines Consortium (SMC) and the All Wales Therapeutics and Toxicology Centre (AWTTC)) and one of the evidence assessment groups (EAGs) used by NICE. One of the authors (SD) interpreted this information to develop preliminary lists of the roles and their requirements within those organisations. These descriptions were amended following discussions with staff members within the organisations.
Development of role-specific competencies
Using the previous information, we categorised skill and knowledge requirements for each role into separate topics (eg, ‘statistics’ or ‘evidence review’) and different levels of expertise to define the set of competencies.
This work followed the requirements of the Mission Karmayogi (MK) framework, which was launched by the Government of India in September 2020. The MK framework aims to transform capacity building for government employees by enabling them to understand their roles and linking them to required competencies.15 16 The MK framework categorises competencies into three groups: behavioural (eg, attention to detail), functional (eg, project management) and domain-specific (health data analysis).
For our work to fit into the MK framework, we needed to specify each competency in terms of four levels. These four levels are required to cover the range of competencies required across the entire Indian public sector, not just within the HTA system. Additionally, we were asked to limit our scope to domain-specific competencies, as sufficient behavioural and functional competencies had already been defined from previous work relating to MK.
This process produced five domain competencies, covering:
HTA
Economic evaluation
Clinical evidence review
Information resources
Statistics and study design
An example of how different competencies and levels align with the skills and knowledge needed for one role within an EAG is shown in figure 2. The same information for all roles within that EAG is given in online supplemental appendix 1 of the online supplemental materials.
Adapt competencies to meet HTAIn needs
The competencies described in the UK HTA systems could not be simply transplanted into the Indian system. Consequently, they were revised by a stakeholder group led by experts from the Indian HTA system. It was composed as follows:
Three principal investigators from HTAIn Regional Resource Centres (SK, BS and SP)
Two researchers or staff from the HTAIn Secretariat at the Ministry of Health and Family Welfare (KR and AS)
One public health practitioner and two HTA practitioners from the Centre for Global Development (AM, KT and MS)
One health economist, one information specialist and one clinical effectiveness reviewer from the University of Sheffield (SD, AB and CC).
After three iterations, a final set of competencies and levels was agreed upon. The full list of HTA competencies is listed in online supplemental appendix 2 of the online supplemental materials.
Develop assessment framework
An essential part of the MK framework is the assessment of staff undertaking specific roles; however, there is no guidance as to the preferred assessment strategy. Our approach was informed by examining examples of competency-based assessment in a variety of settings. The following assessment methods were identified as most relevant: written examinations, assignments, multiple-choice questions (MCQs), portfolios, oral examinations, presentations and certification.
Further review of these methods identified two forms of MCQs: ‘knowledge-based MCQs’ and ‘task-based MCQs’. Knowledge-based MCQs ask questions relating to facts that do not require analytical work. An example would be, ‘What arithmetical measure of central tendency represents the most frequent observation’ (1) mean, (2) median, (3) mode, (4) kurtosis or (5) quartile. Task-based MCQs require students to undertake an analytical task and then answer questions that relate to the outputs of that task. A simple task-based MCQ could provide a single column of numeric data and then ask questions about its mean, median, variance, SD, etc. A complex task would be to provide a dataset and then ask questions about the results of hypothesis tests.
An initial screening of all these approaches was undertaken in order to optimise the mix of assessment methods for both students and assessors. In conjunction with stakeholders, it was agreed to focus on six methods: knowledge-based MCQs, task-based MCQs, assignments, recorded presentations, portfolios, CVs and certification.
Each skill within a competency level was matched to at least one assessment method by an experienced academic (SD, AB or CC). Initial forms of assessment were then discussed to improve consistency between the different competencies. The revised set of assessments was then reviewed by Indian stakeholders to ensure that they were locally relevant and logistically feasible.
The selection of assessment methods was based on:
Relevance to the learning outcome. For example, MCQs are not appropriate when assessing a student’s ability to write a report.
Assessor capacity. In recognising that many more public sector staff would require assessment of Level 1 competencies than those being assessed for Level 4 competencies, less resource-intensive assessment methods were prioritised for lower competency levels.
The types and numbers of assessments identified for each level of competency are summarised in table 1.
Task-based MCQs play a prominent role in the proposed assessment of HTA competencies in India as they can be designed to assess a range of simple and complex tasks, whilst also being easy to mark.
Assignments were considered appropriate for higher-level competencies, as candidates needed to be able to demonstrate critical thinking, identify potential solutions and be capable of communicating specific recommendations. These competencies cannot be assessed through MCQs; they must be assessed through written assignments (or portfolios).
Recorded presentations offer a valuable tool for roles where communication skills are required, and while some of these can be assessed through assignments or portfolios, presentational skills may sometimes be essential. Consequently, in selective cases, presentations need to be assessed directly.
Several competencies not only require demonstration of certain skills but also evidence of experience in applying them in multiple situations; portfolios were considered the most appropriate and feasible method. It is anticipated that the portfolios will allow candidates to evidence their own work, perhaps accompanied by a bespoke, overarching critique, reflection or commentary.
Certification was recognised as potentially valuable in two capabilities relating to Levels 3 and 4 Information Resources, which require a staff member to provide proof of their experience. It should be noted that these two levels were considered to extend beyond specific HTA activities; however, they are thought to be relevant to other potential public sector roles. The full set of assessment methods is given alongside the skills and competencies in online supplemental appendix 3 of the online supplemental materials.
Other similar approaches
Other attempts have been made to develop a set of competencies for HTA,17 as well as in the related but broader topic of health economics and outcomes research.18 In common with our approach, documentary analysis and expert opinion were used to develop those competencies. These previous attempts also used surveys and workshops to supplement and ensure that their competencies were relevant to a broad range of settings. By contrast, our approach generated competencies for a specific country.
Also of note is a HTA skills assessment tool that has been developed by Bidonde and colleagues.19 However, this tool is limited to the self-assessment of respondent confidence and experience in conducting the various components of an HTA and, as such, it is of limited use for the summative assessment required by this project.
Next steps
This proposed competency and assessment framework has yet to be applied within Indian organisations. While efforts were made to ensure the operational relevance of the levels, knowledge and skills, the extent to which competencies and assessments match how HTA is delivered ‘on the ground’ in India remains to be seen.
While acknowledging concern that the framework was initially based on processes found in the UK, we were careful to ensure that those systems aligned closely with those in India and that the final set of competencies was the product of extensive engagement with stakeholders in India. It should also be recognised that the English, Welsh and Scottish processes are not outliers, with other countries having developed similar processes and methods,20 21 with these being implicitly endorsed by international organisations that support their key features.22 23 As such, the key features of the proposed competency framework are expected to have widespread relevance to other agencies around the globe. Also, since its completion, the competency framework has been shown to be relevant to India by its use in the development of a short course on ‘HTA in decision-making’ for India (course report available upon request from the authors).
This framework is expected to be of value to public employees within the HTA ecosystem of India by setting out an explicit set of knowledge and skill requirements needed for the effective performance of any role that has been matched to our HTA competencies. To enable this, our competencies need to be matched to public sector roles in India. Once that is complete, the assessment of an individual’s performance against those competencies needs to be undertaken, with training being made available to support the development of staff. With these resources in place, uptake is expected to be encouraged by requiring assessments for recruitment and promotion.
Assessing the impact of the adoption of this framework will be difficult as its aim is to improve job performance and, ultimately, organisational and system performance. However, it should be possible to measure pass rates for the assessments and assess how these change over time as an indicator of improvements in the alignment of staff competencies to their roles. Other methods being considered are self-assessment of employee confidence in undertaking HTAs, perhaps using the work of Bidonde and colleagues,19 or a documentary analysis of the processes and outputs.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Acknowledgments
We would like to thank the following people for their help in understanding the roles within the HTA organisations within the UK: Professor Matt Stevenson (University of Sheffield), Ailsa Brown and Iain Leslie (Scottish Medicines Consortium), Professor Graham Scotland (University of Aberdeen), Gail Woodland and Anthony Williams (All Wales Therapeutics and Toxicology Centre), Professor Dyfrig Hughes (University of Bangor) and Pilar Pinilla-Dominguez (National Institute for Health and Care Excellence).
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors SD, AM, KT and MS collaborated on the concept and design. SD led the development of the competency framework for health technology assessment in India, with contributions from AM, KT, MS, SSK, BSB, SP, AB, CC and AS. SD created the first draft of the manuscript. AM, KT, MS, SSK, BSB, SP, AB, CC and AS reviewed the manuscript and made critical revisions for important intellectual content. AM obtained funding and provided supervision.
Funding The study was funded by the Bill and Melinda Gates Foundation (BMGF), INV-003239.
Competing interests SSK, BSB and SP are the principal investigators of HTAIn Regional Resource Centers. KT, MS and AS previously worked for the HTAIn secretariat.
Provenance and peer review Commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.