Evaluating Healthcare Governance Policies: A Scoping Review Protocol

doi:10.21203/rs.3.rs-927138/v1

Download PDF

Study Protocol

Evaluating Healthcare Governance Policies: A Scoping Review Protocol

https://doi.org/10.21203/rs.3.rs-927138/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background: Governance policies provide structures and processes through which healthcare systems are managed. Existing literature defines strategies to evaluate operational (e.g. program) and clinical (e.g., patient-provider) healthcare interventions; the equivalent strategies to evaluate governance policies are less well developed. The aim of the proposed scoping review is to examine the extent, nature and range of approaches used to evaluate healthcare governance policies.

Methods: Informed by the Joanna Briggs Institute guidelines and the Arksey and O’Malley framework, the proposed study will conduct a keyword search of both health and social sciences databases, including Ageline (EBSCOhost 1978-2020), CINAHL with Full Text (EBSCOhost 1981-2020), EconLIT (EBSCOhost 1886-2020), Medline (Ovid 1946-2020), Global Health (Ovid 1973-2020) and Scopus (1970-2020). The grey literature – Public Documents (desLibris), Theses & Dissertations (ProQuest) and Google Advanced – will also be searched to ensure comprehensive identification of studies. Any evaluation of healthcare governance policies published in English will be included. Findings will be presented using Preferred Reporting Items for Systematic Reviews and Meta-analyses: Extension for Scoping Reviews (PRIMSA-ScR). Our cross-disciplinary team will critically assess the identified literature.

Discussion: Findings from the proposed scoping review will provide insight into the ways in which healthcare governance policies have been evaluated and offer future research directions. Based on initial literature scans and consultations with policy workers, we expect to demonstrate the need for more robust (i.e., deliberate, methodical) approaches to evaluate healthcare governance policies, which in turn requires meaningful partnerships to enrich the transactional space between research and policy.

Health Economics & Outcomes Research

Health Policy

healthcare

governance

policy

evaluation

scoping review

Healthcare governance broadly refers to policies used to direct and authorize the delivery of health services; the extent to which operational and clinical components of the healthcare system are influenced by governance policies remains an important area of inquiry [1, 2]. Frameworks related to implementation science emphasize the strong role that governance policy have on program innovations [3, 4], but to date there exists relatively little understanding of the ways in which governance policies are assessed (and hence, the extent to which governance policies achieve their intended purpose) [5, 6, 7]. It is well known that ‘good’ governance is accountable, effective, efficient, participatory, responsive and transparent [8, 9]. Yet to date we have little understanding of the ways in which governance policies meet their intended goals [5–7].

Healthcare is delivered in complex settings. Approaches to evaluating healthcare have predominantly focused on clinical (e.g., provider-patient) and program-level interventions [3, 4]; however, to date the equivalent strategies to evaluate healthcare governance policies have yet to be developed [5, 6]. Given the tremendous (and growing) proportion of budgetary spending on healthcare [10] evaluations of policies governing the healthcare system are imperative. Well-defined evaluation approaches are needed to elucidate the major effects (intended or unintended) of healthcare governance policies, enabling decision makers to better identify aspects of policies that require ongoing innovation and transformation [5–8].

From a preliminary review of provincial government documents [11] in combination with the academic and grey literature [1, 2, 5–10, 12–14], we posit that the existing knowledge in the area of governance-level evaluation tends to be highly conceptual (e.g., theories, frameworks) and not specific to healthcare systems. At present, it is unclear what approaches have been used to evaluate healthcare governance policies; in particular, what has been measured and how [5–7, 12]. Given the dearth of applied evidence, we plan to conduct a scoping review to answer the following research question: What approaches have been used to evaluate healthcare governance policies? The proposed scoping review is intended to help fill this knowledge gap.

Overview

Our team will conduct a scoping review of the academic and grey literature with the aim of describing how healthcare governance policies have been evaluated. The proposed review will identify different evaluative practices, knowledge gaps, and provide future research directions. The scoping review is informed by Joanna Briggs Institute guidelines [15] and the Arksey and O’Malley framework [16]. Results will be reported narratively and using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) [17–19]. We will search several databases (unconstrained by geography or timeframe), identify the volume and breadth of the relevant literature, assess and map healthcare governance evaluation strategies, and from this collective knowledge suggest future research directions.

Eligibility criteria

Participants

The proposed scoping review will not use participant details (e.g., target population) as the basis for study selection. In place of participants, we will consider studies and other literature sources that describe how governance-level healthcare policies are assessed.

Concepts

The main concept of interest for this scoping review is evaluation (assessment, measurement) methodology applied to the policy cycle in healthcare governance.

Context

The context will be global. Documents will not be limited by geography or time period. We will only include literature if it directly evaluates governance-level healthcare policies (e.g., direct or authorize health service delivery).

Types of sources

The review will include any published academic or grey literature that evaluates, as the primary objective, healthcare governance-level policies. Studies that solely evaluate operational or clinical policies (e.g., standards of practice) will be excluded. Prospective studies not yet completed, commentaries, recommendations, editorials, guidelines, conference abstracts, and other literature that do not directly evaluate healthcare governance policies will be excluded. Only studies published in English will be included.

Search strategy

A health services librarian worked with the core research team to develop and execute a search strategy in Medline. The librarian then adapted the Medline search to other databases. The search strategy was peer-reviewed by another librarian using the Peer Review of Electronic Search Strategy (PRESS) checklist [19]. The search strategy included the National Library of Medicine’s Medical Subject Headings (MeSH) terms, keywords, and adjectives (two degrees of separation) related to evaluation, monitoring, policy and governance such as “Evaluation Studies as Topic, Health Policy and Health Care Reform” (see Table 1). We applied a filter to exclude animal studies. Additionally, records that were non-English, commentaries, editorials, guidelines and letters were excluded.

Table 1

Search strategy in Medline electronic database
Date	Database	Keywords	Search results
26/08/2020	Medline	• Line 1: ((evaluat* or assess or effect3 or assessment or feedback or apprais* or directive* or mandat* or strateg) adj2 (report or research or qualitative or index or indexes or methodolog)).ti,kf.	11,385
		• Line 2: (monitor? or monitored or monitoring).ti,kw.
		• Line 3: Evaluation Studies as Topic/ or Health Care Evaluation Mechanisms
		• Line 4: cost-benefit analysis/ or cost-benefit*.ti,kw
		• Line 5: (policy or policies or legislation or regulation or legislate* or regulate* or mandat*).ti,kw
		• Line 6: (policy or policies).ab. /freq = 2
		• Line 7: public policy/ or health policy
		• Line 8: (government* or governance or governed or governing or govern).tw,kw
		• Line 9: health care reform/ or international health regulations
		• Line 10: or/1–4 [Evaluating and Monitoring]
		• Line 11: or/5–7 [Policy]
		• Line 12: or/8–9 [Governance]
		• Line 13: 10 and 11 and 12
		• Line 14: limit 13 to english language
		• Line 15: limit 14 to (comment or editorial or guideline or letter or practice guideline)
		• Line 16: 14 not 15
		• Line 17: 16 not (exp animals/ not humans.sh.)

Information sources

A search for evidence will be completed in the following electronic databases: Ageline (EBSCOhost; 1978–2020), CINAHL with Full Text (EBSCOhost; 1981–2020), EconLIT (EBSCOhost; 1886–2020), Medline (Ovid; 1946–2020), Global Health (Ovid; 1973–2020) and Scopus (1970–2020). Our grey literature search will look for reports and theses from a variety of different resources including Public Documents (desLibris), Theses & Dissertations (ProQuest) and Google Advanced Search.

Study selection

All identified studies will be uploaded into EndNote V9.1 (Clarivate Analytics, PA, USA) and Rayyan (Qatar Computing Research Institute, Doha, Qatar) [20]; duplicates will be removed. All titles and abstracts will be screened against the predefined inclusion and exclusion criteria to identify potentially relevant papers. All titles and abstracts will be screened independently and in duplicate by two research assistants (RAs) using Rayyan. Three steps are planned to ensure that Title and Abstract Screening occurs in a rigorous manner. First, a pilot with 40–60 title and abstracts will be completed by three team members to test the screening tool; team members will discuss their decisions and reach consensus regarding the final inclusion / exclusion criteria. Second, a document and training process will be developed for the two RAs. As part of the training, RAs will independently screen the same (40–60) titles and abstracts completed by the team. Disagreements will be resolved through group discussion. Third, following this initial training, the two RAs will screen 50–100 new title and abstracts and then meet with primary investigator (PI) to review and discuss results. Given the complexity of the topic, several meetings are expected to ensure accuracy and consistency of screening. Title and abstracts will be labeled as “include”, “exclude” or “maybe”; any identified as “maybe” will be reviewed by PI.

Erring on the side of over-inclusion is preferred during title and abstract screening; hence all papers identified as relevant (“include”) in this stage by at least one of the RAs will be included in the second (“full-text review”) stage (i.e., no consensus finding is needed). All studies meeting inclusion criteria during the title and abstract process will be uploaded to Covidence systematic review software (Veritas Health Innovation, Melbourne, Australia). Electronic copies of full documents will be obtained for full-text screening; if there is no full text available, at least one attempt will be made to locate the document from another institution. Full-text screening will be completed in duplicate by RAs. Reasons for exclusion will be documented and PI will review the full-text results to ensure accuracy. We will document the number of potential studies identified, the number of abstracts and full-text articles reviewed, and the number of studies eliminated at each stage of the selection process. The results of the search and the study selection process will be reported narratively and using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) [17, 18].

Data extraction

A data extraction form will be developed iteratively by the study team (see Table 2 for an example). The goal of the proposed scoping review is to extract information to understand the nature and context of the governance policy (e.g., description of policy, target population), the approach (e.g., variables of interest, sources of data) to evaluating the policy, and any potential learnings (e.g., challenges, opportunities) to support future evaluations of healthcare governance. Five to ten articles will be selected to pilot, discuss, and amend the draft form. Once the data extraction form has been finalized, two team members will independently extract data from each article. Any disagreements will be discussed and resolved through consensus.

Table 2

Example data extraction
Author(s)
Year of publication
Country / Region
Type of literature
Name / description of governance policy
Target population(s) of governance policy
Variable(s) used to assess governance policy
Source(s) of data in the assessment of governance policy
Limitations of the data (if provided)

Data analysis and presentation

Studies will be organized based on country or region, variable(s) measured, and source(s) of data. The data extracted will be content analyzed generating themes as well as categories regarding evaluation approaches, processes and content. The results will be presented in a summary addressing the review objectives as well as where knowledge gaps remain. Diagrammatic figures of the study selection process and data extraction will be included. This overview describing approaches to assessment of diverse topics covered by governance-level health policies aims to inform and support the development of an evaluation framework to be used by health researchers and policy workers with the overarching goal of promoting evidence-informed governance.

Evaluation is a vital component of health system transformation [3]. Policy decisions can be enhanced when there is clear knowledge about expected and unexpected outcomes, opportunities for innovation, and “value for money” [5, 18, 19]. Developing robust evaluation plans at earlier stages of the policy cycle offers opportunities to clarify goals as well as layout various meanings and measures of success. To date, there is a paucity of practical evidence elucidating the ways in which policies governing healthcare delivery have been evaluated [2–6, 10, 11, 17–19]. The collection, synthesis and translation of knowledge regarding evaluation approaches can be used to understand the current strengths and challenges associated with evaluating governance-level health policies. Collectively, this knowledge offers opportunities for decision-makers to enhance the ways in which their healthcare governance policies are evaluated, and in turn, enhance policies that govern the healthcare system.

Findings from the proposed scoping review will identify and critically assess existing approaches to evaluate healthcare governance policies. Results will help to support health system transformation and to propose future research directions.

primary investigator (PI); research assistants (RA)

Ethics approval and consent to participate

Not applicable.

Adherence to national and international regulations

Not applicable.

Consent for publication

Not applicable.

Availability of data and materials

We have duly cited all studies; data is presented in the form of references.

Competing Interests

The authors declare that they have no competing interests.

Funding

Primary investigator (HAF) and scoping review funded by the Canadian Institutes of Health Research (CIHR) – Health System Impact Fellowship (via Institute of Population and Public Health) and Manitoba Health Seniors and Active Living (MHSAL), with contributions from the Manitoba Primary Care and Integrated Health Care Innovation Network (MPN). The funding bodies did not participate in the design of the study, collection, analysis, interpretation of data, or in writing the manuscript.

Authors’ contributions

HAF, MBD and GH conceived the study. HAF, MBD, GH and AP contributed towards the protocol design and study plan. CM developed and conducted the search strategy with input from HAF, MBD, GH and AP. HAF, MBD, GH and AP drafted the protocol, with assistance from CM. All authors read and approved the final manuscript. HAF is the guarantor of the review.

Acknowledgments

We thank Christine Neilson, Master of Librarian and Information Studies (MLIS; University of Manitoba) for peer review of the MEDLINE search strategy; Christine is the Manitoba's Health Information and Knowledge Network (MHIKNET) Librarian for Manitoba Health, Seniors & Active Living (MHSAL). We would also like to acknowledge knowledge-user members of our study team: Brie DeMone and Fiona Jeffries (MHSAL), who contributed to the study parameters and will be involved in subsequent phases of the program of research.

Osborne SP. Introduction the (New) Public Governance: a suitable case for treatment?. In The new public governance? 2010 Jan 21 (pp. 17-32). Routledge.
Howlett M. From the ‘old’to the ‘new’policy design: design thinking beyond markets and collaborative governance. Policy sciences. 2014 Sep;47(3):187-207.
Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Annals of internal medicine. 2012 Aug 7;157(3):207-10.
Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, Griffey R, Hensley M. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research. 2011 Mar 1;38(2):65-76.
Head BW. Three lenses of evidence‐based policy. Australian Journal of Public Administration. 2008 Mar;67(1):1-1.
Head BW. Toward more “evidence‐informed” policy making?. Public Administration Review. 2016 May;76(3):472-84.
Rijke J, Brown R, Zevenbergen C, Ashley R, Farrelly M, Morison P, van Herk S. Fit-for-purpose governance: a framework to make adaptive governance operational. Environmental Science & Policy. 2012 Oct 1;22:73-84.
Gisselquist RM. Good governance as a concept, and why this matters for development policy. WIDER Working Paper; 2012.
Hawkins B, Parkhurst J. The 'good governance' of evidence in health policy. Evidence & policy: a journal of research, debate and practice. 2016 Nov 18;12(4):575-92.
World Health Organization. Global spending on health 2020: weathering the storm. World Health Organization; 2020 [cited 24 Jun 2021]. Available from: https://www.who.int/publications/i/item/9789240017788
Farag MA. Guide to Policy Development. Winnipeg, MB: Office of the Auditor General of Manitoba. 2003 [cited 24 Jun 2021]. Available from: https://www.oag.mb.ca/wpcontent/uploads/2011/06/PolicyDevelopmentGuide.pdf.
Institute of Health Economics. Frameworks for post-policy implementation review. Edmonton AB: Institute of Health Economics. 2012 [cited 24 Jun 2021]. Available from: https://www.ihe.ca/download/frameworks_for_post_policy_implementation_review.pdf.
Hertting N, Vedung E. Purposes and criteria in network governance evaluation: How far does standard evaluation vocabulary takes us?. Evaluation. 2012 Jan;18(1):27-46.
Schoenefeld J, Jordan A. Governing policy evaluation? Towards a new typology. Evaluation. 2017 Jul;23(3):274-93.
Peters MD, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, McInerney P, Godfrey CM, Khalil H. Updated methodological guidance for the conduct of scoping reviews. JBI Evidence Synthesis. 2020 Oct 1;18(10):2119-26.
Arksey H, O'Malley L. Scoping studies: towards a methodological framework. International journal of social research methodology. 2005 Feb 1;8(1):19-32.
Moher D, Liberati A, Tetzlaff J, Altman DG, Prisma Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS medicine. 2009 Jul 21;6(7):e1000097.
Tricco AC, Lillie E, Zarin W, O'Brien KK, Colquhoun H, Levac D, Moher D, Peters MD, Horsley T, Weeks L, Hempel S. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Annals of internal medicine. 2018 Oct 2;169(7):467-73.
McGowan J, Sampson M, Salzwedel DM, Cogo E, Foerster V, Lefebvre C. PRESS peer review of electronic search strategies: 2015 guideline statement. Journal of clinical epidemiology. 2016 Jul 1;75:40-6.
Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan—a web and mobile app for systematic reviews. Systematic reviews. 2016 Dec;5(1):1-0.

Download PDF

Version 1

posted

You are reading this latest preprint version

Evaluating Healthcare Governance Policies: A Scoping Review Protocol

Status:

Version 1

Abstract

Background

Methods

Discussion

Conclusion

List of abbreviations

Declarations

References

Status:

Version 1