Meta-evaluation in Primary Health Care Evaluative Studies: a Scoping Review

Background: The evaluation of health policies and programs expanded in both practical and theoretical-methodological elds, with a strong inuence on health system quality, decisions of managers, sustainability, and improvement of the primary care services. However, guaranteeing the qualication and credibility of the evaluation processes using meta-evaluations is a huge challenge for evaluators. This review aimed to map meta-evaluation use in Primary Health Care evaluative studies. Methods: We conducted a scoping review based on the Joanna Briggs Institute manual and guided by PRISMA Extension for Scoping Reviews (PRISMA-ScR). A systematic and comprehensive three-step search was performed in January and February 2021 in multidisciplinary health science databases and gray literature. No limits concerning publication dates were set. Data extraction and eligibility were performed by two independent authors and interpreted using thematic analysis. The themes were collated, and a narrative summary of the ndings reported. Results: There were 11,641 potential studies identied from the search, nally, 23 studies were included in the analysis. Four thematic groups were identied: search results; characteristics of the included studies; meta- evaluation type, involved stakeholders, and Evaluative standards applied; potentialities and challenges in the meta-evaluation primary health care. Most were summative meta-evaluations focusing on health programs and primary care management and followed the quality standards of the Joint Committee on Standards for Educational Evaluation. We observed low involvement of service users and active participation of managers, health professionals, researchers like stakeholders. We also highlight the importance of formative and democratic meta-evaluations expanding the use of evaluations. Conclusion: Results showed the potential and challenges of using meta-evaluation in primary health care during or when the evaluations have been completed. Highlighted the importance of permanent and systematic analysis of the quality of evaluations. Meta-evaluators have the technical responsibility of the evaluation process, stimulus to the action of those involved, and decision-making capacity. There is a paucity of published literature on meta-evaluation in primary health care. Hence, further studies are recommended to qualify the processes and results of the evaluations capable of contributing to the improvement and qualication of public policies and health practices.


by Arksey and O'Malley
and Levac et al. [21]: formulation of research question; identifying relevant studies; study selection; data extraction and coding; collating, summarizing, reporting, and discussing results; and stakeholder consultation.
The study was approved by research ethics committee of Onofre Lopes University Hospital -Federal University of Rio Grande do Norte (CAAE: 84537418.1.0000.5292)., and direct participation of people in the study occurred only during consultation with stakeholders. The methodology used was previously reported in a protocol registered in Figshare [22].

Formulation of research question
Study question was de ned by consensus among authors and formulated using PCC mnemonic (Population, Concept, and Context) [18]: How meta-evaluation is being used in PHC evaluative studies?

Identifying relevant studies
We observed a lack of reviews regarding the topic after consulting JBI Evidence Synthesis, Cochrane Database of Systematic Reviews, and Campbell Library. Researchers specialized in bibliographic search developed the initial search strategy, and the standard strategy was developed after a pilot search.
Descriptors were chosen according to Medical Subject Headings (MeSH) and Health Sciences Descriptors (DeCS), combining keywords and Boolean operators 'AND' and 'OR' to contemplate PCC mnemonic terms.
Additional le 1 presents the strategy performed in each database.
The following multidisciplinary health science databases were used: MEDLINE/PubMed, Scopus, Web of Science, Virtual Health Library, and Scienti c Electronic Library Online (SciELO). Grey literature was identi ed in international repositories from Europe (DART-Europe E-Theses Portal, Repositórios Cientí cos de Acesso Aberto de Portugal [RCAAP], and Electronic Theses Online Service [EthOS]), Africa (National ETD Portal South African Theses and dissertations), North America (Theses Canada -Library and Archives Canada), and South America (Catálogo de Teses e Dissertações da Coordenação de Aperfeiçoamento de Pessoal de Nível Superior [CAPES]). Google Scholar and reference lists of identi ed studies were also consulted for grey literature.

Study selection
Searches were performed between January 21 and February 12, 2021, using a three-step search strategy [8]: 1. Identi cation of descriptors and keywords, with an initial exploratory search in two databases, followed by the construction of the search strategy; 2. De nition and search across all databases; 3.
Search for additional sources in the selected publications references. Search strategies are presented in additional le 1.

Study selection followed the Preferred Reporting Items for Systematic Review and Meta-Analyses
(PRISMA-P) [23]: identi cation, screening, eligibility, and inclusion.
Mendeley software was used for identi cation, management, and organization, and duplicates were removed. The second step screened potentially relevant studies through the extensive title and abstract reading by two independent reviewers (OGBJ and CRDVS). Eligibility was performed through full-text reading by two independent reviewers. In case of disagreement, a third reviewer (SACU) was consulted.
Eligibility criteria included PHC meta-evaluation studies (quantitative and qualitative, regardless of method used); theses and dissertations (grey literature); electronically available in Portuguese, Spanish, or English languages; in accordance with research question; and from inception to February 2021; Primary studies not involving meta-evaluation, literature reviews, theoretical essays, specialist opinions, manuals, and books were excluded.

Data extraction and coding
Data were extracted and encoded using a form based on JBI template [18] and adapted by authors, containing the following information: characterization of studies ( rst author, publication year, country of origin, aim, and study design); meta-evaluation type, involved stakeholders, and Evaluative standards applied; potentialities and challenges in the meta-evaluation primary health care. Data extraction form are presented in additional le 2.
Collating, summarizing, reporting, and discussing results Narrative analysis and absolute ad relative frequencies of year, country, aim, design, and meta-evaluation type were performed. Narrative synthesis regarding evaluation criteria or standard, involvement of stakeholders, meta-evaluation objects, and learned lessons were performed using thematic analysis [24].
Results were reported as narrative presentations, boxes, and gures to facilitate information synthesis.

Stakeholder consultation
Results of this review were presented to four stakeholders (i.e., researchers with experience in PHC metaevaluation) to ful ll the following objectives recommended by Levac et al. [21]: preliminary sharing of study ndings (i.e., knowledge transfer and exchange mechanism) and development of effective dissemination strategies and ideas for future studies.

Result
In all, 11,553 articles and 88 dissertations and theses., were obtained from all databases during the initial search, of which 6,598 titles met the eligibility criteria at the title screening stage. A total of 21 titles were identi ed as duplicates and removed using Mendeley Desktop. After eligibility criteria, exclusion of duplicates, analysis of titles and abstracts, and full-text reading, in two consecutive assessments, 23 publications were included for data extraction in this review (Fig. 1). All the included studies somehow demonstrated the uses of meta-evaluation in the evaluative studies developed in the PHC.
As for the challenges of meta-evaluation in PHC, they were identi ed, in descending order: Time and nancial resource requirement more than the available [27,29,38,44]; Adequacy of evaluation standards to political, economic, social, and cultural reality, especially in vulnerable societies [30,40,42,43]; Involvement of interested parts in all steps of the evaluative process [25,31,41]; Negotiation and con icts of interest mediation between actors [31,39,43]; Consider limitations of the evaluated object when proposing recommendations [30,47]; Ideological, political, and conjunctival factors may interfere in metaevaluation utility [26,29]; Ethical risk in using results when meta-evaluation favors the prosecutor [41]; Use limited by reduced disclosure of results [25]; Evaluation standard used partially or randomly [42]; Some program designs di cult evaluation and meta-evaluation [38]; Participation of internal evaluators is not always viable [38]; Low credibility of evaluators among decision-makers limits the use of results [29]; Previous experiences little responsive and/or negative weaken the involvement of those interested [26]; Deal with possible consequences of major actors participation, such as con icts [31]; Sensibilization of involved actors to actual demands of the process [31].

Discussion
Regarding PHC, evaluative studies are considered an important element to health systems reform, while international agencies suggest a broad PHC approach based on the "Renewing Primary Health Care in the Americas" document [49,50]. In Brazil, evaluative studies were directed to PHC when this model became the focus of the health system and the rst level of contact for individuals, families, and the community.
Three social and historical processes in Brazil favor meta-evaluation predominance [51]: Brazilian National Health System (SUS) organization, which formulates several sectoral policies and requires evaluation of its results; request of international funding institutions to evaluate impacts of health programs; and partnership between Brazilian Ministry of Health and Universities to evaluate policies, programs, and projects.
The predominance of publications from Brazil may be attributed to the increased number of postgraduate programs, mainly in Collective Health eld, with research funding, and discussions regarding health evaluations [52]. Moreover, researchers and the Brazilian Ministry of Health enhanced evaluations methodologically and conceptually since 2000, contributing to debates regarding the importance of evaluative cultures [5,53].
Meta-evaluation studies of health programs and projects involve peculiar characteristics of health services. All studies developed in Brazil are related to PHC planning and management. Therefore, the focus re ects structural moments of PHC in this country. Among these, the following can be highlighted: 1) Family Health Strategy growth from 2000 stimulated studies to evaluate implantation, execution, and impact of this health reorientation in the national territory [54,55,56]. In this perspective, we point to the

Brazilian Ministry of Health initiative (supported by the World Bank), named Baseline Studies of the Project for the Expansion and Consolidation of Family Health, focused on consolidation of Family Health
Strategy and primary care forti cation. This project and the National Policy for the Evaluation of Primary Health Care, which tried to consolidate the institutionalization process of evaluation, were topics of evaluative studies [57].
2) PMAQ-AB creation in 2011 and increased number of studies evaluating its effects on primary care forti cation, access expansion, and service quality improvement [58]. With this program, nancial incentives are based on results (i.e., payment by performance), an increasing tendency in Brazil and other countries [59,60]. Moreover, external evaluation of PMAQ-AB was studied because of its detailed information and possibility of novel experiences for institutionalizing a Brazilian evaluative culture.
To Stu ebeam [10], evaluation eld has advanced, especially methodologically, and evaluators should submit their evaluations to meta-evaluation. This process ensures improved e ciency and effectiveness of evaluations and provides reliable ndings and conclusions of public, professional, and institutional interests.
According to the present results, methodological approaches of meta-evaluations triangulate techniques and informants within a qualitative approach, which is considered positive since it allows an overview on Studies suggest meta-evaluation should follow international and national standards to guide the evaluative process and assure quality [67]. However, meta-evaluation is not limited to applying criteria because they generically provide a different context according to the evaluated region.
Summative meta-evaluation was present in most studies, corroborating with Elliot and Orlando Filho [64]. This assessment is performed after evaluation to verify quality according to standards and criteria.
However, formative perspective occurs during the evaluative process and guides evaluators on planning, conduction, improvement, interpretation, and communication [68]. Formative meta-evaluation emphasizes the learning process due to constant feedback during the evaluative process, indicating solid and weak points and guiding decisions [69].

Implications for practice and research
Most studies emphasized formative meta-evaluations approaches, their pedagogical characteristics, and the opportunity to democratize evaluations; therefore, is necessary include and increase the use of this approach, guaranteeing negotiation of interests and values of interested parties, validating them on health context especially PHC. This corroborates the assertion that when evaluation is considered an intervention that can be evaluated (e.g., meta-evaluations with scienti c approaches, debates, negotiations, and learning), it stimulates changes in the reality of services to institutionalize practices and improve quality [70]. A recent movement is expanding the intention to learn with participatory, transformative, and emancipatory evaluations [71].
Expanding utility in evaluations is essential for evaluators' performance. In meta-evaluations, the term "use" is relevant in the debate about "utility" pattern [40]. Use is the evaluation outcome, an important characteristic to be considered in the evaluative process [72,73,74]. According to Patton [75], technical and methodological rigors are not su cient to ensure of utility. Therefore, evaluations must respond to interests and necessities of the audience [76]. For example, different interest groups (i.e., stakeholders) can be inserted in the evaluative process [77], increasing the introduction of other groups, such as policy developers, managers, professionals, users, and clients [78].
We observed a low insertion of stakeholders in meta-evaluation, especially service users. This was found even in formative meta-evaluations where participation of stakeholders may be expanded. Metaevaluators should stimulate action of those involved and have technical responsibility in the evaluative process and decisive capacity based on results. The involvement of different stakeholders also strengthens the evaluative design and expand data collection and interpretation. This involvement increases the use of evaluation results, purpose of evaluation designs, elaboration of instruments for data collection, and results from analysis and dissemination [79,80,81].
Formative, pedagogical, democratic, and dialogical aspects and quali cation and credibility of evaluations for decision-making are strengths of meta-evaluations. In contrast, important challenges can also be observed, such as ensuring stakeholder involvement, negotiating con icts between them, and demand for nancial resources, time, and adequacy to international evaluation standards according to reality and political, economic, social, and cultural contexts.
Our study shows limited published research on meta-evaluation of PHC, regardless the growing tendency of evaluative studies. Involvement of stakeholders was low in most meta-evaluations, indicating a gap in literature mainly among PHC users. We hope our study will stimulate research studies on formative, democratic, and participative meta-evaluations to expand the use of evaluations. We also recommend the development of more primary studies to verify quality of current or subsequent evaluations, expanding health evaluation credibility since there is increase in its applicability in health systems. Most of the included studies were qualitative, this demonstrates the need to develop more studies with a quantitative or mixed approach. Moreover, systematic review with meta-analysis could be performed to assess the impact of meta-evaluations on decision-making by managers and health service practices.

Strengths and limitations of the study
This scoping review probably is the rst broad study to map evidence on the application of metaevaluation in evaluative studies developed in PHC, providing comprehensive and detailed information on the potential and challenges of its use. The study met criteria for scoping reviews [82] and followed methodological references, checklists, and protocol [22].
Although the scoping review was conducted in line with the guidelines of the methodology, we still need to acknowledge some limitations. We did not evaluate quality of publications. However, this step is not essential due to the exploratory and descriptive nature of a scoping review. We did not include o cial government documents from gray literature. However, the search was performed to reach the highest number of publications regarding the topic. Even though databases for peer-reviewed publications and gray literature were included with no lter limits and a high-sensitivity search strategy was performed it is possible that there are other relevant studies in other databases that were not captured. It is possible researches on meta-evaluations existed under different terminologies that were not captured in the review. Nevertheless, we included DeCS and MeSH terms, and a wide range of keywords as well to help address this. We recommend future studies to conduct additional searches in those databases that were not captured by this study.

Conclusions
This scoping review mapped the application of meta-evaluation in evaluative studies developed in PHC. It was possible to consider that meta-evaluation lies in the ethical aspects of the evaluator and quality of evaluation while performing the study, when nished, or both times.
Results showed the potential and challenges of using meta-evaluation. This strategy may encourage evaluations, qualify its processes and results, support decision-making, improve of public policies and health practices, especially when it has formative, pedagogical, and democratic character. Metaevaluators are technically responsible for the evaluation process, in stimulating the action of those involved, and the decision-making capacity.
Some features of this study can be highlighted as recommendations: expansion of meta-evaluation in PHC to become a culture of both evaluation and meta-evaluation; use of evaluative standards (e.g., JCSEE) providing credibility and quality and enabling technical or instrumental values and interlocutions between individuals and knowledge construction; development of formative meta-evaluations with greater stakeholder involvement; and strengthening of meta-evaluations based on potentials of the activity, its challenges, and execution.

Consent for publication
Not applicable.

Availability of data and materials
The datasets supporting the conclusions of this article are available through the detailed reference list.

Competing interests
All other authors declare they have no con ict of interest.