Information Technology in the Dissemination of Results of the Primary Health Care Evaluation in Brazil: A Qualitative Study

Background: Considering advancements in information technology for disseminating results of evaluations, it is essential to demonstrate the functionalities and performance of these practices in order to achieve the objectives established by the evaluation proponents. This study aimed to evaluate the use of information technology for disseminating results obtained from the Primary Health Care evaluation in Brazil. Methods: A qualitative single-case study was performed with triangulation of techniques and key informants. Analyses of public documents and �eld research were conducted. Convenience sample of 54 participants (seven from the Department of Primary Care/Ministry of Health and 47 from teaching and research institutions) was selected from the external evaluation team. Content analysis was performed, and the Atlas ti.8.4.24 software organized, managed, and categorized textual data. Codi�cation was performed by peers (three researchers) while experts validated codes (two researchers). Right after, codes were extracted and categorized. Results: Dissemination of results is complex due to multiple components. Incorporation also favored advances in data observation and implementation of data openness, becoming available for professionals, policymakers, and universities. Challenges regarding delivery time of reports, access restriction, lack of interest, and digital skills in�uenced reaching expected goals of the proponent. Conclusion: Information technology for disseminating results of evaluations may improve performance of health teams and managers and rede�ne priorities of work processes to strengthen Primary Health Care. Emphasis in planning, technological infrastructure, digital inclusion, and policy decision making is recommended to disseminate data effectively to the population.


Abstract
Background: Considering advancements in information technology for disseminating results of evaluations, it is essential to demonstrate the functionalities and performance of these practices in order to achieve the objectives established by the evaluation proponents. This study aimed to evaluate the use of information technology for disseminating results obtained from the Primary Health Care evaluation in Brazil.
Methods: A qualitative single-case study was performed with triangulation of techniques and key informants. Analyses of public documents and eld research were conducted. Convenience sample of 54 participants (seven from the Department of Primary Care/Ministry of Health and 47 from teaching and research institutions) was selected from the external evaluation team. Content analysis was performed, and the Atlas ti.8.4.24 software organized, managed, and categorized textual data. Codi cation was performed by peers (three researchers) while experts validated codes (two researchers). Right after, codes were extracted and categorized.
Results: Dissemination of results is complex due to multiple components. Incorporation also favored advances in data observation and implementation of data openness, becoming available for professionals, policymakers, and universities. Challenges regarding delivery time of reports, access restriction, lack of interest, and digital skills in uenced reaching expected goals of the proponent.
Conclusion: Information technology for disseminating results of evaluations may improve performance of health teams and managers and rede ne priorities of work processes to strengthen Primary Health Care. Emphasis in planning, technological infrastructure, digital inclusion, and policy decision making is recommended to disseminate data effectively to the population.

Background
At the end of the evaluative process, there is a motivation to learn, understand, and share results among involved parties and society members [1]. Several evaluations collect valuable and informative data regarding their programs and services; however, proponents often do not share these results with stakeholders [2].
In this sense, dissemination of results is essential to conclude the evaluative process. The goal is to completely disclosure data through impartial and consistent reports [3]. Personnel responsible for the evaluation must ensure that all involved individuals and institutions fully access results [4].
Dissemination of results can make data, information, and knowledge available and usable by applying diverse channels (communication routes) and modalities (actual layout of communication). Its implementation requires adequate planning to achieve public policymakers, researchers, managers, professionals, or health system users [3] and may change policies, programs, or practices and social, nancial, and political support [2].
Data are more accessible, readable, and free of restriction due to the development and use of Information Technology (IT), which helps collect and process information in different ways using digital tools [5,6]. However, this is not restricted to exposition since data infrastructure is being actively developed in healthcare to improve decision-making and allow the creation of products, services [5] and new methods for data visualization to ll gaps between producers and consumers of knowledge [7]. This reality can be observed in an extensive evaluation of Primary Health Care (PHC) in Brazil, namely External Evaluation of National Program for Access and Quality Improvement in Primary Care (AE/PMAQ-AB), which veri ed a set of patterns regarding access and quality. Some questions were used for team certi cation and information gathering to guide improvements in public health policy. The program had three evaluative cycles AE/PMAQ-AB data were systematized and publicized to facilitate and optimize its use [10]. Transparency for disclosing results is also in the guideline of the program [12]. To attend to this demand, evaluators provided technological tools supported by IT, such as online disclosure of descriptive and analytical reports, the Portraits of Primary Health Care, and publication of microdata (i.e., data obtained in the lowest level of disaggregation) [8-10]. As a result, Brazil presents a large national data repository on PHC [13] used by policymakers, academic community, and government agencies [10,11].
Currently, interest increased in the eld of dissemination of results, speci cally regarding data observation and use. Some authors claim this topic achieved a critical stage capable of in uencing the evaluation experts to search for better strategies for communication projects [14][15][16]. For this, empirical research is needed to evaluate functionality and performance of these practices to reach expected goals established by proponents [16].
Despite research demand and the increased number of open data sources available in health [17], the way evaluation can use technology to open new channels for data disclosure by different stakeholders is considered underexplored. This study aimed to evaluate the use of information technology for disseminating results obtained from the Primary Health Care evaluation in Brazil.

Study design
This study was conducted inside a larger research project investigating standard of quality of AE/PMAQ-AB. A qualitative single-case study was performed to evaluate IT use for disseminating results [18]. Triangulation of techniques and key informants were adopted to establish different analyses.
Sociomateriality was adopted to understand the interaction between evaluators and IT and implications of practices in the dissemination of results [19].
Evaluation was intentionally chosen due to its innovative technological system for dissemination of results and protagonism in the National Program for Access and Quality Improvement in Primary Care (PMAQ-AB), whose objective is to induce the expansion of access and quality improvement in order to allow greater transparency and effectiveness of government actions directed to Primary Health Care. The PMAQ-AB has an extensive, complex and innovative payment system that considers the performance of primary health care teams [8].

Population sample and study location
Initially, we analyzed documents from the Brazilian Ministry of Health regarding dissemination of results. Public documents presenting methods for disseminating results and online platforms were included. Five documents were analyzed (Table 1).  [20].
The item "Meeting stakeholder needs through the result dissemination method", representing the subdimension "Result dissemination" inserted in the dimension "Sharing Lessons Learned" was used. The following questions guided data collection: "In your opinion, were documents available (with results, analysis, and guidance) to those interested in the evaluation process?", "Did availability occur at an opportune time?", and "Did methods for disclosing results attend necessities and interests of those involved in the evaluative process?".

Data collection procedures
Seven semi-structured interviews were performed: one from the coordination of the General Commission for Monitoring and Evaluation of Primary Care from DAB/MS; and six from the general coordination of IEPs. The focal group followed Kitzinger [21] and was performed with one moderator and one rapporteur. Seven focal groups were conducted: one group with seven technicians from coordination of DAB/MS, and six groups with at least six researchers from AE/PMAQ-AB. Participants were informed about study objectives and voluntary participation before data collection. Informed consent form and voice recording authorization form were obtained from all participants [22].
Interviews and focal groups were recorded (audio) for approximately one hour. Few notes summarizing answers were taken. In the focal group, notes were taken by the moderator. Right after, all material was transcribed and transcripts were read and compared with original recording.
Interviews and focal groups were identi ed only with the rst letter of the technique (I) or (FG), followed by sequence of data collection (e.g., 1, 2, n.). The evaluator was identi ed as I-DAB and FG-DAB.

Data analysis
Atlas ti.8.4.24 was used for organization, management, and categorization of textual data. Content analysis was conducted according to Bardin [23]. The software allowed codi cation of recorded units (categorization basis) and organization in groups by code, with possibility of elaborating analytical memos and registering comments regarding empirical data.
Peer reading strategy (group of three researchers) was adopted to select quotes (message segment-context unit) and their linking to a speci c code.
Transcriptions were carefully read in this step, and codi cation occurred after consensus (Table 2). Comments were noted, and analytic memos were constructed using experiences and references. This moment assured familiarity with corpus data and initial systematization of ideas to interpret quotes. Speci c reports were elaborated for each code group using the software and submitted to experts (two researchers) to validate relations between quotes and codes (recoding could occur whenever needed). In the following step, corpus encoding was concluded to allow interpretation and inference [23].
After these steps, we separated all codes of the item "Meeting stakeholder needs through the result dissemination method" from subdimension "Result dissemination". Therefore, the material was analyzed and organized in four thematic categories: IT use to disseminate AE/PMAQ-AB results; online reports in information disclosure process; Portraits of Primary Health Care as open data strategy; and database access in a virtual environment. Technology was essential to disseminate results, especially for information transparency and ease of its use, as highlighted below: [...] data systematization and publicity obtained clearly and objectively during the third cycle to facilitate and optimize use of data. From this perspective, the goal is always to move forward, seeking new consulting tools for the produced information. [...] systems, microdata, individualized reports per state, municipality, and team can be performed by accessing the e-Gestor and DAB/MS portal (D1).
According to the interviewer, despite the mechanism of dissemination of results, advancements regarding accessibility are needed due to di culty in using the produced information.
[...] we need advances in methods even more accessible for managers and health workers, for permanent use of results in action plans to improve what is being performed (I2).
Despite the use of technology, dissemination of results to interested parties at an opportune time is still challenging due to di culties regarding investment, reduced technical team, and large number of variables in the instrument, according to DAB/MS: We are trying to solve problems that hinder availability of results at an opportune time. Investments in technology and professionals are di cult. We lack a stronger IT development group to make these data available; lack of su cient technical team; the instrument has many variables, making its validation di cult (FG-DAB).
Online reports in information disclosure process The main purpose of reports was data and information sharing, mainly for managers and their teams, to contribute to action plans for PHC. According to the quote below, reports are available in a system developed by the demandant.
It is possible to access analytical and descriptive reports in the e-Gestor system, an important tool offered by the Ministry of Health for workers and managers to re ect some evaluated aspects of PMAQ. [...] promoting its use in local and municipal planning and resetting processes regarding work of teams and management to strengthen PHC (D2).
Technology use enabled availability of analytical reports to public access. These reports show data analysis from AE/PMAQ-AB and some recommendations to improve team performance.
Analytical reports are public documents available at the Primary Care Department/PMAQ/e-Gestor that consist of external evaluation analysis and suggestions to improve team performance (D3).
The main objective of this report is to provide analytical elements to support discussion and re ection of results and improve important services.
[...] to support primary care managers and teams in re ecting their work process and organization of health service access, from welcoming to spontaneous demand and team schedules. [...] may support discussion and re ection to improve these services (D3).
For DAB/MS, these reports present accessible data organization and availability, with pleasant navigability for interested parties.
[...]analytical reports have more pleasant interface and navigability, all with public access; any person can have access. [...] This improvement is attributed to quali cation of the instrument and data, not to mention that tablets are better programmed (FG-DAB).
Despite the effort to structure reports with interface of easy access, managers and health teams still experience di culties: [...] analytical reports are great; the problem is that people cannot read information contained in the reports. Therefore, we have a problem because the municipality takes that report and does not know what to do with it [...] (I5).
Only managers have access to descriptive reports, and authentication with login and password is needed for information regarding PMAQ-AB team certi cation. Results show the possibility to follow team performance.
Descriptive reports are public access documents available at the portal of Primary Care Department/PMAQ/e-Gestor that contains PMAQ results by team, municipality, and state. Indexes are presented and compared in these reports, and teams receive certi cation in the rst and second cycles of the program (in previous cycles) (D2).
Limited access of the manager to descriptive reports was mentioned as limitation to dissemination of results since information was restricted and used according to interest of the manager. DAB/MS considers data openness could increase usage: Descriptive reports are under login and password of the manager, which is less interested in discussing these results. Maybe with public access, direct access without bureaucracy for teams, use of reports would have been greater (FG-DAB).
DAB/MS considers that methods for disclosing results are preliminary; however, it recognizes efforts to improve reports and contribute with its use by managers and teams.
[...] I think our methods for disclosing results are still very preliminary; we have few reports, we improve in each cycle, but we are far from offering reports to contribute with use of results by manager and team workers for practical changes [. Main resources available from Portraits of Primary Health Care provide data utility and produce information to be used by interested parties.
Appropriation of results of the third PMAQ-AB cycle and use and transformation of data into useful information in both primary care monitoring and management processes for teams and managers are one of the main goals of the proposed system (D4).
Organization for data access is performed by different methods, depending on objectives of interested parties.
Data can be accessed in four ways: by thematic, which provides variables grouped by theme and it is possible to reach the lowest level of disaggregation that are questions from the questionnaire; access by variables that compose the questionnaire; certi cation access, which provides information regarding performance of evaluated teams; and the last access is crossing variables, which crosses data from PMAQ external evaluation (D4).
As described above, thematic access allows simpli ed navigation of the instrument; access using variables can be performed individually or together by crossing variables of the instrument. Performance of evaluated teams can be veri ed using certi cation access.

Database access in virtual environment
Public access to AE/PMAQ-AB microdata (database) (Ministry of Health website) was another channel used for disseminating results. The virtual environment to access microdata comprises six modules, and data sheets related to each module can be downloaded according to geographic area (Brazil, State, and City). This sheet contains evaluated data and a dictionary to explain terms and codi cations used for results (D5).
Despite the concern about data organization and availability, work with dataset is complex and may hinder the production of information for knowledge use: [...] it is complex to transform those data into information. [...] I think that translation those data in a way that people can use in practice is challenging (I7).
Moreover, DAB/MS rea rms the di culty in using and understanding the available microdata.
[...] I question if our way of providing data facilitates its use and understanding. Microdata is di cult to use. We know that managers do not use all data, and we have not facilitated this use (FG-DAB).

Discussion
This study evaluated IT for disseminating AE/PMAQ-AB results. Technology use in evaluation allowed the development of different disclosure channels and methods for data observation, making results available, accessible, and interactive for interested parties.
However, after analyzing speeches, reports did not reach proposed goals due to delayed disclosure, probably because of lack of interest by managers and teams. Restricted access to descriptive reports and di culty in understanding microdata were considered limitations for using reports. Most of these challenges may be overcome with adequate planning for disseminating results. To ensure greater accessibility, the plan must be designed in the early stage of evaluation and continued regardless of results [24].
According to the literature, online platform makes evaluation results more available for professionals and policymakers and establish connections between academic and evaluation elds [25]. This context offers an opportunity to participate, explore, re ect, generate knowledge, and increase comprehension of the evaluative process [14]. However, disclosure of results must be performed carefully to avoid overestimating practical effects of data observation using technological platform [16].
Analytical reports from AE/PMAQ-AB present pleasant navigability. Results are available by different methods (e.g., infographics, graphics, tables, and textual elements) and provide information to a larger public. Information from reports must be precise, reliable [26,27], adjusted for the interested group, and provide information regarding who performed the evaluation and relevant discoveries to allow individuals to produce their judgments [28].
Adjustment of reports in the evaluation eld is essential to interested individuals since dissemination of results is one of the main standards of quality. Efforts to make data interpretable are needed; otherwise, the reader would need assistance to interpret and comprehend information [15]. Currently, technology made visual resources more accessible for evaluators, who should consider capacitation to reduce frustration and misinterpretation [14].
Regarding learning process and elaboration of knowledge, the combination of visual and verbal explanation may increase comprehension and interest [14; 16]. Although reports are being published online to ensure accessibility for general public, researchers, managers, and professionals, evaluators must ensure that collected data provide intense dialogues with community or health service users, stimulating their participation in decisive processes.
Microdata is another channel used to disseminate results, in which demandant of AE/PMAQ-AB opt to make database publicly available for access, download, analysis, and knowledge generation. However, it is not known whether this method facilitates understanding or use by municipal managers.
Although technology use enhances extensive data collection and storage that can be shared using servers or cloud platforms [27], capacity of analysis is low [14; 27]. Nevertheless, the way government manages raw data is criticized because availability must be collaborative to build shared knowledge spaces [5].
Portraits of Primary Health Care completes the mechanisms for disseminating AE/PAMQ-AB results. This system allows users to transform large data sets into smaller subsets to produce useful information that was previously di cult without this interactive tool. This tendency remains inserted in the main purposes of data observation since it increases comprehension and participation in the evaluative process [14].
Guided by data openness concept and considering a large amount of information, data observation allowed individuals to interact directly with results.
This reality may be associated with development of technological tools and internet access, stimulating evaluators to publicly share data and become information consumers [14; 25]. The mechanism used by AE/PMAQ-AB to disseminate results demands digital knowledge and skills beyond evaluative practices. This continuous development of data observation and interaction and advances regarding data openness may become common in the evaluation eld and reorganize evaluative practices [29].
Efforts of AE/PMAQ-AB proponents are observed to implement open data to promote transparency, use, and involvement of main policymakers, government agencies, and academic community. Moreover, productive partnerships between government and population may solve public problems and social changes. Data are considered source of power, and its opening is a strategy to redistribute power [6].
Therefore, it is necessary to progress from simple knowledge dissemination to action for effective use of data in decision-making; otherwise, this dissemination will be characterized only as information transparency [17; 30]. Regarding evaluation in the health eld, the use of technological resources to disseminate results may favor social participation due to the possibility of amplifying different narratives and allowing insertion of the population in public management. Participation of the population and dialog between governments and citizens may occur according to open data management [31].
As indicated, more accessible tools and active interventions for Knowledge translation are needed to eliminate possible gaps and, enable its use in practice and decision-making.
Although this study presents different data sources, speeches from municipal managers, health teams, and developers of the technology used by AE/PMAQ-AB were unavailable and may be considered a limitation. Moreover, results cannot be generalized since this is a case study.

Conclusion
IT use was effective for dissemination of results. We observed a potential distribution of knowledge access due to extensive evaluation conducted in a short period among cycles and in a large territorial extension with regional disparities and high number of municipalities. However, managers lacked appropriation of knowledge produced by AE/PMAQ-AB due to possible overestimation of practical effects of technology.
Evaluators must provide a larger interval among evaluative cycles to sensitize PHC managers regarding obtained data and reduce gaps between data dissemination, use, and transformation into useful information to support discussions, re ections, and service improvements. Investments in technological infrastructure are essential, including expansion of digital inclusion. Political directions are needed to effectively deliver data to the population and improve PHC access and quality. We recommend focusing on di culties of using produced knowledge from evaluations to strengthen health policies.

Consent for publication
Not applicable.

Availability of data and materials
The datasets generated during and/or analyzed during the current study are not publicly available due to the privacy of personal information essential to link the databases but are available from the corresponding author on reasonable request.

Competing interests
All other authors declare they have no con ict of interest. . The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.