Data were analysed from multiple sources (Table 2). Detailed results against each of the data collection methods are presented in Additional File 3 and the findings at the level of the two country teams are presented as case studies in Additional File 4. The case studies highlight the successes, challenges, KT capacity and skill-building, KT and EIP value and culture, lessons learnt, most significant change, next steps and resulting publications for each of the countries (Additional File 4). In addition, findings from all sources were used to address each of the evaluation questions and sub-questions (Additional File 5).
Themes identified across the evaluation methods
Analysis of the data from all sources identified a number of key themes. The main issues within each theme are presented in Table 3. More detail for each theme is presented below, illustrated using quotes from the interviews and analyses of the data from each source (Additional File 3), where relevant.
Demonstrating value
Demonstrating the value of the EVIPNet initiative is essential for building commitment to the Network, crystallizing the vision, and securing further resources and sustainability.
Filling a gap
Stakeholder and country interviews suggested a lack of awareness of any similar initiative currently existing in Europe. Most respondents of the online country evaluation (18/23, 78%) believed that their country had benefited from being part of EVIPNet Europe to date (Additional File 3 – Fig. 2). However, there was a mixed response to increasing interactions between policy-makers, stakeholders and researchers (10/23, 43%), which requires more time for building trust, understanding and relationships.
Gaining commitment
Many stakeholders discussed the challenges of demonstrating the value of EVIPNet Europe. This was partially seen to be tied to perceptions that countries were already doing EIP, when in reality they were not, as evidence is used only in an ad-hoc manner, or there are unrealistic ideals around what EIP can achieve, which are then not realized. This can manifest as countries thinking that they do not need EVIPNet. Others felt that there was a lack of understanding from senior policy-makers about what evidence is, or that evidence is not used routinely even if the intentions are there. Furthermore, EIP was perceived as a complex and fairly abstract topic, for which success is not easily demonstrated and impact is not easily measured. This makes it challenging to demonstrate the return on investment and promote interest and long-term commitment.
“I think a good approach has been to start with, for example, the evidence briefs for policy and then you can get people interested, you can show that this is a product that we can produce, and this is how it can be used and this is how it can make a change. People might understand better what this is about and might be more willing to commit to it.”
[Secretariat stakeholder, S3]
Sharing success
It is important to demonstrate the value of the Network’s activities through stories of success from other countries as part of EVIPNet Europe or globally to increase a country’s understanding and commitment to EVIPNet, particularly when it joins EVIPNet. In addition to the EVIPNet Europe introductory document Conceptual background and case studies [19], collaboration between the WHO Secretariat and other EVIPNet Global member countries helps in sharing experiences and provides examples of good practice. Now that the Network is maturing, good examples and success stories of EVIPNet Europe are emerging (Additional File 4).
Sharing the EVIPNet Europe vision
The vision for EVIPNet Europe is frequently communicated through Secretariat-level products, including through its own strategy, evidenced in the documentary review. Lack of awareness of and engagement with EVIPNet Europe were mentioned as barriers at the country level by respondents to the online country evaluation, with suggestions that further promotion among the wider stakeholder community was required to support this national commitment. Using strategic levers and advocacy to gather commitment was felt to be an enabler for developing EVIPNet further, such as further discussion at high-level regional policies and meetings.
Secretariat stakeholders also expressed a need for greater awareness-raising at a high level, providing a clearer mandate for countries to join the Network and demonstrating support for EVIPNet Europe, as was done at the Sixty-sixth Regional Committee for Europe meeting during which the EIP Action Plan was adopted and at the Sixty-eighth Regional Committee for Europe meeting during the ministerial lunch on health information.
Network growth
Horizontal growth
The Network has grown from 13 countries initially to 21 within a few years, and more countries are seeking to join, especially western European countries (e.g. Austria). This demonstrates the need for EVIPNet Europe to increase country capacity in KT to support implementation of the EIP Action Plan and catalyse the achievement of important societal goals, such as the SDGs. This growth is also reflected in the social media analysis (Additional File 3, Part B).
Resources for growth
Growth of the Network presents both opportunities and challenges. Many said that the current resources were insufficient to fully meet member countries’ needs, with the Secretariat operating below the necessary staffing levels, often relying on a temporary outsourced workforce of consultants, interns, fellows and seconded persons. As this is likely to be exacerbated by further growth, there is a need for greater resources, both human and financial, to support an increase in Network members in order to maintain the current high standards and the existing close personal intra-Network relationships. Further, strategic decisions are needed on where the focus should be.
“I think that the Secretariat is unfortunately understaffed, underfinanced and completely overworked.” [Secretariat stakeholder, S3]
“…because now we are broadening we are taking on board more and more countries and actually trying to engage them so actually we are spreading, and there is a demand for more countries, but we also need a strategy to support the countries who have subscribed in the beginning to maintain and continue their collaboration. So it’s actually a dual process.”
[Secretariat stakeholder, S2]
Another suggested area for improving Network growth and expanding support as well as income was working with other stakeholders and other WHO programmes, as many felt that the use of evidence was core to the work and mandate of WHO, and important for all areas. Two promising approaches that were highlighted in the interviews are the country cohorts (Textbox 1) and biennial collaborative agreements (BCAs) (Textbox 2).
[Insert Textbox 1 and 2]
Variation in development
Another obvious challenge expressed was that this growth had led to an increased variability of Network member countries in terms of levels of awareness, skills and knowledge. Those who have been members since the beginning will clearly have different training and support needs to those that have just joined. Therefore, the training and multicountry meetings become more challenging to organize to meet the disparate demands of these members at different stages of development. More emphasis would be needed in the future on the use of the train-the-trainer model.
EVIPNet Global was identified as a possible source for the increased resources and global coordination required, including the unrealized potential for global capacity-building events.
“I think what really is required is for us to have interaction with headquarters also in the capacity-building events (…) that we have global events, global capacity-building events where these people can come together, that then depends on headquarters.” [Secretariat stakeholder, S8]
Sustainability of country work
There are certainly sustainability challenges, which are likely to be exacerbated by further growth, with many expressing that, at this point, activities would not be possible or would not continue without WHO funding. There are initial signs of Network member countries having successfully submitted funding proposals and/or seeking their own funding. Not all member countries seem, however, to have the skills to write business cases and applications for funding. Additional training by the WHO Secretariat could be useful. However, technical training was considered a more important priority, as is securing stable staffing at country level to enable sustainability.
Vertical growth (institutionalization)
Despite this growth, establishment of KTPs within EVIPNet Europe is still required. Although there are examples of positive developments, these have yet to materialize formally. This lack of institutionalization of KTPs was also expressed by many respondents during the country online evaluation. Many felt this was the obvious next step and area for future development, both strategically and in terms of support and capacity development for member countries. It was acknowledged that institutionalization was not an easy or quick process but necessary if this Network was to become sustainable and self-sufficient.
“So for me personally I think this is something that we have to tackle, because we are spending a lot of effort in the situation analysis…but for me personally not sufficient KTPs, concrete KTPs, have been established.” [Secretariat stakeholder, S2]
Developing a structure
EVIPNet Europe has established a structure, including a Secretariat with dedicated staff to coordinate and support the Network. There is also a regional and global steering group and governance mechanisms. However, many stakeholders were not aware of the accountability processes for EVIPNet Europe and these have not been emphasized in publicly available documents. There was widespread understanding from stakeholder interviews that M&E was an important but challenging aspect to providing accountability. However, the documentary review revealed that no country evaluations have been conducted to date. It is expected that the development of the new M&E framework will create a new momentum, including the development of an additional simple data collection/reporting methodology. Progress is captured to some extent in the published annual reports.
Network activities and processes
The EVIPNet approach
The logical step-wise approach to and methodology for EIP of the EVIPNet Europe process was appreciated. Interviewees and survey respondents highlighted the same standardized process to be followed, including how and where to look for evidence, considering EIP stakeholders, cost–effectiveness, harms and benefits, and producing policy options, having a PD, etc.
“…because you cannot disagree with the EVIPNet principles, it’s just so logical that things should be set up in this way.” [Country team stakeholder, B1]
Despite the ability to adapt to the country context, stakeholders felt that the sequence in which countries develop KT products could be even further adapted to country-specific needs, such as starting off with rapid response mechanisms to provide opportunities for quicker wins and to demonstrate the benefits of EVIPNet Europe. This would also initiate the creation of a culture for EIP before embarking on concrete KT activities such as an EBP and PD, if appropriate.
Challenges reported in the online country evaluation included a lack of available research evidence within countries, or lack of awareness of the sources of evidence, the attitudes of policy-makers and lack of a culture of evaluation. The online country evaluation also highlighted the lack of mandate for developing evidence briefs as a challenge; while some countries followed the process, others needed a ministerial order.
Development of support instruments by the Secretariat
The documentary analysis highlighted five support instruments and resources developed and published by the WHO Secretariat to date [19, 43-46]. Countries participating in the case studies reported using the support instruments and that they were relevant and useful, although many took longer than expected to finalize. However, the credibility of an official WHO publication was felt to be worth the trade-off in time and process, and important for maintaining the high technical standards and reputation to enable buy-in with senior decision-makers and other stakeholders. The PD process was particularly important in developing new relationships, exchanging opinions and expertise, and increasing understanding among experts and country stakeholders.
“Yes, they were the basics for the work that we have done, so yeah, I mean I could speak for quite some time about each of them, but the general position is yes, each one of them has been very helpful.” [Country team stakeholder, B4]
Using these EVIPNet support instruments, one situation analysis (SA) had been published [47] and seven were currently under development. In addition, two EBPs had been published when the documentary analysis was conducted, with a further 10 in draft form.
Some stakeholders thought that the development of further instruments might be overwhelming, while others provided examples of additional tools that could be developed, including more support for developing a rapid response service, instruments for appraisal of qualitative evidence, citizen engagement and organizational management (planned for the near future). One area mentioned by several stakeholders as a possible gap was more information or support for institutionalizing a KTP within countries, including writing business cases, and how to finance a KTP, ensuring available resources to bring in topic experts while retaining a core of EIP experts.
Technical capacity-building
Capacity-development workshops are an important part of the EVIPNet Europe approach. The documentary review revealed that a number of face-to-face and webinar sessions have been conducted by the WHO Secretariat, and have grown over time. This includes training as part of multicountry and national workshops. Despite no standardized evaluation question being used, average participant scores for these sessions are high. Interviews also highlighted the value of these capacity-development activities, in both the provision of technical skills and support for practical applications, not just knowledge, which were felt to be missing from other similar programmes. Mostly it was changes in the skills and knowledge of participants that were reported, although some shifts in attitudes were suggested, particularly from the PDs bringing different people together to understand each other.
Capacity-building support and the EVIPNet support instruments are clearly valued, with examples of how these had, or hopefully would, change policy. However, these currently benefit only those who attend sessions and/or develop country KT products. Some mentioned attempts at cascading this knowledge from those who had attended courses to stakeholders within countries, but there was still a need identified to widen the pool of EIP experts within countries.
Intra-Network communication
Face-to-face communication between countries and with the Secretariat occurred mostly during the development of products and other specific country activities or during multicountry meetings. Most stakeholders suggested that communication was largely one way from the WHO Secretariat, pushing information out and supporting countries. Others felt that some countries seemed somewhat uncomfortable in approaching the Secretariat for support, highlighting the need to build relationships and trust. Many stakeholders stressed the importance of face-to-face communication in establishing and building these relationships.
“…but I think that the multicountry meetings are indispensable in the sense that if you don’t meet the person and see them and talk to them, then, I mean it’s easier to approach someone if you have met them in the past, so in that sense the multicountry meetings are a key moment in that communication.”
[Country team Stakeholder, B4]
Documentary analysis suggests that attendance at multicountry meetings has remained similar over time in terms of the numbers of attendees. However, attendance by WHO country office staff has reduced from 92% of member countries represented in 2013 to 16% in 2017. Some members and stakeholders also raised language issues in interviews and as part of the online evaluation. Despite translated products being available and training sessions being held with Russian interpreters, it was felt that this was not the same as having someone who truly understands EIP and the EVIPNet methodology, and can explain and convey the details, nuances and passion accordingly, resulting in some elements being potentially lost in translation.
Social network communication across the EVIPNet Europe network, as indicated by the use of Yammer, has decreased over time and there has been variation in participation and type of posts (Additional File 3, Part B).
Sharing and networking
The documentary analysis highlighted 18 conference presentations related to EVIPNet to date, and seven peer-reviewed publications authored by both the WHO Secretariat and country team members. Initially, these were mostly led by the Secretariat, but increasingly, countries are leading this external promotion, e.g. there are now four peer-reviewed papers with country members as lead authors. Many of the papers have been published in WHO publications (Panorama and Eurohealth), with limited exposure to other sources, which may limit global reach, e.g. via journal database searches.
Sharing and networking were mentioned by several respondents of the online country evaluation, with lessons learnt and experiences exchanged with other countries cited as an important part of multicountry meetings, country cohorts, Yammer and direct country contact. The need for more opportunities for mentorship and participation in international events was suggested to enhance this further. Other examples included countries receiving mentoring from the other EVIPNet Europe member countries and EVIPNet Europe partners (e.g. EVIPNet Chile, Lebanon Knowledge to Policy [K2P] Center and McMaster Health Forum) and study visits to other countries as well as the WHO Regional Office for Europe to learn about EIP.
“[EVIPNet Europe] is also a good networking platform and it supports us when I see other WHO [country teams] are struggling with the same difficulties and see how they have solved them, etc.”
[EVIPNet Europe national champion]
Greater involvement of EVIPNet Global and the role of headquarters in coordinating and ensuring knowledge exchange and sharing globally was requested.
People
Relationships
Many of the interviewees felt that individuals, personalities and relationships were important. Individual members have developed strong relationships, with trust and rapport built and fostered over time by the dedicated Secretariat Coordinator, who is key to this. The role of individual personalities and networking skills was seen as vital to the success of the Network.
This related not only to the relationship with the Secretariat, with almost all representatives of Network member countries expressing gratitude for the support received, particularly from the Coordinator, but also the internal relationships with the WHO country office and Ministry of Health (MoH). Getting the right country team together was highlighted, with a need for team members to be highly motivated individuals. Within countries, developing a relationship between the national champion and WHO country office staff was an important part of the EVIPNet process, facilitated by WHO country offices. The latter are also crucial in coordinating and planning the future of EIP with the national champions.
In the country online evaluation, some countries reported coordination between agencies to be a barrier, e.g. interaction between policy-makers and the research community was lacking, coordination between ministries was slow, the MoH lacked a strong voice and had no interaction with the WHO country office.
Staff turnover
Different models were adopted for national champions, with some countries maintaining continuity where possible and others frequently nominating new representatives. While stakeholders recognized the pros and cons of these approaches (breadth versus depth), continuous retraining of champions and developing relationships, and lack of progression of knowledge and understanding caused by turnover of national champions were felt to be challenging. Furthermore, many stakeholders suggested that there was a need for countries to gain experience, not just in the technical skills-building capacity but also in applying this in practice, facilitated by hands-on technical support and coaching to develop understanding, as has been offered by the Secretariat. This is also made more challenging if staff turnover is high or untimely. Many references were also made to the challenges caused by turnover of staff in all areas; national champions, WHO country offices, MoHs and the Secretariat. While there were examples of this turnover bringing about positive changes (e.g. shift in priorities to align with EVIPNet), this was mostly seen to be negatively impacting on Network development.
“ And in some countries they do nominate the same people for every year’s workshop so that they can, you know, increase skills; in other countries it’s more common that every year they want to give a different person the opportunity to participate so then it’s a bit difficult in terms of continuity, at the same time you’re exposing more people to the content.” [Secretariat stakeholder, S3]
Staff turnover was also mentioned as causing delays to the creation of country KT products and institutionalization of the process, with decision-maker/government changes being the main ones cited. Turnover at the political level also had implications for how members advocate for EVIPNet, given the need for political buy-in. However, with short terms of office, ministers want to see impact quickly, which is challenging, while changing policy and seeing a measurable impact takes longer, and M&E is also not yet well developed in member countries.
Creating partnerships
During the growth of the Network, the Secretariat sought and developed a number of collaborations, highlighted in the documentary review. These include external collaborations with HINARI, SORT IT (the Structured Operational Research and Training IniTiative), EUPHA (European Public Health Association), and the Cochrane Collaboration, K2P Center, McMaster Health Forum and Wellcome Trust. Internally, collaboration with the Division of Health Emergencies and Communicable Diseases has been established in the WHO Regional Office for Europe, which is expanding.
Formal partnerships at the country level in support of the work of national country teams at the time of the evaluation was more limited. There are, however, indications of country teams undertaking intercountry collaborative activities, e.g. study visits, informal mentoring from EVIPNet Europe and EVIPNet Global member countries. Cohorts of countries working on the antimicrobial resistance (AMR) EBPs are a new collaborative step demonstrating added value, as these have created communities of practice and allow sharing of experiences (Box 1).
Human and financial resources
Human capacity and financial resources were raised as an issue for country teams as well as the Secretariat. When asked about the human, time, skills and financial resources invested, some were not aware of these details. Of those who were, most suggested that resources were limited, and work was done by a small number of staff or consultants hired for a specific activity. Almost all finances to support EVIPNet Europe were from WHO. Country teams are often reliant on limited staff, without dedicated funding or terms of reference.
The WHO Secretariat has been creative in identifying alternative sources for human resources and expanded in-kind support, e.g. via half-year short-term fellowships and secondments. The document review highlighted the variation and turnover in WHO Secretariat staff and the variation in financial resources for the Secretariat over time. However, posts are included in the WHO human resource plans, highlighting commitment. Further, the operational plans suggest that the WHO Secretariat has continued to achieve its own EVIPNet Europe objectives, despite these limitations. The Secretariat has also been successful in utilizing organizational resources, including core funding for intercountry work, and BCA funds (unique to the European Region) for country-level work (Box 2).