Implementing Implementation Science; A Bibliometric Study of Qualitative Research in Clinical Articial Intelligence

Implementation science is a pragmatic and multidisciplinary eld, centred on the application of a wide range of theoretical approaches to close ‘know-do’ gaps in healthcare. The implementation science community is made of individuals on a continuum between academia and practice, but it is unclear to what extent the theoretical deliberations of implementation academics are translated into the work of implementation practitioners and on to patient benet. This bibliometric study aims to use the eld of clinical articial intelligence(AI) implementation to sample the prevalence and character of theoretically informed implementation practices.


Abstract
Background Implementation science is a pragmatic and multidisciplinary eld, centred on the application of a wide range of theoretical approaches to close 'know-do' gaps in healthcare. The implementation science community is made of individuals on a continuum between academia and practice, but it is unclear to what extent the theoretical deliberations of implementation academics are translated into the work of implementation practitioners and on to patient bene t. This bibliometric study aims to use the eld of clinical arti cial intelligence(AI) implementation to sample the prevalence and character of theoretically informed implementation practices.

Methods
Qualitative research of key stakeholder perspectives on clinical AI published between 2014-2021 was systematically identi ed. Following title, abstract and full-text screening eligible articles were characterised in terms of their publication, AI tool and context studied, the theoretical approach if any and the research methods and quality. Descriptive, comparative and regression statistics were applied.

Results
One-hundred-and-eleven studies met the eligibility criteria, with monthly eligible publication rate increasing from 0.7-4.0 between 2014-2021. Eligible studies represented 23 different nations and 25 different clinical specialities. A theoretical approach was explicitly employed in 39 (35.1%) studies though 6 of these described novel theoretical approaches(15.1%) and the most frequently used theoretical approach was only used 3 times. There was no statistically signi cant trend in the prevalence of theoretically informed research within the study period. Of the 25 theoretically informed studies conducted in Europe or North America 19(76%) used theories that originated in the same continent.

Conclusions
The theoretical approaches which characterise implementation science are not being put to use as often as they could, and the means by which they are selected also seems suboptimal. The eld may facilitate a greater synergy between theory and practice if the focus shifts from unifying implementation theories on to unifying and expanding the implementation community. By making more theoretical approaches accessible to practitioners and supporting their selection and application, theory could be more effectively harnessed to close healthcare's 'know-do gaps'.

Introduction
Implementation science is a relatively young eld drawing on diverse epistemological approaches.(1) Its pragmatic goal of bridging 'know-do gaps' to improve real-world healthcare necessitates this multidisciplinary approach. However, the challenge of aligning qualitative and quantitative methodologies has been a persistent source of agitation within the eld.(2) Implementation science centres around the application of theoretical approaches to inform or evaluate implementation in a particular healthcare context. Consequently, de ning and standardising the terminology and structure of these theoretical approaches has received considerable attention. (3)(4)(5)(6)(7)(8) Evidence synthesis,(4) consensus methods(9) and passive taxonomies (10) have been used on many occasions, in attempts to unify the eld with no evidence of growing or imminent agreement. This dogmatic discourse serves the interests of academics, but it risks alienating practitioners on whom the real-world impact of the eld depends. (11) To objectively gauge the disconnect between the two ends of implementation science's academia-practice continuum, the present study seeks to examine the prevalence of theoretically informed work within a representative eld. De ning exact boundaries to this multidisciplinary eld holds challenge and controversy, which can be avoided by sampling research which indisputably focuses on the process of implementation. This is why qualitative methods were made an inclusion criterion for the present study. They hold the necessary sensitivity to investigate the various contextual factors shaping implementation, but also the complexity inherent to implementation. The second motive for a focus on qualitative methods are their foundational relationship with theoretical approaches. An additional analytical layer of theory may be undesirable in quantitative research. In qualitative research however, as set out by Collins and Stockton, theory provides focus and organisation to the study, exposes and obstructs meaning, connects the study to existing scholarship and terms and identi es strengths and weaknesses. (12) The relevance of the present study to contemporary implementation science also depends on sampling a prominent and active eld of health research. Clinical arti cial intelligence (AI) forms such a sample.
Computer based AI was conceived more than 50 years ago and has been incorporated into clinical practice through computerised decision support tools for a few decades. (13,14) However, advancing computational capacity and the feasibility and potential of deep learning methods have galvanised public and professional enthusiasm for all applications of AI, including healthcare. (15) The acknowledgment of this potential is formalised in the embedment of clinical AI into national healthcare strategic plans and by the recent surge of regulatory approvals issued for 'software as a medical device'.
(16-19) Scienti c publication on clinical AI tools has dramatically increased in recent years, though the proportion of these using qualitative research methods is yet to be described. (20) Despite this, clinical AI remains an area with little demonstrable bene t to real-world patient care and so the well-cited 'AI chasm' provides an exemplary 'implementation gap'. (21,22) This bibliometric study aims to describe the prevalence of theoretical approaches in contemporary health systems research, by sampling the eld of clinical AI implementation. Each eligible study and the theoretical approach it may employ will be characterised to provide insights into the frequency and nature of qualitative methods used in clinical AI research.

Methods
This bibliometric study is part of a qualitative evidence synthesis which followed the PRISMA 2020 reporting guidelines (Supplementary materials 1) and was executed without amendments as published in an a priori protocol. (23,24) A search string combining concepts related to AI, healthcare and qualitative methods was designed and tuned for sensitivity and speci city in MEDLINE(Ovid) ( Figure 1). (25) This search string was translated to Scopus, CINAHL(EBSCO), ACM Digital Library and Science Citation Index (Web of Science) to cover computer science, allied health, medical and grey literature (Supplementary materials 2). The search was executed in April 2021 and reviewed literature from 2014, which marked the rst market authorisations for software as a medical device granted in Europe and the USA.(16) Only English language indexing was required, there were no exclusion criteria relating to full-text language. The initial results were de-duplicated using Endnote x9.3.3 (Clarivate Analytics, PA, USA) and two independent reviewers (JH, MA) performed full title and abstract screening against the criteria ( Figure   1) using Rayyan.(26) The process was overseen by an information specialist (FB) and screening disagreements were arbitrated by a separate topic expert (GM). Eligible review and protocol manuscripts were included for reference hand searching only. Full-text review was performed independently in duplicate by two independent reviewers (JH, MA), with the same topic expert (GM) arbitrating. Corresponding authors were contacted up to three times when additional eligible data or reports were suspected ( Figure 2).
Two reviewers (JH, MA) jointly extracted characteristics from 10% of articles together before completing extraction independently. These characteristics included a quality score using the Joanna Briggs' Institute (JBI) 10-point Checklist for Qualitative Research. (27) Data concerning the year and type of publication, source title and eld, source impact factor, implementation context, theoretical approach use, study methods and study participants were also collected. This included the nature of the clinical AI tool studied, categorised as rule-based or not depending on whether it executed a pre-existent human dictated decision tree or guideline, or drew on a non-human means of decision making, e.g. regression modelling or machine learning. For each theoretical approach identi ed the index article was sourced and underwent full-text review by a single reviewer (JH). Nilsen's taxonomy of theoretical approaches was used to facilitate the organisation of the wide range of approaches employed to study the implementation of clinical AI tools and applications ( Figure 3).(10)

Results
Of the 111 eligible reports, 104(93.7%) were full reports in peer-reviewed journals, with three(2.7%) short reports, three(2.7%) theses and one(0.9%) conference abstract. Citations of all eligible papers are available in Supplementary Materials 3. Only two articles were not in English language (Italian and German). Among journal publications, the median impact factor was 3.1 (Interquartile range 2.1,4.5) and 56(58.3%), 6(6.3%) and 34(35.4%) were medical, nursing or midwifery and informatics journals respectively. The median quality score using the 10-point JBI Checklist for Qualitative Research was 8 (IQR 7,8). There was a signi cant increase in the rate of all formats of eligible publication over time
In total 39(35.1%) studies stated some form of application of a theoretical approach. In six(15.4%) cases this was a novel implementation theory and the most frequently used theoretical approaches had just three applications (Table 1)

Discussion
Considering the use of theoretical approaches within eligible studies, it is striking that only one third explicitly used a theoretical approach. In contrast to a prior scoping review in the context of guideline implementation, these data did not suggest this low rate of theory application is increasing. (31) The heterogeneity was also notable, with no single theoretical approach being used more than three times and with highly varied modes of use (Table 1). This is not surprising given the plethora of theoretical approaches already documented within the eld of implementation science and our own observation that 15% of the studies applied a novel theoretical approach.(32) However, it does raise the issue of how authors come to select a theoretical approach and if this process can be optimised. Given the apparent lack of association between the use of theoretical approaches or research quality with the impact factor of the publishing journal, there are no clear external incentives for researchers to rigorously construct their approach. If the judicious selection and application of implementation theory is not acknowledged by this crude but widely used marker of esteem, then other motives may in uence research practice. Only 14(43.8%) of the studies applying a pre-existent theoretical approach provided any kind of rational for their selection and these explanations appeared to be rather vague, e.g. simply stating the method was commonly used. Taken alongside the observed tendency for authors to apply theoretical approaches that originated within their country or continent, it seems that choices made by researchers are based on convenience and familiarity to some extent at least. This status quo seems unlikely to fully harness the full potential of theoretical approaches. A more methodical pairing of each implementation case's needs and each theoretical approach's value proposition could substantially improve the quality of implementation practices. (33) These observations are in support of calls to move the focus within implementation science away from the theoretical approaches themselves and on to increasing the accessibility and e cacy of their use. (34) The alternatives of reverential commitment to a handful of dominant theories, or continuing attempts to create a unifying theory of implementation appear untenable in the face of an ever expanding catalogue of theoretical approaches. Embracing the full extent and value of available theory will require a kind of higher order consolidation however, as implementation science cannot expect to achieve its goals if its advocates must absorb all published approaches. Such a requirement would limit meaningful movement of individuals and expertise between the worlds of research and practice. The use of theoretical approaches must be supported with an interface that is accessible to individuals at any point on the continuum between implementation academic and practitioner. (11,33) To be sustainable this interface will require a persistent incentive for an independent entity to maintain and update a library of theoretical approaches, searchable by various attributes that describe their suitability to different implementation needs. Such tools, agnostic of any particular theoretical approach, have been very successfully applied in other qualitative methodologies and are becoming established within implementation science. (33,35) This innovative implementation platform has the potential to minimise the theoretical knowledge requirements demanded of users and welcome an even broader audience to the implementation science community.
The present data have also illustrated the paucity of qualitative enquiry within the eld of clinical AI implementation. A recent bibliometric study, similarly aiming to categorise published work on clinical decision support, but without any methodological constraints, identi ed 88-fold as many publications over the same time period.(36) Such imbalances limit understanding of the context into which tools are to be placed. This under-appreciation and under-investigation of context is a major contributor to the so- While the present study addresses the paucity of qualitative assessments of real-world theoretical approaches in implementation science, it has some clear limitations. Implementation science is a multidisciplinary eld with poorly described boundaries.(2) A core assumption of the present study is that research into stakeholder perspectives of clinical AI tools using at least some qualitative methods would be broadly acknowledged within the eld. This study's value also depends on the assumption that approaches within the present sample are generalisable to implementation science practice to some extent. Analysing the motives for theory selection also proved challenging as authors rarely justify their choice, and when they do it is often easy to align the same arguments with a different theoretical approach. Further exploration of authors' individual collaborations, epistemological commitments or prospective inquiry may yield a more informative insight into these motives.

Conclusions
A real-world sample of implementation science research in the contemporary and prominent eld of clinical AI nds that a minority of research is theoretically informed. The type and mode of theoretical application is highly heterogenous and is at least in part led by convenience rather than the alignment of theory and implementation need. To engage a greater range of stakeholders and harness the full range and value of theoretical approaches, we recommend a shift in focus from unifying implementation theories to expanding and unifying the implementation community. To achieve this, we propose increasing the prominence of a theoretical approach agnostic interface as a gateway between implementation science research and practice.

Competing interests
Two of the authors (JH and PK) collaborate with DeepMind, a company involved in the development and application of clinical AI tools.

Funding
This study is funded by the National Institute for Health Research (NIHR) through the academic foundation programme for the second author (MA) and through a doctoral fellowship (NIHR301467) for the rst author (JH). The funder had no role in the design or delivery of this study Authors' contributions JH contributed to the conception and design of the work, the acquisition, analysis and interpretation of the data and drafted the manuscript. MA contributed to the acquisition and analysis of the data. PK contributed to the design and conception of the work. FB contributed to the design of the work, the acquisition, analysis and interpretation of data and revised the manuscript. GM contributed to the conception and design of the work, the data acquisition, analysis and interpretation of data and revised the manuscript. All authors approved the submitted version and all authors agree to be personally accountable for their own contributions and to ensure that questions related to the accuracy or integrity of any part of the work are appropriately investigated, resolved and the resolution documented in the literature.

Figure 1
Eligibility criteria informing the search string design and screening process. Histogram of the countries in which eligible reports drew their participants. Where recruitment took place across more than one nation, the category of 'International' was assigned. AI = Arti cial Intelligence, UK = United Kingdom, USA =United States of America