Of the 111 eligible reports, 104(93.7%) were full reports in peer-reviewed journals, with three(2.7%) short reports, three(2.7%) theses and one(0.9%) conference abstract. Citations of all eligible papers are available in Supplementary Materials 3. Only two articles were not in English language (Italian and German). Among journal publications, the median impact factor was 3.1 (Interquartile range 2.1,4.5) and 56(58.3%), 6(6.3%) and 34(35.4%) were medical, nursing or midwifery and informatics journals respectively. The median quality score using the 10-point JBI Checklist for Qualitative Research was 8 (IQR 7,8). There was a significant increase in the rate of all formats of eligible publication over time (Figure 4)(Kendell’s tau r=0.691, p=0.018). Access to potentially eligible reports from two published protocols was not possible despite attempted correspondence.(28, 29)
Research context
Although there was some representation from developing nations, 88.1% of the 101 reports focusing on a single nation were in countries meeting the United Nations Development Programme’s definition of ‘high human development’(Figure 5).(30) The median human development index of the host nations for these 101 reports was 0.929(IQR 0.926,0.944).
In terms of the clinical AI application studied, 31(27.9%), 24(21.6%) and 56(50.5%) studies considered hypothetical, simulated and clinical applications respectively. The nature of the AI tool under investigation was rule based in 66(59.5%) of reports, non-rule based in 41(36.9%) and not specified in 4(3.6%). The tools studied were aimed directly at the public in 5(4.5%) studies, primary care in 45(40.5%), secondary care in 43(38.7%), mixed settings in 3(2.7%) and unspecified in 15(13.5%). Application was studied across a broad range of clinical specialties, though primary care dominated (28.8%, Figure 6). The tools used scalar and categorical clinical data in 83(74.8%) of studies, imaging data in 9(8.1%) and mixed inputs in 1(0.9%) to perform triage, diagnostic, prognostic, management or unspecified tasks in 17(15.3%), 15(13.5%), 8(7.2%), 47(42.3%) and 23(20.7%) studies respectively.
Research method characteristics
Clinicians, patients, managers, industry representatives, academics and carers were participants in 95(85.6%), 27(24.3%), 25(22.5%), 15(13.5%), 9(8.1%) and 6(5.4%) studies respectively. Interviews, focus groups, surveys, think aloud exercises, observation and mixed data collection methods were used in 54(48.6%), 19(17.1%), 12(10.8%), 1(0.9%), 1(0.9%) and 24(21.6%) studies respectively. Thematic analysis, framework analysis, content analysis, constant comparative analysis, descriptive analysis, grounded theory approaches, other specified and unspecified data analysis methods were used in 39(35.1%), 11(9.9%), 13(11.7%), 4(3.6%), 6(5.4%), 8(7.2%), 3(2.7%) and 27(24.3%) studies respectively.
In total 39(35.1%) studies stated some form of application of a theoretical approach. In six(15.4%) cases this was a novel implementation theory and the most frequently used theoretical approaches had just three applications (Table 1). Only 1 study explicitly drew on two separate theoretical approaches. There was no statistically significant change in the frequency of theoretical approach use over time comparing the 43 studies published before the median year of publication, 2019, (39.5%) with the 68 published thereafter (30.9%)(χ2, p=0.349). Of the 15 European studies that used pre-existent theoretical approaches, 10(66.7%) used theoretical approaches originating in Europe with the remainder originating in the USA. Of the 10 studies from North America, nine(90.0%) used theoretical approaches originating in the USA, with a single Canadian study using Methontology from Spain. Performing univariate linear regression with source impact factor as the dependent variable, neither the JBI qualitative research checklist score (unstandardised B = 0.12, 95% confidence interval -0.09, 0.32) or use of theoretical approach (unstandardised B = 0.55, 95% CI -0.23, 0.94) had significant predictive value.
Table 1
Characteristics of all 23 pre-existent theoretical approaches used by eligible clinic artificial intelligence implementation studies. USA = United States of America, UK = United Kingdom
Theoretical approach | Year | Author’s nationality | Nilsen's taxonomy(10) | Frequency |
Awareness-to-Adherence Model(37) | 1996 | USA | process model | 1 |
Behaviour Change Theory(38) | 1977 | USA | classic theory | 1 |
Behaviour Change Wheel(39) | 2011 | UK | implementation theory | 1 |
Biography of Artefact(40) | 2010 | UK | classic theory | 1 |
Consolidated Framework for Implementation Research(4) | 2009 | USA | determinant framework | 3 |
Clinical Performance Feedback Intervention Theory(6) | 2019 | UK | implementation theory | 1 |
Disruptive Innovation Theory(41) | 1995 | USA | classic theory | 1 |
Fit Between Individuals Task and Technology(42) | 2006 | Germany | evaluation framework | 1 |
Flottorp Framework(43) | 2013 | Norway | determinant framework | 1 |
Heuristic Evaluation(44) | 1990 | Denmark | determinant framework | 1 |
Methontology(45) | 1997 | Spain | process model | 1 |
Normalisation Process Model(46) | 2007 | UK | process model | 1 |
Normalisation Process Theory(47) | 2009 | UK | implementation theory | 2 |
Process Evaluation Framework(48) | 2013 | UK | evaluation framework | 1 |
Programme Sustainability Assessment Tool(49) | 2014 | USA | determinant framework | 1 |
Rapid Assessment Process(50) | 2001 | USA | process model | 3 |
Rogers' Theory of Diffusion(51) | 1962 | USA | classic theory | 1 |
Siitig and Singh Framework(52) | 2010 | USA | process model | 2 |
Strong Structuration Theory(53) | 2007 | UK | process model | 2 |
Technology Acceptance Model(54) | 1989 | USA | determinant framework | 3 |
Theoretical Domains Framework(7) | 2005 | UK | determinant framework | 1 |
Theoretical Framing Theory(55) | 1999 | USA | classic theory | 1 |
Unified Theory of Acceptance and Use of Technology(56) | 2003 | USA | determinant framework | 2 |