Artificial Intelligence in Medicine: A Systematic Review of Guidelines on Reporting and Interpreting Studies

doi:10.21203/rs.3.rs-3430896/v1

Background

Reporting guidelines developed for medical artificial intelligence (AI) studies are structured tools that address general and/or AI-specific methodological and reporting issues. We aimed to systematically review published medical AI reporting guidelines and checklists and evaluate aspects that can support the choice of the tool in a particular research context.

Methods

We searched PubMed, Scopus, and Web of Science until February 2023. Citations and Google were searched in addition. We included peer reviewed articles of reporting guidelines or checklists applicable for medical AI research. Screening, article selection and data extraction was performed in duplicate. We extracted publication details, the guidelines’ aims, target audiences, development process, focus area, structure, number of items and recorded the number of Google Scholar citations as a proxy to usage.

Results

From 821 records, and additional sources, 24 guidelines were included (4 narrative guidelines, 7 general reporting checklists, 4 study design specific checklists, 9 clinical area specific checklists). 13 studies reported the guideline development methods, 10 guidelines were registered in the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network. In 224 sections, the guidelines contained 704 items in total. The number of items per checklist varied between 10 and 66. Nine guidelines broadly followed the Introduction, Methods, Results, and Discussion (IMRAD) structure, 12 the machine learning pipeline method (i.e., sequential steps from data processing to model training and evaluation) and 3 had other structure.

Conclusions

Currently there is no consensus about the structure and format about AI reporting guidelines. The guidelines’ structure and level of detail varied significantly which makes difficult for researchers to follow how detailed and standardized a medical AI study report should be. The robustness of development process and support from the literature suggests CONSORT-AI as the most established tool. Such AI extensions of clinical study guidelines may not cover all the application fields of AI in medicine. In certain research contexts, an established checklist for main study types, and a general AI-based checklist may be used in parallel to provide most useful guidance in designing, writing and interpreting medical AI studies.

artificial intelligence

machine learning

medical studies

guideline

usability

explainability

Artificial intelligence (AI) and machine learning (ML), unless specified otherwise, hereunder collectively referred to as AI have been researched in academia since the mid-20th century. Recent advances in computing power, data storage, research methodology, and skilled human resources have accelerated their application of AI in medicine, with the potential to transform healthcare and the life sciences industries (1, 2).

The use of AI is rapidly developing in many clinical areas including cardiology, neurology, oncology, haematology, nephrology, gastroenterology, orthopaedics, and rheumatology, clinical approaches such as precision medicine, genomics, and telemedicine, and care components such as triage, diagnosis, prognosis, monitoring, and treatment (3, 4). The growing importance of AI in medicine is well illustrated by the fact that the number of approved AI-based devices by the US Food and Drug Administration (FDA) increased from 9 in 2015 to 77 in 2019 with 24 new devices approved in the first quarter of 2020. Furthermore, a total of 240 AI-based medical devices were approved in Europe between 2015 and 2020 (5).

With the rapid advancement of AI-based medical technologies, there has been an increasing need for evidence on their risks and benefits, based on high-quality clinical trials. Methods for observational and interventional studies (clinical trials) in clinical epidemiology have been established and used in evidence-based medicine in the past decades (6–8). In parallel, significant improvements have been made to standardize how clinical studies are designed and reported (9). The impact of reporting quality should not be underestimated. First, good quality data can only be reported from well-designed, conducted, and well-documented studies. Hence, reporting requirements may positively influence trial design (9). Second, poor reporting of a high-quality clinical trial may hamper its clinical usability, and vice versa, inadequate, or incomplete reporting of a poor-quality clinical trial may obscure its weaknesses and can lead to incorrect medical decisions. Third, systematic literature reviews and meta-analyses, which represent the highest level of medical evidence, require well-structured, and transparently reported data (10).

The need for standardization has driven the development of reporting guidelines and checklists. While we acknowledge the difference, we will refer to reporting guidelines and checklists interchangeably in this paper. (Guidelines without checklists will be denoted as narrative guidelines.) The compliance with relevant reporting guidelines is a criterion for publication in many medical journals, which facilitates their uptake among researchers (9). For instance, the Consolidated Standards of Reporting Trials (CONSORT) checklist for randomized controlled trials has been simultaneously published in 9 medical journals and translated to 12 languages (11). It includes 25 items, indicating the information that should be provided about each section of the study, with 27 extension versions for specific clinical or methodological applications. Reporting guidelines in the CONSORT family follow a standardized structure and undergo rigorous development process (12), recommended by the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network, ‘an international initiative that seeks to improve the reliability and value of published health research literature by promoting transparent and accurate reporting and wider use of robust reporting guidelines’ (13)

The development and therapeutic applications of AI algorithms presented new complexities and potential sources of bias that had not been addressed in former reporting guidelines (14). Several studies reported alarming results about the methodological quality and reporting of medical AI studies. Despite the comparable diagnostic accuracy of deep learning algorithms and healthcare professionals in the interpretation of medical imaging, the absence of external validation and inadequate reporting standards undermine the credibility of these results (15). Most medical imaging studies reported between 2010–2019 that compared the diagnostic accuracy of deep-learning algorithms with human experts had high risk of bias, deviated from reporting standards, and lacked data and code sharing (16). External validation was performed in only 6% of 516 radiological-AI studies reported in 2018, and none of the studies featured all important design elements of clinical AI validation (17). Likewise, none of the papers from 2015–2018 on the use of diagnostic ML models used a reporting guideline, and most of them lacked adequate details on participants (18).

As a response, research teams, international organizations, and publishers have established reporting guidelines for medical AI studies to ensure that results are replicable, transparent, and provide sufficient information for inclusion in future evidence syntheses (14, 19, 20). AI reporting guidelines are diverse in terms of their target audiences, aims, scope, structure, and the rigorousness of their development process. Therefore, with all the good intentions, AI reporting guidelines present a new challenge to authors and physicians alike. Which one to follow? What represents best practice in reporting? Have I missed a new reporting guideline that fits better my research context?

Given the rapid development in the field (21–25), the present research aims to provide a comprehensive systematic literature review on medical AI reporting guidelines. Specifically, we aim to analyse the goals, target audiences, development process, focus area, structure, and usage of guidelines. With our review we aim to aid researchers in the choice of the reporting guideline that represents best practice in their research context in a particular clinical area or research setting.

Search strategy

We searched PubMed, Scopus, and the Web of Science databases for potentially relevant publications from the inception to September 28, 2022. The search was updated on February 12, 2023. The detailed search syntaxes are provided in Supplementary File 1.

The reference list of the included studies was also reviewed to find any other eligible documents. Furthermore, in February 2023, a literature search was conducted using the Google search engine to identify additional pertinent studies using the following keywords: checklist, guideline, reporting, standard, recommendations, artificial intelligence, machine learning, deep learning, medical, medicine, health, clinician, doctor, and healthcare.

Selection criteria

Articles were included if they presented an original reporting guideline applicable for medical AI research. Both narrative guidelines, and detailed checklists were considered. Peer reviewed publications in any language were considered as eligible without restriction on publication date or publication type. Studies were excluded if they published a guideline development research protocol or the use of reporting guidelines for the assessment of medical AI research.

Study selection

Records were deduplicated and imported into an Excel spreadsheet. Two reviewers independently screened the titles and abstracts of the publications against the inclusion criteria. Potentially eligible records were subjected to full-text screening by two independent reviewers. In both stages, disagreements between reviewers were resolved by consensus.

Data extraction and synthesis

To characterise guidelines, we extracted their name, the title and year of publication, the journal, the first author’s name and country of affiliation. We also recorded the clinical area or study design in focus. The target audience of the guidelines was divided into three distinct groups: A) application developers, B) clinicians and model users, and C) authors, reviewers, and editors. Furthermore, we recorded if details of the development process were reported and whether the guideline was registered on the EQUATOR Network website. If reported, we extracted seven key components of guideline development from EQUATOR framework. These involved a literature review, a Delphi consensus survey, expert consensus meeting, pilot testing, obtaining funding, a policy for periodic updates, and endorsement by a journal or professional society (12, 13, 26). Furthermore, as a proxy indicator of the use of the guidelines in academic research, the number of citations for each was retrieved from the Google Scholar database on July 18, 2023.

When analysing the structure of guidelines, we extracted all elements of their structure that is one level above the reporting items. We compared the overall structure and level of detail, while a detailed comparative analysis of their content at the reporting item level was beyond the scope of this research. We counted the total number of reporting items presented in the guidelines. We considered the smallest units of text in the guideline structure (e.g., phrase, sentence, or paragraph) as reporting items. Although some guidelines described multiple reporting elements within an item, we considered them as part of the respective item without further breakdown.

We synthesized results using simple descriptive statistics. The associations between variables were explored via appropriate bivariate tests after checking the distributional assumptions. For reporting, we followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (27).

Search results

The search in PubMed, Scopus, and Web of Science yielded a total of 1397 hits. After the elimination of 567 duplicates, 821 records were screened by title and abstract. We checked 193 full-text publications against the predefined eligibility criteria, resulting in the inclusion of 20 studies. In addition, four studies were identified through the search of the reference lists of included articles and the complementary Google search. In total, 24 studies were included for further investigation. Details of the article screening and selection are shown in Fig. 1. The list of excluded studies is provided in Supplementary file 2.

Main characteristics of the publications

Table 1 summarizes the 24 articles involved for further analysis. The first guideline, CHARMS (28), focusing on the appraisal of systematic reviews of predictive modelling studies was published in 2014, followed by a guideline on the reporting of ML predictive models by Luo et al (29) from 2016. Most guidelines were subsequently published in 2020 (n = 9, 38%) and 2021 (n = 9, 38%), followed by 2022 (n = 3, 13%) and one guideline in 2023.

The first authors’ affiliation was mainly from the United States (n = 9, 38%), followed by the United Kingdom (n = 4, 17%), Australia (n = 2, 8%) and Canada (n = 2, 8%). Other guidelines were published by authors with affiliation from France, Italy, the Netherlands, Sweden, Spain, Germany, and Switzerland. The 24 guidelines were published in 22 journals, with Nature Medicine (n = 3, 13%) being the most common.

The CHARMS guideline (28) received the highest number of citations on Google Scholar (N = 1080), followed by Luo et al (29) which was cited 571 times. Four guidelines, CONSORT-AI (30), SPIRIT-AI (31), CLAIM (32), and MI-CLAIM (33), attracted more citations than the average (N = 178). They were cited 523, 444, 442, and 212 times, respectively. All highly cited guidelines included a point-by-point checklist.

Table 1

Summary of the 24 included reporting guidelines for medical AI studies
First author (Year) / Country of affiliation	Name of the guideline / N of reporting items	Focus of the guideline	Journal	Purpose of the guideline	Google Scholar citations^a
Narrative guidelines
Buvat (2021) / France (34)	T.R.U.E. / 4	Nuclear medicine	The Journal of Nuclear Medicine	To aid the identification of studies reporting ground-breaking developments in AI-based research in nuclear medicine.	15
Stevens (2020) / US (35)	NS^b / 19	ML in clinical research	Circulation: Cardiovascular Quality and Outcomes	A guideline for transparent and systematic presentation of outcomes from ML analyses, addressed primarily for clinical researchers. Designed to supplement current clinical reporting requirements.	90
Faes (2020) / UK (36)	NS / 8	ML clinical studies	Translational Vision Science & Technology	Improve the quality of research on the therapeutic use of ML by equipping clinicians and researchers with the tools they need to conduct their own rigorous assessments.	111
Bates (2020) / US (37)	NS / 8	Clinical research of AI-based interventions	Annals of Internal Medicine	Suggestions for reporting standards to enable the assessment of the incremental benefits of ML and AI, and remove barriers from their clinical adoption.	53
General reporting checklists
Al-Zaiti (2022) / US (21)	ROBUST-ML / 30	ML in clinical studies	European Heart Journal - Digital Health	Increase physicians' understanding of ML by equipping them with the information and tools required to comprehend and evaluate clinical research focusing on ML.	10
Cabitza (2021) / Italy (19)	NS / 55	ML in clinical studies	International Journal of Medical Informatics	To analyse the scientific rigor of a medical ML contribution and the reliability of its findings qualitatively.	93
Olczak (2021) / Sweden (38)	CAIR / 40	Clinical AI research	Acta Orthopaedica	Clinical reporting guidelines for artificial intelligence and ML; guidance on selecting appropriate outcome indicators.	33
Scott (2021) / Australia (39)	NS / 12	ML algorithm in healthcare	BMJ Health Care Informatics	To evaluate the clinical usefulness of ML technologies in healthcare.	58
Hernandez-Boussard (2020) / US (40)	MINIMAR / 21	AI in healthcare	Journal of the American Medical Informatics Association	To facilitate the diffusion of algorithms across healthcare systems, enable transparency to address any biases and unintended effects, and encourage the use of secondary resources through promoting external validation and encouraging the use of secondary resources.	145
Norgeot (2020) / US (33)	MI-CLAIM / 19	Clinical AI modelling	Nature Medicine	To propose a baseline for reporting to guarantee transparency and practicality in the use of AI in healthcare.	212
Luo (2016) / Australia (29)	NS / 52	ML predictive models in biomedical research	Journal of Medical Internet Research	To develop guidelines for the application of prediction models based on ML in healthcare settings.	571
Checklists for specific study designs
Liu (2020) / UK (30)	CONSORT-AI / 49	Randomised clinical trials involving interventions with AI component	BMJ Nature Medicine Lancet Digital Health	To establish a standard for the reporting of clinical trials employing artificial intelligence-based therapies.	523
Rivera (2020) / UK (31)	SPIRIT-AI / 66	Clinical study protocol involving AI method	BMJ Nature Medicine Lancet Digital Health	To enhance the comprehensiveness of clinical trial protocol documentation.	444
Vasey (2022) / UK (24)	DECIDE-AI / 38	Early-stage clinical evaluation of AI-driven decision support systems	BMJ Nature Medicine	To facilitate the evaluation of research and the reproducibility of their results in healthcare studies using AI-based decision support systems.	96
Moons (2014) / Netherlands (28)	CHARMS / 35	Systematic reviews of prediction modelling studies	PLOS Medicine	To assist with the formulation of a review question and evaluation of all forms of primary prediction modeling studies for systematic reviews.	1080
Checklists for specific clinical areas
Daneshjou (2021) / US (41)	CLEAR Derm / 25	Image-based AI in dermatology	JAMA Dermatology	To synthesize the minimal current material to serve as a guide for dermatological AI developers and reviewers.	42
Haller (2022) / Switzerland (23)	R‑AI‑DIOLOGY / 21	AI in clinical neuroradiology	Neuroradiology	To assist neuroradiologists in evaluating an AI tool for clinical neuroradiology applications.	5
Kwong (2021) / Canada (42)	STREAM-URO / 29	ML in urology	European Urology Focus	To improve ML literacy in the field of urology by establishing a standard for reporting ML applications.	18
Mongan (2020) / US (32)	CLAIM / 42	AI in medical imaging	Radiology: Artificial Intelligence	A recommended method for reporting medical imaging studies.	442
Mörch (2020) / Canada (43)	Canada protocol / 36	AI in mental health	Artificial Intelligence In Medicine	Focusing on mental health and suicide prevention, this study explores methods to more effectively identify and respond to ethical challenges in AI.	13
Schwendicke (2021) / Germany (44)	NS / 31	AI in dental research	Journal of Dentistry	Instructions for the design, execution, and reporting of research with dental AI.	102
Sengupta (2020) / US (45)	PRIME / 28	Cardiovascular Imaging-Related ML	Cardiovascular Imaging	To ensure that the ML models used in cardiovascular imaging studies are reported consistently, this comprehensive guide and associated checklist have been developed.	97
Naqa (2021) / US (46)	CLAMP / 26	AI in medical physics	Medical Physics	To guarantee rigorous and repeatable research of AI / ML in the area of medical physics, introducing a new, necessary checklist for AI / ML applications in Medical Physics (CLAMP).	20
Cerdá‑Alberich (2023) / Spain (22)	MAIC–10 / 10	AI in medical images	Insights into Imaging	A guide for examining publications related to AI in medical imaging, with a focus on study design and evaluation.	4
^a On July 18, 2023; ^b NS: Not specified

Guideline development process

Thirteen articles (54%) provided methodological details about the guideline development process. Ten guidelines (42%) were registered in the EQUATOR website. Two (8%) guidelines were extensions to existing EQUATOR guidelines (CONSORT-AI, SPIRIT-AI), the rest were standalone guidelines (92%). From the seven main components of guideline development defined by the EQUATOR Network, a literature review, a Delphi survey, an expert consensus meeting, and pilot testing were reported by 13 (54%), 6 (25%), 7 (29%), and 6 (25%) guidelines, respectively. Eleven guidelines (46%) reported a funding source, 4 guidelines, albeit vaguely, referred to a future update policy, and 6 guidelines (25%) were adapted or endorsed by a journal or professional organisation. The development of guidelines involved on average 2.3 (SD 2.0, range 0–7) out of the 7 investigated components.

The development process was most comprehensive for CONSORT-AI (30) with all seven development steps completed, followed by SPIRIT-AI (31), which involved all key steps except the reporting of endorsement by a journal or professional organisation. The development of DECIDE-AI (24) was also comprehensive, but no reference was made about intentions to update it in the future.

The mean (SD) development steps of narrative guidelines, general clinical checklists, study design specific checklists and clinical area specific checklists were 0.5 (0.6), 1 (1.2), 5.3 (1.7), and 2.6 (1.6), respectively, with significant difference between the groups (ANOVA, F_3,20=10.53, p < 0.001). Registration on the EQUATOR Network and obtaining funding were associated with more comprehensive development processes, while the endorsement by journals or professional societies was not an indicator of methodological rigor. The mean (SD) development components (except the grouping variable) in registered studies were 3.4 (2.3) versus 1.4 (1.4) in studies not registered in EQUATOR (Welch’s t test, p = 0.031). Funded studies featured 2.6 (1.9), whereas non-funded studies featured 1.1 (1.3) development steps (Welch’s t test, p = .034). The difference between endorsed and not endorsed guidelines was not significant (Welch’s t test, p = 0.95). Registration on EQUATOR or endorsement was not associated with higher citation counts. However, the Google Scholar citation count was higher for funded guidelines (mean: 303.8, SD: 102.9) than for those without funding (mean: 71.9, SD: 17.1) (Welch’s t-test, p = 0.049). Further development details are reported in Table 2.

Table 2

Development details of the included guidelines
Studies	Development methods reported	Registered in EQUATOR website	Literature review	Delphi survey	Expert consensus meeting	Pilot testing	Funded	Update policy was stated	Journal / Society
Narrative guidelines
T.R.U.E. / Buvat (2021) (34)	×	×	×	×	×	×	×	×	✓
Stevens (2020) (35)	×	✓	×	×	×	×	×	×	×
Faes (2020) (36)	×	×	×	×	×	×	×	×	×
Bates (2020) (37)	×	×	×	×	×	×	×	×	×
General reporting checklists
ROBUST-ML / Al-Zaiti (2022) (21)	×	×	×	×	×	×	✓	×	×
Cabitza (2021) (19)	✓	×	✓	×	×	×	×	×	✓
CAIR / Olczak (2021) (38)	×	×	×	×	×	×	✓	×	×
Scott (2021) (39)	✓	×	✓	×	×	×	×	×	×
MINIMAR / Hernandez-Boussard (2020) (40)	×	×	×	×	×	×	×	×	×
MI-CLAIM / Norgeot (2020) (33)	×	✓	×	×	×	×	×	×	×
Luo (2016) (29)	✓	✓	✓	✓	×	×	✓	×	×
Checklists for specific study designs
CONSORT-AI / Liu (2020) (30)	✓	✓	✓	✓	✓	✓	✓	✓	✓
SPIRIT-AI / Cruz Rivera (2020) (31)	✓	✓	✓	✓	✓	✓	✓	✓	×
DECIDE-AI / Vasey (2022) (24)	✓	✓	✓	✓	✓	✓	✓	×	×
CHARMS / Moons (2014) (28)	✓	×	✓	×	✓	✓	✓	×	×
Checklists for specific clinical areas
CLEAR Derm / Daneshjou (2021) (41)	✓	×	✓	×	✓	×	✓	×	×
R-AI-DIOLOGY / Haller (2022) (23)	×	×	×	×	×	×	×	×	×
STREAM-URO / Kwong (2021) (42)	✓	✓	✓	×	✓	×	×	✓	×
CLAIM / Mongan (2020) (32)	×	✓	×	×	×	×	✓	×	✓
Canada protocol / Mörch (2020) (43)	✓	×	✓	✓	×	×	×	×	×
Schwendicke (2021) (44)	✓	✓	✓	✓	×	✓	×	×	✓
PRIME / Sengupta (2020) (45)	✓	✓	✓	×	×	×	✓	✓	✓
CLAMP / Naqa (2021) (46)	×	×	×	×	×	×	×	×	✓
MAIC-10 / Cerdá‑Alberich (2023) (22)	✓	×	✓	×	✓	✓	✓	×	×

Main characteristics of the guidelines

Four papers (17%) were narrative guidelines (34–37) and twenty (83%) comprised a checklist. Seven checklists (30%) were formulated as general AI reporting standards without specific focus on any particular research domain (19, 21, 40, 29, 30, 33, 35–39).

Four guidelines (17%) explicitly stated their focus on distinct study designs, encompassing randomized controlled trials (30) clinical trial protocols (31), early-stage clinical evaluation of AI-driven decision support systems (24), and systematic reviews of prediction modelling studies (28). Two of these are AI-related extensions of well-established checklists, namely the CONSORT for randomised trials (11) and the SPIRIT for clinical trial protocols (47).

Nine guidelines (38%), were designed to address various clinical areas including urology (42), neuroradiology (23), mental health (43), medical physics (46), medical imaging (22, 32), dermatology (41), dentistry (44), and cardiovascular imaging (45). While discussing general AI-related standards, through the journal or the elaboration examples, five (21%) more guidelines could be indirectly associated with nuclear medicine (34), cardiovascular medicine (21, 35), ophthalmology (36), and orthopaedics (38). Some areas overlapped, such as cardiovascular imaging, medical imaging, and cardiovascular medicine.

Of the total publications analysed, 20 (74%) were designed for authors, reviewers, and editors, while eight (30%) were tailored to clinicians and model users, and three (11%) were intended for application developers. Two guidelines (7%) did not specify their intended audience (29, 37). Overall, Table 3 provides an overview of the focus areas covered by the guidelines and their respective target audiences.

Table 3. Characterisation of guidelines by their focus areas and target audiences

Analysis of the structure of reporting guidelines

The domains and areas covered by the guidelines and checklists (as classified in the original paper) with the number of respective items are summarized in Table 4. The structure, and level of detail of both the narrative guidelines and point-by-point checklists showed significant heterogeneity.

Nine guidelines followed the usual IMRAD (Introduction, Methods, Results, and Discussion) structure of research articles with one or more additional sections (e.g., Title/Abstract, Statements/Other information) or omissions (22, 29–32, 38, 42, 46, 48). The number of reporting items within the IMRAD group ranged between 10 (22) and 66 (31) with a median of 40.

Albeit with greater variation, the structure of other 12 guidelines followed the ML pipeline method of clinical AI studies as summarized by MI-CLAIM (i.e., Study design, Data and optimisation, Model performance, Model examination, Reproducibility) with frequent additions of partial clinical information domains (e.g., Participants, Outcomes, or Clinical Deployment) or omissions (19, 21, 41, 45, 23, 28, 33, 35–37, 39, 40). The number of items within the ML group ranged between 8 (36, 37) and 55 (19) (median: 21).

The third group comprised three guidelines with structures not fitting into either the IMRAD or ML pipeline frameworks (34, 43, 44) with items ranging between 4 (34) and 36 (43).

The structure of checklist differed considerably within subgroups. The most similar structure was observed across the checklists for specific study designs, as three (CONSORT-AI, SPIRIT-AI, DECIDE-AI) out of the four followed IMRAD with additional sections such at Title/Abstract and Statements/Other information. Although the fourth checklist (CHARMS) in this subgroup was also designed to be used by authors, namely researchers performing systematic review studies, its structure corresponded more the ML pipeline focusing on details of data and the model (26 items), with less emphasis on participants and results (9 items).

From the general checklists two articles followed the IMRAD structure (29,38) and five the ML pipeline (19, 21, 33, 39, 40). Among checklists for specific clinical areas four followed IMRAD (22, 32, 42, 46), and three the ML pipeline (23, 41, 45). CLAIM integrated the ML workflow elements within the IMRAD format (32).

Altogether, the 24 guidelines contained 704 items in 224 sections. Many items were complex statements covering more than one reporting element. The mean number of items (i.e., depth of detail) differed significantly by the type and focus of guidelines. The mean (SD) item count of narrative guidelines, general checklists, checklists for specific study design and checklists for specific clinical areas were 9.8 (6.4), 32.7 (16.8), 47 (14.0) and 27.6 (9.0) respectively (ANOVA, F_3,20=6.31, p = 0.003). However, we found no association between the depth of detail and guideline structure, with mean (SD) item count of 39.1 (16.3) for guidelines with IMRAD, 23.4 (13.0) for those with ML pipeline and 23.7 (17.2) for guidelines with other structure (ANOVA, F_2,21=3.17, p = 0.063). Furthermore, we found no association between the type (i.e., narrative, or general) and focus (i.e., study design specific, or clinical area specific checklists) and the structure of guidelines (Fisher’s exact test, p = 0.272). However, the structure of guidelines and the comprehensiveness of their development were associated, with mean (SD) 2.8 (1.9), 0.9 (1.2), and 2.3 (1.5) development steps of guidelines with IMRAD, ML pipeline or other structures, respectively (ANOVA, F_2,21=3.72, p = 0.041). The number of development steps and the number of items showed moderate positive correlation (r = 0.56, p = 0.005).

Table 4

Structure of the reporting guidelines
Narrative guidelines
Guideline	Sections	Structure
T.R.U.E. / Buvat (2021) (34)	Four questions with elaboration • Is it true? • Is it reproducible? • Is it useful? • Is it explainable?	Other
Stevens (2020) (35)	Explanations and examples around three areas • Research Question and ML Justification (6 items) • Data sources and pre-processing (training and validation data) (7 items) • Model Training and validation (6 items)	ML pipeline of clinical AI studies
Faes (2020) (36)	Eight questions with elaboration • Was the Study Methodology Prespecified? • Is the Model Being Evaluated in its Intended Stage in the Care Pathway? • Do the Authors Provide Sufficient Clarity on How the Data Were Split? • Are the Image Labels Likely to Reflect the True Disease State? • How Is Diagnostic Accuracy Reported? • Is the Dataset Used in Model Development Reflective of the Setting in Which the Model Will Be Applied? • Is the Output of the Model Interpretable and Can it Be Interrogated? Are Differential Diagnoses and Estimates of Confidence Provided? • Is the Performance Reproducible and Generalizable?	ML pipeline of clinical AI studies
Bates (2020) (37)	Two parallel sections: “recommendations for study design and conduct”, and “recommendations for reporting”. Four main issues under each section, with corresponding broad items, which cover multiple elements. • Validation (3 items) • Uncertainty (1 item) • Implementation (1 item) • Data (3 items)	ML pipeline of clinical AI studies
General reporting checklists
ROBUST-ML / Al-Zaiti (2022) (21)	Quality items with corresponding important red flags to observe, organised in five domains: • General design considerations (3 items) • Data quality considerations (6 items) • Feature engineering considerations (5 items) • Model development considerations (10 items) • Considerations for clinical utility (6 items)	ML pipeline of clinical AI studies
Cabitza (2021) (19)	Questions, which may cover multiple items, are organised around 6 domains • Problem understanding (6 questions) • Data understanding (3 questions and 13 subitems) • Data preparation (4 questions and 2 subitems) • Modelling (3 questions) • Validation (8 questions and 9 subitems) • Deployment (7 questions)	ML pipeline of clinical AI studies
CAIR / Olczak (2021) (38)	Recommendations follow the reporting structure of a research article (5 parts) • Title and abstract (2 items) • Introduction (1 item) • Methods (9 items, 25 subitems) • Results (1 item) • Discussion and other information (2 items)	IMRAD
Scott (2021) (39)	12 questions • What is the purpose of the algorithm? (1) • How good were the data used to train the algorithm? (2a) • To what extent were the data accurate and free of bias? (2b) • Were the data standardised and interoperable? (2c) • Were there sufficient data to train the algorithm? (3) • How well does the algorithm perform? (4) • Is the algorithm transferable to new clinical settings? (5) • Are the outputs of the algorithm clinically intelligible? (6) • How will this algorithm fit into and complement current workflows? (7) • Has use of the algorithm been shown to improve patient care and outcomes? (8) • Could the algorithm cause patient harm? (9) • Does use of the algorithm raise ethical, legal or social concerns? (10)	ML pipeline of clinical AI studies
MINIMAR / Hernandez-Boussard (2020) (40)	Reporting items and their descriptions are organised in four domains • Study population and setting (4 items) • Patient demographic characteristics (5 items) • Model architecture (8 items) • Model evaluation (4 items)	ML pipeline of clinical AI studies
MI-CLAIM / Norgeot (2020) (33)	Statements of good reporting practices are organised around 6 domains in 5 parts • Study design - Part 1 (5 items) • Data and optimisation - Parts 2,3 (5 items) • Model performance - Part 4 (3 items) • Model examination – Part 5 (5 items) • Reproducibility – Part 6: choose appropriate tier of transparency (1 item)	ML pipeline of clinical AI studies
Luo (2016) (29)	Topics and reporting elements are organised along the reporting structure of a research article (5 parts) • Title and abstract o Title (1 item) o Abstract (5 items) • Introduction o Rationale (2 items) o Objectives (1 item) • Methods o Describe the setting (2 items) o Define the prediction problem (8 items) o Prepare data for model building (11 items) o Build the predictive model (10 items) • Results o Report the final model performance (6 items) • Discussion o Clinical implications (1 item) • Limitations of the model (4 items) o Unexpected results during the experiments (1 item)	IMRAD
Checklists for specific study designs
CONSORT-AI / Liu (2020) (30)	Reporting items are AI-related extensions to the CONSORT checklist, organised in 25 sections along the reporting structure of a clinical study report (7 parts) • Title and abstract o Title and abstract (2 AI extension items) • Introduction o Background and objectives (2 items + 1 AI extension) • Methods o Trial design (2 items) o Participants (2 items + 3 AI extensions) o Interventions (1 item + 6 AI extensions) o Outcomes (2 items) o Sample size (2 items) • Randomization o Sequence generation (2 items) o Allocation concealment mechanism (1 item) o Implementation (1 item) o Blinding (2 items) o Statistical methods (2 items) • Results o Participant flow (2 items) o Recruitment (2 items) o Baseline data (1 item) o Numbers analysed (1 item) o Outcomes and estimation (2 items) o Ancillary analyses (1 item) o Harms (1 item + 1 AI extension) • Discussion o Limitations (1 item) o Generalisability (1 item) o Interpretation (1 item) • Other information o Registration (1 item) o Protocol (1 item) o Funding (1 item + 1 AI extension)	IMRAD
SPIRIT-AI / Cruz Rivera (2020) (31)	Reporting items are AI-related extensions to the SPIRIT checklist, organised in 33 sections along the reporting structure of a clinical study protocol (8 parts) • Administrative information o Title (1 item + 2 AI extensions) o Trial registration (2 items) o Protocol version (1 item) o Funding (1 item) o Roles and responsibilities (4 items) • Introduction o Background and rationale 2 items + 2 AI extensions) o Objectives (1 item) o Trial design (1 item) • Methods: participants, interventions and outcomes o Study setting (1 item + 1 AI extension) o Eligibility criteria (1 item + 2 AI extensions) o Interventions (4 items + 6 AI extensions) o Outcomes (1 item) o Participant timeline (1 item) o Sample size (1 item) o Recruitment (1 item) • Methods: assignment of interventions (for controlled trials) o Sequence generation (1 item) o Allocation concealment (1 item) o Implementation (1 item) o Blinding (masking) (2 items) • Methods: data collection, management and analysis o Data collection methods (2 items) o Data management (1 item) o Statistical methods (3 items) • Methods: monitoring o Data monitoring (2 items) o Harms (1 item + 1 AI extension) o Auditing (1 item) • Ethics and dissemination o Research ethics approval (1 item) o Protocol amendments (1 item) o Consent or ascent (2 items) o Confidentiality (1 item) o Declaration of interests (1 item) o Access to data (1 item + 1 AI extension) o Ancillary and post-trial care (1 item) o Dissemination policy (3 items) • Appendices o Informed consent materials (1 item) o Biological specimens (1 item)	IMRAD
DECIDE-AI / Vasey (2022) (24)	AI-specific and general reporting items in 17 AI-specific and 10 generic* reporting themes organised around the reporting structure of a clinical study report (6 parts) • Title and abstract o Title (1 item) o Abstract (1 item)* • Introduction o Intended use (2 items) o Objectives (1 item)* • Methods o Research governance (1 item)* o Participants (3 items) o AI system (3 items) o Implementation (2 items) o Outcomes (1 item)* o Safety and errors (2 items) o Human factors (1 item) o Analysis (1 item)* o Ethics (1 item) o Patient involvement (1 item)* • Results o Participants (2 items) o Implementation (2 items) o Main results (1 item)* o Subgroup analysis (1 item)* o Modifications (1 item) o Human-computer agreement (1 item) o Safety and errors (2 items) o Human factors (2 items) • Discussion o Support for intended use (1 item) o Safety and errors (1 item) o Strengths and limitations (1 item)* • Statements o Data availability (1 item) o Conflicts of interest (1 item)*	IMRAD
CHARMS / Moons (2014) (28)	Reporting items are organised in 11 domains • Source of data (1 item) • Participants (4 items) • Outcome(s) to be predicted (6 items) • Candidate predictors (or index tests) (5 items) • Sample size (2 items) • Missing data (3 items) • Model development (5 items) • Model performance (2 items) • Model evaluation (2 items) • Results (3 items) • Interpretation and discussion (2 items)	ML pipeline of clinical AI studies
Checklists for specific clinical areas
CLEAR Derm / Daneshjou (2021) (41)	Items are organised under 4 sections and anonymous subsections • Data (15 items organised in 5 subsections) • Technique (4 items organised in 2 subsections) • Technical assessment (4 items organised in 2 subsections) • Application (2 items in 1 subsection)	ML pipeline of clinical AI studies
R-AI-DIOLOGY / Haller (2022) (23)	Reporting items (questions) are organised under 10 domains • Disease/domain (2 items) • Preselection of cases and reference dataset (4 items) • Data parameters (2 items) • Data quality check (1 item) • Anonymization, coding and de-identification (2 items) • Data storage and processing (2 items) • Integration in the radiologist’s workflow (2 items) • Update (3 items) • Validation and labels (2 items) • Ground truth and reference (1 item)	ML pipeline of clinical AI studies
STREAM-URO / Kwong (2021) (42)	AI related reporting items (and explanations) are mapped to those of the TRIPOD statement, organised along the reporting structure of a clinical study protocol (6 parts) • Title, abstract (1 item) • Introduction (2 items) • Methods (12 items) • Results (8 items) • Discussion (3 items) • Supplemental materials (3 items)	IMRAD
CLAIM / Mongan (2020) (32)	Items are organised along 7 sections of the reporting structure of a clinical study, and nine topics • Title or Abstract (1 item) • Abstract (1 item) • Introduction (2 items) • Methods o Study design (2 items) o Data (7 items) o Ground Truth (5 items) o Data Partitions (3 items) o Model (3 items) o Training (3 items) o Evaluation (5 items) • Results o Data (2 items) o Model performance (3 items) • Discussion (2 items) • Other information (3 items)	IMRAD
Canada protocol / Mörch (2020) (43)	Items are organised in five broad categories • Description (8 items) • Privacy and transparency (8 items) • Security (6 items) • Health-related risks (6 items) • Biases (8 items)	Other
Schwendicke (2021) (44)	Items are listed under two main sections • Planning and conduction (9 items) • Reporting (22 items)	Other
PRIME / Sengupta (2020) (45)	Items are organised under 7 sections • Designing the Study Plan (5 items) • Data Standardization, Feature Engineering, and Learning (6 items) • Selection of Machine Learning Models (6 items) • Model Assessment (2 items) • Model Evaluation (4 items) • Best Practices for Model Replicability (3 items) • Reporting Limitations, Biases and Alternatives (2 items)	ML pipeline of clinical AI studies
CLAMP / Naqa (2021) (46)	Items are organised under 6 sections • Abstract (3 items) • Introduction (3 items) • Materials (9 items) • Methods: machine learning algorithm (3 items) • Methods: performance statistics (5 items) • Discussion (3 items)	IMRAD
MAIC-10 / Cerdá‑Alberich (2023) (22)	Checklist items (with descriptions specifying multiple reporting elements) are paired with four article sections with overlap • Introduction (1 item) • Materials and methods (5 items) • Materials and methods, results (1 item) • Results, discussion (1 item) • Discussion (2 items)	IMRAD

According to our knowledge, this is the first systematic review that provides a broad overview of reporting guidelines for medical AI studies. While all 24 included guidelines aim to improve the transparency of reporting, and hence, indirectly improve the quality and clinical utility of medical AI studies, they were heterogenous in terms of their target audiences, focus area, development process and structure. The multiplicity and variety of items in the 24 reporting guidelines also reflect that there is not yet an established methodological framework for designing and reporting AI-based studies.

The proliferation of methodological frameworks and definitions is a general phenomenon in the digital transformation of healthcare (49–51). Medical AI reporting guidelines are no exception to this trend. With over 700 items, albeit with overlap, the number of concepts that should be covered in an AI study report is overwhelming. A qualitative content analysis found similar heterogeneity in the concepts covered by medical AI reporting guidelines (52).

The guidelines’ level of detail varied significantly regardless of whether they followed the IMRAD, ML pipeline or other structure. This variety makes it difficult for authors or clinicians to get a firm grip on what represents best practice in reporting, and how detailed a standardized medical AI study report should be. This may delay the clinical uptake of medical AI technologies, hamper evidence syntheses and consequently impede reliable information retrieval, which is essential for the development of automated systematic literature reviews (53).

Although the study design, clinical area, or a target journal may guide the choice of a reporting guideline, it remains a question, what represents a universal standard for best reporting practices. For a hint, authors may look at, which underwent the most robust development process, namely CONSORT-AI, SPIRIT-AI, and DECIDE-AI with 7, 6, and 5 development steps and 49, 66, and 38 reporting items, respectively. All these guidelines follow the IMRAD structure. Indeed, the more comprehensively developed guidelines were more detailed, and the IMRAD structure was associated with more robust development and greater depth of detail. However, the number of AI-specific items was only 14 in CONSORT-AI, 16 in SPIRIT-AI, and 28 in DECIDE-AI, suggesting that broad consensus supports a lower number of essential items compared to the most comprehensive general AI-related checklists, which featured more than 50 items (19, 29). CONSORT-AI and SPIRIT-AI focus on randomised clinical trials, while DECIDE-AI is concerned about clinical implementation pilot studies of AI-based technologies. While the long-awaited TRIPOD-AI checklist (54) for predictive modelling and the STARD-AI checklist (55) for diagnostic accuracy studies are under development, AI extensions of clinical study guidelines may not cover all the application fields of AI in medicine. In certain research contexts, an established checklist for a special study design, and a general AI-based checklist may be combined for optimal results.

The citation count may be another clue for authors to select well-established reporting guidelines. In this regard, CHARMS, the checklist by Luo et al., and CONSORT-AI stand out of the crowd with respectively 1080, 573, and 523 Google Scholar citations. The performance of CONSORT-AI is notable, as this guideline was published in 2020, several years after CHARMS (2014) and the work by Luo et al. (2016). While the checklist by Luo et al. has been tested (and cited) over time, it has not been updated for nine years since its publication (29). A policy for regular updates was generally lacking from the guidelines, with only four guidelines making remarks about the need for future updates (30, 31, 42, 45), which is an important aspect in an area of rapid methodological development. While the number of citations for these guidelines seems impressive, these numbers also show their low adoption rates in the literature. Since 2016, the publication of the first clinical study reporting guideline, over 110.000 AI-related studies were published on PubMed (56), while the total citation count for all identified guidelines was only 4277 over the same period.

The publication outlets of guidelines provided useful guidance in the selection of guidelines, although not without some discrepancies. The ten guidelines registered on the EQUATOR website had more robust development than the non-registered ones, but some registered guidelines did not report details of their development (33, 35). However, CHARMS, a well-developed and the most cited guideline was not registered on EQUATOR. The citation analysis did not signal a greater uptake (more citations) of registered guidelines. To signal their quality, well developed guidelines should be registered on the EQUATOR website. CONSORT-AI, and SPIRIT-AI were published simultaneously in three prestigious journals (BMJ, Nature Medicine, Lancet Digital Health), and DECIDE-AI was published in two (BMJ, Nature Medicine). However, the author guidelines of BMJ and Nature Medicine recommended the use of CONSORT and reporting guidelines from EQUATOR, but did not mention CONSORT-AI (57, 58). The author guidelines of Lancet Digital Health recommended CONSORT-AI and SPIRIT-AI, but not MI-CLAIM, which was published in the same journal (59). Review articles that aimed to cover reporting guidelines for medical AI, usually mentioned CONSORT-AI and SPIRIT AI, along with guidelines under development, such as the STARD-AI, TRIPOD-AI and PROBAST-ML extensions of renowned EQUATOR guidelines. Reviews also mentioned CLAIM (proposed by the Radiological Society of North America), MINIMAR (published in the Journal of the American Medical Informatics Association), DECIDE-AI, MI-CLAIM, CHARMS, CAIR, and the guidelines from Luo, Cabitza, Bates, and Scott (14, 20, 60, 61).

Beyond the eminent examples cited by previous reviews, being the first systematic review in the field, our study introduces in detail a comprehensive list of reporting guidelines and demonstrates the challenges of guideline choice from a user perspective. Our analysis supports also the work of guideline developers. However, the limitation of our study is that some of our analyses required judgement, especially about delineating narrative guidelines or checklists, as well as about the inclusion of some articles, which were somewhat loosely related to study reporting. Potential omissions include a critical checklist to assess patient benefit (62), or an extensive report from the National Academy of Medicine (63). We also excluded guidelines under development, such as STARD-AI, TRIPOD-AI or PROBAST-AI.

Altogether, currently there is no professional consensus on the content and structure of the reporting guidelines on medical AI studies. The variety of reporting guidelines poses a challenge for researchers to follow what represents best practice in reporting a medical AI study. Based on the robustness of development process and support from the literature, the CONSORT-AI extension is the most established tool. However, focusing on randomised controlled trials, CONSORT-AI does not cover the breadth of potential clinical and study design aspects of medical AI studies. While other AI extensions of EQUATOR guidelines are awaited, probably in combination with established EQUATOR guidelines, a modular AI-specific reporting guideline would provide the greatest flexibility for researchers and clinicians to follow the best practice in designing, writing and interpreting medical AI studies. Such guideline should undergo a robust development under the cooperation of powerful organisations and should be widely publicized among the medical AI research community to achieve its desired impact.

AI: Artificial Intelligence

EQUATOR: Enhancing the QUAlity and Transparency Of health Research

IMRAD: Introduction, Methods, Results, and Discussion

ML: Machine Learning

FDA: The US Food and Drug Administration

CONSORT: Consolidated Standards of Reporting Trials

PRISMA: Preferred Reporting Items for Systematic reviews and Meta-Analyses

Ethics approval and consent to participate

Not applicable

Consent for publication

Not applicable

Availability of data and materials

All data generated or analysed during this study are included in this published article [and its supplementary information files]

Competing interests

The authors declare that they have no competing interests.

Funding

This project has been supported by the National Research, Development, and Innovation Fund of Hungary, financed under the TKP2021-NKTA-36 funding scheme (Project: Development and evaluation of innovative and digital health technologies; Subproject: Evaluation of digital medical devices: efficacy, safety and social utility).

Authors' contributions

Conception and design: ZZ, LK, HMN, JC, LG, MP; data collection: ZZ, HMN, MP; analysis: ZZ, HMN, MP; drafting the manuscript: ZZ, HMN; supervision: LG, MP; obtain funding: LK, LG, MP. All authors have read and approved the final manuscript.

Acknowledgements

Not applicable

Weissler EH, Naumann T, Andersson T, Ranganath R, Elemento O, Luo Y, et al. The role of machine learning in clinical research: transforming the future of evidence generation. Trials [Internet]. 2021 Dec 16;22(1):537. Available from: https://trialsjournal.biomedcentral.com/articles/10.1186/s13063-021-05489-x
Davenport T, Kalakota R. The potential for artificial intelligence in healthcare. Futur Healthc J [Internet]. 2019 Jun;6(2):94–8. Available from: https://www.rcpjournals.org/lookup/doi/10.7861/futurehosp.6-2-94
Briganti G, Le Moine O. Artificial Intelligence in Medicine: Today and Tomorrow. Front Med [Internet]. 2020 Feb 5;7. Available from: https://www.frontiersin.org/article/10.3389/fmed.2020.00027/full
Busnatu Ștefan, Niculescu AG, Bolocan A, Petrescu GED, Păduraru DN, Năstasă I, et al. Clinical Applications of Artificial Intelligence—An Updated Overview. J Clin Med [Internet]. 2022 Apr 18;11(8):2265. Available from: https://www.mdpi.com/2077-0383/11/8/2265
Muehlematter UJ, Daniore P, Vokinger KN. Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis. Lancet Digit Heal [Internet]. 2021;3(3):e195–203. Available from: http://dx.doi.org/10.1016/S2589-7500(20)30292-2
Masic I, Miokovic M, Muhamedagic B. Evidence Based Medicine - New Approaches and Challenges. Acta Inform Medica [Internet]. 2008;16(4):219. Available from: http://www.scopemed.org/fulltextpdf.php?mno=6371
Chidambaram AG, Josephson M. Clinical research study designs: The essentials. Pediatr Investig [Internet]. 2019 Dec 21;3(4):245–52. Available from: https://onlinelibrary.wiley.com/doi/10.1002/ped4.12166
Stephenson JM. Overview of study design in clinical epidemiology. Sex Transm Infect [Internet]. 2000 Aug 1;76(4):244–7. Available from: https://sti.bmj.com/lookup/doi/10.1136/sti.76.4.244
Moher D. Reporting guidelines: doing better for readers. BMC Med [Internet]. 2018 Dec 14;16(1):233. Available from: https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-018-1226-0
Papakostidis C, Giannoudis P V. Meta-analysis. What have we learned? Injury [Internet]. 2023 May;54:S30–4. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0020138322004235
Schulz KF, Altman DG, Moher D. CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med [Internet]. 2010 Dec 24;8(1):18. Available from: http://bmcmedicine.biomedcentral.com/articles/10.1186/1741-7015-8-18
Moher D, Schulz KF, Simera I, Altman DG. Guidance for Developers of Health Research Reporting Guidelines. PLoS Med [Internet]. 2010 Feb 16;7(2):e1000217. Available from: https://dx.plos.org/10.1371/journal.pmed.1000217
EQUATOR. Enhancing the QUAlity and Transparency Of health Research [Internet]. 2023 [cited 2023 Jun 12]. Available from: https://www.equator-network.org/
Shelmerdine SC, Arthurs OJ, Denniston A, Sebire NJ. Review of study reporting guidelines for clinical studies using artificial intelligence in healthcare. BMJ Heal Care Informatics [Internet]. 2021 Aug 23;28(1):e100385. Available from: https://informatics.bmj.com/lookup/doi/10.1136/bmjhci-2021-100385
Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Heal [Internet]. 2019 Oct;1(6):e271–97. Available from: https://linkinghub.elsevier.com/retrieve/pii/S2589750019301232
Nagendran M, Chen Y, Lovejoy CA, Gordon AC, Komorowski M, Harvey H, et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ [Internet]. 2020 Mar 25;m689. Available from: https://www.bmj.com/lookup/doi/10.1136/bmj.m689
Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers. Korean J Radiol [Internet]. 2019;20(3):405. Available from: https://www.kjronline.org/DOIx.php?id=10.3348/kjr.2019.0025
Yusuf M, Atal I, Li J, Smith P, Ravaud P, Fergie M, et al. Reporting quality of studies using machine learning models for medical diagnosis: a systematic review. BMJ Open [Internet]. 2020 Mar 23;10(3):e034568. Available from: https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2019-034568
Cabitza F, Campagner A. The need to separate the wheat from the chaff in medical informatics. Int J Med Inform [Internet]. 2021 Sep;153(xxxx):104510. Available from: https://doi.org/10.1016/j.ijmedinf.2021.104510
Ibrahim H, Liu X, Denniston AK. Reporting guidelines for artificial intelligence in healthcare research. Clin Experiment Ophthalmol [Internet]. 2021 Jul 25;49(5):470–6. Available from: https://onlinelibrary.wiley.com/doi/10.1111/ceo.13943
Al-Zaiti SS, Alghwiri AA, Hu X, Clermont G, Peace A, Macfarlane P, et al. A clinician’s guide to understanding and critically appraising machine learning studies: a checklist for Ruling Out Bias Using Standard Tools in Machine Learning (ROBUST-ML). Eur Hear J - Digit Heal. 2022;3(2):125–40.
Cerdá-Alberich L, Solana J, Mallol P, Ribas G, García-Junco M, Alberich-Bayarri A, et al. MAIC–10 brief quality checklist for publications using artificial intelligence and medical images. Insights Imaging [Internet]. 2023;14(1). Available from: https://doi.org/10.1186/s13244-022-01355-9
Haller S, Van Cauter S, Federau C, Hedderich DM, Edjlali M. The R-AI-DIOLOGY checklist: a practical checklist for evaluation of artificial intelligence tools in clinical neuroradiology. Neuroradiology. 2022;64(5):851–64.
Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nat Med [Internet]. 2022 May 18;28(5):924–33. Available from: https://www.nature.com/articles/s41591-022-01772-9
Zrubka Z, Gulacsi L, Pentek M. Time to start using checklists for reporting artificial intelligence in health care and biomedical research: a rapid review of available tools. In: Szakál A, editor. 2022 IEEE 26th International Conference on Intelligent Engineering Systems (INES) [Internet]. Budapest: IEEE; 2022. p. 000015–20. Available from: https://m2.mtmt.hu/api/publication/33051159
Simera I, Moher D, Hirst A, Hoey J, Schulz KF, Altman DG. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med [Internet]. 2010 Dec 26;8(1):24. Available from: http://bmcmedicine.biomedcentral.com/articles/10.1186/1741-7015-8-24
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ [Internet]. 2021 Mar 29;n71. Available from: https://www.bmj.com/lookup/doi/10.1136/bmj.n71
Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, et al. Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist. PLoS Med. 2014;11(10).
Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: A multidisciplinary view. J Med Internet Res. 2016;18(12):1–10.
Liu X, Rivera SC, Moher D, Calvert MJ, Denniston AK. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: The CONSORT-AI Extension. BMJ. 2020;370.
Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ, Darzi A, et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Nat Med. 2020;26(9):1351–63.
Mongan J, Moy L, Kahn CE. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiol Artif Intell. 2020;2(2):e200029.
Norgeot B, Quer G, Beaulieu-Jones BK, Torkamani A, Dias R, Gianfrancesco M, et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat Med [Internet]. 2020 Sep 9;26(9):1320–4. Available from: https://www.nature.com/articles/s41591-020-1041-y
Buvat I, Orlhac F. The T.R.U.E. Checklist for identifying impactful artificial intelligence-based findings in nuclear medicine: Is it true? Is it reproducible? Is it useful? Is it explainable? J Nucl Med. 2021;62(6):752–4.
Stevens LM, Mortazavi BJ, Deo RC, Curtis L, Kao DP. Recommendations for Reporting Machine Learning Analyses in Clinical Research. Circ Cardiovasc Qual Outcomes. 2020;13(10):E006556.
Faes L, Liu X, Wagner SK, Fu DJ, Balaskas K, Sim D, et al. A clinician’s guide to artificial intelligence: How to critically appraise machine learning studies. Transl Vis Sci Technol. 2020;9(2):1–9.
Bates DW, Auerbach A, Schulam P, Wright A, Saria S. Reporting and implementing interventions involving machine learning and artificial intelligence. Ann Intern Med. 2020;172(11):S137–44.
Olczak J, Pavlopoulos J, Prijs J, Ijpma FFA, Doornberg JN, Lundström C, et al. Presenting artificial intelligence, deep learning, and machine learning studies to clinicians and healthcare stakeholders: an introductory reference with a guideline and a Clinical AI Research (CAIR) checklist proposal. Acta Orthop. 2021;92(5):513–25.
Scott I, Carter S, Coiera E. Clinician checklist for assessing suitability of machine learning applications in healthcare. BMJ Heal Care Informatics. 2021;28(1).
Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum information for medical AI reporting): Developing reporting standards for artificial intelligence in health care. J Am Med Informatics Assoc. 2020;27(12):2011–5.
Daneshjou R, Barata C, Betz-Stablein B, Celebi ME, Codella N, Combalia M, et al. Checklist for Evaluation of Image-Based Artificial Intelligence Reports in Dermatology: CLEAR Derm Consensus Guidelines from the International Skin Imaging Collaboration Artificial Intelligence Working Group. JAMA Dermatology [Internet]. 2022 Jan 1;158(1):90–6. Available from: https://jamanetwork.com/journals/jamadermatology/fullarticle/2786912
Kwong JCC, McLoughlin LC, Haider M, Goldenberg MG, Erdman L, Rickard M, et al. Standardized Reporting of Machine Learning Applications in Urology: The STREAM-URO Framework. Eur Urol Focus [Internet]. 2021;7(4):672–82. Available from: https://doi.org/10.1016/j.euf.2021.07.004
Mörch CM, Gupta A, Mishara BL. Canada protocol: An ethical checklist for the use of artificial Intelligence in suicide prevention and mental health. Artif Intell Med [Internet]. 2020;108(July):101934. Available from: https://doi.org/10.1016/j.artmed.2020.101934
Schwendicke F, Singh T, Lee J hong H, Gaudin R, Chaurasia A, Wiegand T, et al. Artificial intelligence in dental research: Checklist for authors, reviewers, readers. J Dent. 2021;107(February).
Sengupta PP, Shrestha S, Berthon B, Messas E, Donal E, Tison GH, et al. Proposed Requirements for Cardiovascular Imaging-Related Machine Learning Evaluation (PRIME): A Checklist: Reviewed by the American College of Cardiology Healthcare Innovation Council. JACC Cardiovasc Imaging [Internet]. 2020 Sep;13(9):2017–35. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1936878X20306367
El Naqa I, Boone JM, Benedict SH, Goodsitt MM, Chan HP, Drukker K, et al. AI in medical physics: guidelines for publication. Med Phys. 2021;48(9):4711–4.
Chan AW, Tetzlaff JM, Gotzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ [Internet]. 2013 Jan 9;346(jan08 15):e7586–e7586. Available from: https://www.bmj.com/lookup/doi/10.1136/bmj.e7586
Vasey B, Clifton DA, Collins GS, Denniston AK, Faes L, Geerts BF, et al. DECIDE-AI: new reporting guidelines to bridge the development-to-implementation gap in clinical artificial intelligence. Nat Med [Internet]. 2021;27(2):186–7. Available from: http://dx.doi.org/10.1038/s41591-021-01229-5
Kolasa K, Kozinski G. How to Value Digital Health Interventions? A Systematic Literature Review. Int J Environ Res Public Health [Internet]. 2020 Mar 23;17(6):2119. Available from: https://www.mdpi.com/1660-4601/17/6/2119
Zah V, Burrell A, Asche C, Zrubka Z. Paying for Digital Health Interventions – What Evidence is Needed? Acta Polytech Hungarica. 2022;19(9):179–99.
Burrell A, Zrubka Z, Champion A, Zah V, Vinuesa L, Holtorf AP, et al. How Useful Are Digital Health Terms for Outcomes Research? An ISPOR Special Interest Group Report. Value Heal [Internet]. 2022 Sep;25(9):1469–79. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1098301522019325
Crossnohere NL, Elsaid M, Paskett J, Bose-Brill S, Bridges JFPP. Guidelines for Artificial Intelligence in Medicine: Literature Review and Content Analysis of Frameworks. J Med Internet Res [Internet]. 2022 Aug 25;24(8):e36823. Available from: https://www.jmir.org/2022/8/e36823
Tóth B, Berek L, Gulácsi L, Péntek M, Zrubka Z. Automation of systematic reviews of biomedical literature : a systematic review of studies indexed in PubMed (Preprint). 2023;1–41. Available from: https://doi.org/10.21203/rs.3.rs-3425312/v1
Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma JB, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open [Internet]. 2021 Jul;11(7):e048008. Available from: https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2020-048008
Sounderajah V, Ashrafian H, Golub RM, Shetty S, De Fauw J, Hooft L, et al. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ Open [Internet]. 2021 Jun 28;11(6):e047709. Available from: https://bmjopen.bmj.com/lookup/doi/10.1136/bmjopen-2020-047709
National Library of Medicine. PubMed [Internet]. 2023 [cited 2023 Jul 12]. Available from: https://pubmed.ncbi.nlm.nih.gov/
Nature medicine. Clinical Research [Internet]. [cited 2023 Sep 7]. Available from: https://www.nature.com/nm/editorial-policies/clinicalresearch
BMJ. Article types and preparation [Internet]. [cited 2023 Sep 7]. Available from: https://www.bmj.com/about-bmj/resources-authors/article-types
The Lancet Digital Health. Information for Authors [Internet]. 2023 [cited 2023 Sep 7]. Available from: https://www.thelancet.com/pb-assets/Lancet/authors/tldh-info-for-authors-1654073773513.pdf
Campbell JP, Lee AY, Abràmoff M, Keane PA, Ting DSW, Lum F, et al. Reporting Guidelines for Artificial Intelligence in Medical Research. Ophthalmology [Internet]. 2020 Dec;127(12):1596–9. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0161642020308812
Meshaka R, Pinto Dos Santos D, Arthurs OJ, Sebire NJ, Shelmerdine SC. Artificial intelligence reporting guidelines: what the pediatric radiologist needs to know. Pediatr Radiol [Internet]. 2022 Oct 1;52(11):2101–10. Available from: https://link.springer.com/10.1007/s00247-021-05129-1
Vollmer S, Mateen BA, Bohner G, Király FJ, Ghani R, Jonsson P, et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ [Internet]. 2020 Mar 20;l6927. Available from: https://www.bmj.com/lookup/doi/10.1136/bmj.l6927
Matheny, M., S. Thadaney Israni, M. Ahmed and DW (Editors). Artificial Intelligence in Health Care: The Hope, the Hype, the Promise, the Peril. Matheny, M., S. Thadaney Israni, M. Ahmed and DW, editor. Washington, DC: National Academy of Medicine; 2022.

No competing interests reported.

Artificial Intelligence in Medicine: A Systematic Review of Guidelines on Reporting and Interpreting Studies

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Figures

Introduction

Methods

Search strategy

Selection criteria

Study selection

Data extraction and synthesis

Results

Search results

Main characteristics of the publications

Guideline development process

Main characteristics of the guidelines

Analysis of the structure of reporting guidelines

Discussion

Conclusions

Abbreviations

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1