See the published version of this preprint at BMC Medical Research Methodology.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
Research article
Jenna Marie Reps
Janssen Research and Development Titusville
Corresponding Author
ORCiD: https://orcid.org/0000-0002-2970-0778
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Objective To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to externally validate patient-level prediction models at scale.
Materials & Methods Five previously published prognostic models (ATRIA, CHADS2, CHADS2VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks and a network study was run that enabled the five models to be externally validated across nine datasets spanning three countries and five independent sites.
Results The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and their performances in predicting stroke within 1 year of initial atrial fibrillation diagnosis for females were comparable with existing studies. The validation network study took 60 days once the models were replicated and an R package for the study was published to collaborators at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation.
Discussion This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability and reproducibility of prediction models, but without collaborative approaches it can take three or more years to be validated by one independent researcher.
Conclusion In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months.
Keywords
patient-level prediction; prognostic model; external validation; transportability; collaborative network
This is a list of supplementary files associated with this preprint. Click to download.
Loading...
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Research Methodology.
No community comments so far
Editorial decision: Accept
On 23 Apr, 2020
Submission checks complete
On 17 Jul, 2019
No comments provided
Editorial decision: Minor revision
On 22 Apr, 2020
Editor assigned
On 20 Apr, 2020
Submission checks complete
On 19 Apr, 2020
Editor invited
On 19 Apr, 2020
No comments provided
Editorial decision: Minor revision
On 01 Apr, 2020
Review #1 received
Received 31 Mar, 2020
Reviewer #1 agreed
On 27 Mar, 2020
Reviewers invited
Invitations sent on 23 Jan, 2020
Editor assigned
On 20 Jan, 2020
Submission checks complete
On 19 Jan, 2020
Editor invited
On 19 Jan, 2020
Version 1
Posted 19 Jul, 2019
No comments provided
Editorial decision: Major revision
On 19 Dec, 2019
Review #2 received
Received 15 Dec, 2019
Reviewer #2 agreed
On 26 Nov, 2019
Review #1 received
Received 09 Nov, 2019
Reviewer #1 agreed
On 25 Oct, 2019
Reviewers invited
Invitations sent on 28 Sep, 2019
Submission checks complete
On 17 Jul, 2019
Editor assigned
On 10 Jul, 2019
First submitted
On 09 Jul, 2019
Editor invited
On 09 Jul, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Health Economics & Outcomes Research
Learn more about our company.
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
See the published version of this preprint at BMC Medical Research Methodology.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Jenna Marie Reps
Janssen Research and Development Titusville
Corresponding Author
ORCiD: https://orcid.org/0000-0002-2970-0778
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Research Methodology.
No community comments so far
Editorial decision: Accept
On 23 Apr, 2020
Submission checks complete
On 17 Jul, 2019
No comments provided
Editorial decision: Minor revision
On 22 Apr, 2020
Editor assigned
On 20 Apr, 2020
Submission checks complete
On 19 Apr, 2020
Editor invited
On 19 Apr, 2020
No comments provided
Editorial decision: Minor revision
On 01 Apr, 2020
Review #1 received
Received 31 Mar, 2020
Reviewer #1 agreed
On 27 Mar, 2020
Reviewers invited
Invitations sent on 23 Jan, 2020
Editor assigned
On 20 Jan, 2020
Submission checks complete
On 19 Jan, 2020
Editor invited
On 19 Jan, 2020
Version 1
Posted 19 Jul, 2019
No comments provided
Editorial decision: Major revision
On 19 Dec, 2019
Review #2 received
Received 15 Dec, 2019
Reviewer #2 agreed
On 26 Nov, 2019
Review #1 received
Received 09 Nov, 2019
Reviewer #1 agreed
On 25 Oct, 2019
Reviewers invited
Invitations sent on 28 Sep, 2019
Submission checks complete
On 17 Jul, 2019
Editor assigned
On 10 Jul, 2019
First submitted
On 09 Jul, 2019
Editor invited
On 09 Jul, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Health Economics & Outcomes Research
License:
This work is licensed under a CC BY 4.0 License. Read Full License
View the published article at BMC Medical Research Methodology.
Objective To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to externally validate patient-level prediction models at scale.
Materials & Methods Five previously published prognostic models (ATRIA, CHADS2, CHADS2VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks and a network study was run that enabled the five models to be externally validated across nine datasets spanning three countries and five independent sites.
Results The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and their performances in predicting stroke within 1 year of initial atrial fibrillation diagnosis for females were comparable with existing studies. The validation network study took 60 days once the models were replicated and an R package for the study was published to collaborators at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation.
Discussion This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability and reproducibility of prediction models, but without collaborative approaches it can take three or more years to be validated by one independent researcher.
Conclusion In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months.
This is a list of supplementary files associated with this preprint. Click to download.
Loading...