See the published version of this preprint at BMC Medical Research Methodology.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
Research article
Jenna Marie Reps
Janssen Research and Development Titusville
Corresponding Author
ORCiD: https://orcid.org/0000-0002-2970-0778
License:
This work is licensed under a CC BY 4.0 License. Read Full License
Background To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to scale-up external validation of patient-level prediction models by enabling validation across a large number of heterogeneous observational healthcare datasets.
Methods Five previously published prognostic models (ATRIA, CHADS2, CHADS2VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks. A network study was run that enabled the five models to be externally validated across nine observational healthcare datasets spanning three countries and five independent sites.
Results The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and they obtained mean c-statistics ranging between 0.57-0.63 across the 6 databases with sufficient data to predict stroke within 1 year of initial atrial fibrillation diagnosis for females with atrial fibrillation. This was comparable with existing validation studies. The validation network study was run across nine datasets within 60 days once the models were replicated. An R package for the study was published at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation .
Conclusion This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and a data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability or reproducibility of a prediction model, but without collaborative approaches it can take three or more years for a model to be validated by one independent researcher. In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months. We recommend that researchers developing new prediction models use the OHDSI network to externally validate their models.
Keywords
patient-level prediction; prognostic model; external validation; transportability; collaborative network
This is a list of supplementary files associated with this preprint. Click to download.
Loading...
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Research Methodology.
Version 4
Posted 06 May, 2020
No community comments so far
Editorial decision: Accept
On 23 Apr, 2020
Submission checks complete
On 17 Jul, 2019
No comments provided
Editorial decision: Minor revision
On 22 Apr, 2020
Editor assigned
On 20 Apr, 2020
Submission checks complete
On 19 Apr, 2020
Editor invited
On 19 Apr, 2020
No comments provided
Editorial decision: Minor revision
On 01 Apr, 2020
Review #1 received
Received 31 Mar, 2020
Reviewer #1 agreed
On 27 Mar, 2020
Reviewers invited
Invitations sent on 23 Jan, 2020
Editor assigned
On 20 Jan, 2020
Submission checks complete
On 19 Jan, 2020
Editor invited
On 19 Jan, 2020
No comments provided
Editorial decision: Major revision
On 19 Dec, 2019
Review #2 received
Received 15 Dec, 2019
Reviewer #2 agreed
On 26 Nov, 2019
Review #1 received
Received 09 Nov, 2019
Reviewer #1 agreed
On 25 Oct, 2019
Reviewers invited
Invitations sent on 28 Sep, 2019
Submission checks complete
On 17 Jul, 2019
Editor assigned
On 10 Jul, 2019
First submitted
On 09 Jul, 2019
Editor invited
On 09 Jul, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Health Economics & Outcomes Research
Learn more about our company.
A new preprint design is coming!
We have improved readability, clarity, accessibility, and opportunities for engagement.
See the published version of this preprint at BMC Medical Research Methodology.
This is a preprint, a preliminary version of a manuscript that has not completed peer review at a journal. Research Square does not conduct peer review prior to posting preprints. The posting of a preprint on this server should not be interpreted as an endorsement of its validity or suitability for dissemination as established information or for guiding clinical practice.
Learn more about In Review
Research article
Jenna Marie Reps
Janssen Research and Development Titusville
Corresponding Author
ORCiD: https://orcid.org/0000-0002-2970-0778
Badges
Prescreen
This manuscript passed our Prescreen assessment
Complete author information
Appropriate declaration statements
Potential risks to human health
Prescreen
Peer Review Timeline
CURRENT STATUS: Published
View the published article at BMC Medical Research Methodology.
Version 4
Posted 06 May, 2020
No community comments so far
Editorial decision: Accept
On 23 Apr, 2020
Submission checks complete
On 17 Jul, 2019
No comments provided
Editorial decision: Minor revision
On 22 Apr, 2020
Editor assigned
On 20 Apr, 2020
Submission checks complete
On 19 Apr, 2020
Editor invited
On 19 Apr, 2020
No comments provided
Editorial decision: Minor revision
On 01 Apr, 2020
Review #1 received
Received 31 Mar, 2020
Reviewer #1 agreed
On 27 Mar, 2020
Reviewers invited
Invitations sent on 23 Jan, 2020
Editor assigned
On 20 Jan, 2020
Submission checks complete
On 19 Jan, 2020
Editor invited
On 19 Jan, 2020
No comments provided
Editorial decision: Major revision
On 19 Dec, 2019
Review #2 received
Received 15 Dec, 2019
Reviewer #2 agreed
On 26 Nov, 2019
Review #1 received
Received 09 Nov, 2019
Reviewer #1 agreed
On 25 Oct, 2019
Reviewers invited
Invitations sent on 28 Sep, 2019
Submission checks complete
On 17 Jul, 2019
Editor assigned
On 10 Jul, 2019
First submitted
On 09 Jul, 2019
Editor invited
On 09 Jul, 2019
Metrics
Comments: 0
PDF Downloads: ...
HTML Views: ...
Subject Areas
Health Economics & Outcomes Research
License:
This work is licensed under a CC BY 4.0 License. Read Full License
View the published article at BMC Medical Research Methodology.
Background To demonstrate how the Observational Healthcare Data Science and Informatics (OHDSI) collaborative network and standardization can be utilized to scale-up external validation of patient-level prediction models by enabling validation across a large number of heterogeneous observational healthcare datasets.
Methods Five previously published prognostic models (ATRIA, CHADS2, CHADS2VASC, Q-Stroke and Framingham) that predict future risk of stroke in patients with atrial fibrillation were replicated using the OHDSI frameworks. A network study was run that enabled the five models to be externally validated across nine observational healthcare datasets spanning three countries and five independent sites.
Results The five existing models were able to be integrated into the OHDSI framework for patient-level prediction and they obtained mean c-statistics ranging between 0.57-0.63 across the 6 databases with sufficient data to predict stroke within 1 year of initial atrial fibrillation diagnosis for females with atrial fibrillation. This was comparable with existing validation studies. The validation network study was run across nine datasets within 60 days once the models were replicated. An R package for the study was published at https://github.com/OHDSI/StudyProtocolSandbox/tree/master/ExistingStrokeRiskExternalValidation .
Conclusion This study demonstrates the ability to scale up external validation of patient-level prediction models using a collaboration of researchers and a data standardization that enable models to be readily shared across data sites. External validation is necessary to understand the transportability or reproducibility of a prediction model, but without collaborative approaches it can take three or more years for a model to be validated by one independent researcher. In this paper we show it is possible to both scale-up and speed-up external validation by showing how validation can be done across multiple databases in less than 2 months. We recommend that researchers developing new prediction models use the OHDSI network to externally validate their models.
This is a list of supplementary files associated with this preprint. Click to download.
Loading...