Developing GoodReports.org
The GoodReports website has two main features: authors can complete a questionnaire about their study to receive a reporting guideline suggestion, then can immediately access reporting checklists to fill out on- or offline. Each checklist includes clear instructions, and each reporting item is linked to an explanation of why that item is needed and examples of good reporting, whenever guideline developers have provided such information.
We decided to include reporting guidelines that cover the main generic study designs and are commonly recommended by journals. We started with the 13 popular reporting guidelines highlighted on the EQUATOR homepage. Although published as one reporting guideline, STROBE covers three observational study designs and is made available by the STROBE development group as three separate checklists. We included all three. We added STREGA for genetic association studies (26) as it is included in the BMJ Open guide for authors. The 16 reporting guidelines included in GoodReports are shown in Table 1.
Table 1: List of reporting guidelines in the GoodReports database (15-28)
|
Name
|
Study type
|
1
|
ARRIVE
|
Laboratory animals*
|
2
|
CARE
|
Case reports
|
3
|
CHEERS
|
Economic evaluations
|
4
|
CONSORT
|
Randomised trials*
|
5
|
MOOSE
|
Meta-analyses of observational studies
|
6
|
PRISMA
|
Systematic reviews and meta-analyses*
|
7
|
PRISMA-P
|
Protocols of systematic reviews
|
8
|
SPIRIT
|
Protocols of randomised trials*
|
9
|
SQUIRE
|
Quality improvement studies
|
10
|
SRQR
|
Qualitative studies
|
11
|
STARD
|
Diagnostic test accuracy*
|
12
|
STREGA
|
Genetic association studies
|
13
|
STROBE case control
|
Case-control studies*
|
14
|
STROBE cohort
|
Cohort studies*
|
15
|
STROBE cross sectional
|
Cross-sectional studies*
|
16
|
TRIPOD
|
Prognostic studies
|
* GoodReports entry includes link to explanation and examples
|
We adapted the original questionnaire to our new set of guidelines (Figure 2).
We reduced the use of research and methods jargon as much as possible to improve accessibility and clarified some of the questions in response to initial user feedback. We used multiple-choice options to keep the decision tree short and easy to navigate. The publishers’ copyright licence for MOOSE (19) and SRQR (24) did not automatically allow us to reuse the content to create an openly accessible online checklist. We were granted permission to do so on payment of a licence fee for one and two years respectively, which covered the duration of this study.
Reaching users
GoodReports.org went live in January 2018. We used Penelope.ai (14), a company owned by co-author JH, to attract users. Penelope.ai provides software to journals that automatically checks new submissions and gives immediate feedback to authors to help them meet journal requirements. Penelope.ai allowed us to collaborate with BMJ Open, one of Penelope.ai’s customers, to capture authors in the process of submitting their articles for publication. All authors submitting to BMJ Open can opt to use Penelope.ai, but it is not mandatory.
From January 2018, the Penelope.ai upload form was amended to include the GoodReports guideline recommendation questionnaire (Figure 2). Authors who used Penelope.ai therefore had to answer the checklist finder questionnaire before uploading their manuscript. When appropriate, Penelope.ai’s feedback report on their manuscript included a recommendation to use a reporting guideline and a link to the associated checklist on GoodReports.org.
Below is a representative author journey from one of Penelope.ai’s client journals, BMJ Open.
- Authors begin their submission on BMJ Open: https://bmjopen.bmj.com/pages/authors/
- Authors receive an option for an automated manuscript check: https://app.penelope.ai/manuscript-check/q/bmjopen
- Authors that opt for an automated check answer a few questions about their work and view their feedback online: https://app.penelope.ai/submissions/demo/
- Depending on the information authors give when uploading to Penelope.ai, their feedback may include an instruction to complete a reporting guideline on https://www.goodreports.org
1. Individual user feedback
From 25 January 2018 to 6 November 2019, all Penelope.ai users who received a reporting checklist recommendation as part of their manuscript report were sent an automated email survey a day later about their experience of using GoodReports.org. The sample size was determined by the number of users within that timeframe. We used Typeform.com (40) to collect the survey data, with the first question embedded in the email. We asked users:
1. "You were recently advised to complete a checklist at www.goodreports.org. How useful did you find the checklist?" (rating scale: 0 (least useful) to 10)
1. If a rating of 7 or lower: "Can you explain why you gave a rating of [number]?" (multiple choice. We set 7 as the cut-off for this question as we anticipated that our median rating would be 8/10)
- The checklist items were not relevant to my work
- The checklist was too long
- The checklist was confusing
- The website was confusing
- Other (free text)
2. "How could we make www.goodreports.org more useful?" (free text)
3. From December 2018: “After using the checklist did you make any changes to your manuscript?” (yes/no)
- If yes, “What did you change?” (free text)
- If no, “Why didn't you make any changes?” (free text)
Quantitative responses are reported as counts. The free-text responses for question 3.1 and 3.2 were read and discussed by authors CS and JH. JH then extracted general themes. We report the themes, their frequency, and representative quotes.
2. Reliability of the questionnaire in helping authors find the most appropriate reporting guideline for their work
To determine whether the checklist finder questionnaire generally led users to an appropriate checklist, we selected 100 manuscripts from all of those uploaded to Penelope.ai by BMJ Open authors between 25 January 2018 and 16 February 2019. Manuscripts were randomly selected from Penelope.ai’s database using Python’s randint function. The sample size was determined by the amount of time available for the two investigators to assess the sample of manuscripts.
CS and JH separately read the titles, abstracts, and methods section of each manuscript and decided which, if any, of the 16 guidelines in the GoodReports database should have been recommended. They compared recommendations and resolved conflicts through discussion. They were blinded to the checklist finder recommendation up to this point. The final assessor recommendation was then compared with the recommendation that authors received from the checklist finder questionnaire. Where the recommendations differed, the assessors examined the questionnaire responses and the corresponding manuscripts together and judged the most likely reason for the discrepancy.
We report the percentage of manuscripts where the checklist finder questionnaire and assessor recommendations matched and possible reasons for mismatches. No statistical tests were done, as the purpose was not to prove that the checklist finder questions lead to appropriate checklists, but to identify how the questionnaire could be improved before a more rigorous evaluation.
Use of GoodReports, checklist submission rates, and manuscript completeness
BMJ Open is an existing customer of Penelope.ai. The Editor-in-Chief agreed to allow us access to submitted manuscripts to gather initial data on whether directing users to GoodReports at the point of submission was useful and for use in guiding further development.
We used a sample of BMJ Open submissions to observe how exposure to GoodReports correlated with submission quality. We were interested in whether authors included reporting checklists in their submission and whether using a reporting checklist led authors to add missing information to their manuscripts.
We collected data from all newly submitted manuscripts checked by BMJ Open staff on 9, 10, 11, 25, 28, and 29 May 2018. These dates were selected by the journal and determined the sample size. It was not practical for the journal to increase the length of the data collection period. We only included manuscripts checked for the first time on these dates. We excluded manuscripts that had been first submitted outside the recording window, returned to the author for corrections, and resubmitted within the recording window.
3. Exposure to GoodReports.org and rates of submission of a completed reporting checklist
BMJ Open shared some of the data they collect in their normal day-to-day activity with us, such as whether the submission had previously been checked and notes from the technical editor about unmet journal requirements. These data did not specifically include whether the author had included a reporting checklist in their submission. However, the journal enforces checklists, and we could see the editor’s notes. We were therefore able to count the number of “checklists noted to be missing.”
We split submissions into two groups, those whose authors had opted to check their manuscript with Penelope.ai before submission and received a checklist recommendation, when appropriate, and those whose authors had opted not to use the checker. JH identified whether an author had used Penelope.ai by searching the Penelope.ai logs for the author’s email address and cross-referencing the manuscript’s file names and titles, without knowing whether a reporting guideline had been submitted for that manuscript.
We report the proportion of manuscripts where a checklist had been flagged as missing for each group.
4. Completeness of reporting before and after using a GoodReports reporting checklist
We observed whether authors that used and submitted a reporting guideline checklist from GoodReports.org changed their manuscript and improved the completeness of their reporting as a result.
We started with the subset of manuscripts from the study on submission rates that had:
1) been checked by Penelope.ai before submission to BMJ Open,
2) not withdrawn their submission from BMJ Open, and
3) included a reporting guideline checklist from GoodReports.org.
We conducted a before-and-after study on the included manuscripts. The version that was submitted to Penelope.ai for an automatic pre-submission check (the “before” version) was compared to the version subsequently submitted to BMJ Open (the “after” version).
Manuscripts submitted with checklists obtained elsewhere, such as the EQUATOR Network website or the journal website, were excluded. We wanted to reduce the chance that the authors of manuscripts in our “before” group had used a checklist before visiting GoodReports.org.
JH redacted the title and methods sections of the “before” and “after” versions so that no personal information was shared with assessors. The “before” versions were all in .docx format, so text could be copied and pasted into a fresh Microsoft Word file. The “after” versions were PDFs as BMJ Open automatically converts submissions into PDF and adds watermarks, line numbers, and footers. JH split PDF files into smaller files containing only the title and methods sections for data extraction. These differing file formats meant that assessors could not be blinded as to whether the manuscript was the “before” or “after” version.
Five assessors (JdB, MS, PD, PL, and AK) were allocated a selection of manuscript pairs and assessed the methods sections of the “before” and “after” versions. Each manuscript pair was assessed by three data extractors CS assessed the titles of all 20 manuscripts.
The assessors checked whether the “before” version submitted to Penelope.ai contained adequate information for each item in the methods section of the appropriate reporting checklist. Each item was assessed as present, absent, unclear/partial, or not applicable to that manuscript.
The assessors then checked whether the “after” version submitted to BMJ Open contained the same information as the “before” version or whether the author had added information. Each item was assessed as no change or added information.
CS collated and harmonised the data from the five assessors. In the “before” manuscripts, each item was given an overall code. Items were coded “1” if the item was partially reported or missing and “0” if there was adequate information:
- If two of the three assessments agreed, the majority assessment dictated the overall code.
- If the assessors disagreed and one or two assessors selected “not applicable,” the item was coded “0” indicating there was adequate information for that item.
- If the assessors each gave a different assessment and none selected “not applicable,” the item was coded “1” to indicate information was missing.
In the “after” manuscripts, each item given a rating of “1” (information missing) in the “before” version was given an overall code of “1” if information had been added or “0” if there was no change according to the majority assessment.
For each manuscript pair, we counted the number of items reported adequately in the “before” version and the number of items with more information in the “after” version. As each reporting guideline has a different number of items, we report these counts as percentages.
Ethics and consent
In accordance with the University of Oxford’s policy on the ethical conduct of research involving human participants and personal data (41), ethical approval and informed consent were not required. We used data collected as part of our partner PNLP Ltd.’s optional manuscript checking service, and during the normal course of BMJ Open’s editorial procedures. In accordance with the personal data protection policies of our partners, all data was anonymised before it was shared with the research team.