Establishing Patient Perceptions and Preferences for a Journal Authenticator Tool To Support Health Literacy: A Mixed-Methods Survey and Focus Group Study

doi:10.21203/rs.3.rs-875992/v1

Download PDF

Research article

Establishing Patient Perceptions and Preferences for a Journal Authenticator Tool To Support Health Literacy: A Mixed-Methods Survey and Focus Group Study

https://doi.org/10.21203/rs.3.rs-875992/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Background

Predatory journals are illegitimate journals that do not meet expected publication best practices. Many of these journals can be found using Google, making them readily available to patients searching online for health information. The goal of this study was to obtain information about how patients use the internet to get health information and to determine patient preferences and needs for a journal authenticator tool which would highlight journal transparency practices.

Methods

We conducted a cross-sectional survey of Canadian-based adult patients and caregivers and then a series of online focus groups to further explore the survey responses. Descriptive statistics (counts and percentages) were reported for all quantitative survey items. For text-based responses, we used thematic content analysis. Online focus groups asked patients about the content they would like to see in a journal authenticator tool, how they would like the content visually displayed, how to best share the tool with patients, and how to determine whether the tool was successful over time. Thematic content analysis was conducted to identify core themes discussed. Focus group participants completed a follow-up survey in which they rank ordered the themes identified by perceived importance.

Results

183 participants completed our online survey. A total of 146 (82%) participants indicated they use the internet most often when looking for health information. Sixty-six (37%) indicated they sometimes read original research articles when searching for health information and 92 (52%) participants indicated they sometimes have difficulty knowing if the information they read online is reliable. Eighty-six (49%) participants had never heard of predatory journals. Thirty-nine survey participants indicated their willingness to contribute to subsequent focus groups and a total of 29 participated. Four key topic areas were discussed and 32 themes were identified.

Conclusions

Our findings suggest that patients have expressed a need for a journal authenticator tool and that this tool may provide value to them. The results from this study will help inform the tool’s development to help ensure that it meets the needs of patients.

Trial registration:

This mixed-methods study has been registered on the Open Science Framework: https://osf.io/56ead/

Epidemiology

Health Economics & Outcomes Research

Health Policy

digital health

patient partner

predatory journal

transparency

health literacy

patient engagement

Publishing results from research studies in academic journals is one of the most important ways that medical knowledge is shared. Fake academic journals called ‘predatory journals’ are disrupting this method of sharing knowledge. Predatory journals can be found by patients and the public using a simple Google search. The goal of this study was to understand how patients use the internet to find health information and to understand what patients need in a digital tool to identify predatory journals. This digital tool will help patients when searching for accurate health information from reliable sources.

In the first part of our study, adult Canadian patients and caregivers completed a survey. Many (82%) said they use the internet most often when looking for health information. Some (37%) said they sometimes read academic research articles when searching for health information. Over half (52%) of participants said they sometimes have difficulty knowing if online information is reliable. The second part of our study focused on what patients need in a digital tool to identify predatory journals. The key features of a digital tool include displaying journal features that show credibility, journal features that show lack of credibility, and the institutions associated with the work. Patient partners were involved from the beginning of the study and co-designed the study materials (refer to GRIPP2 checklist).

Our work suggests that a digital tool can help patients and the public in finding and understanding health information and predatory journals. The study results will be used to develop digital tools for patients and public.

Scientific articles in peer-reviewed journals play a major role in disseminating biomedical research. For scientific articles to be useful, they need to be reported and disseminated in a way that meets certain best practice standards. For example, published research ought to be properly indexed and archived - this ensures research is retrievable and that it exists as part of a permanent record over time. There have been several calls to increase the transparency of academic journals, including opening up the peer-review process and making study data publicly available. (1, 2) The impetus for journals to enhance transparency and best practices may be, in part, related to the growing problem caused by predatory journals. (3–5)

Predatory journals are “entities that value self-interest at the expense of scholarship, are characterized by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices” (3–5). Predatory journals do not meet expected best practice standards of scientific publishing, and many appear to exist for the sole purpose of collecting article processing charges commonly used in the open access publishing model. For instance, many predatory journals have published work that has not undergone peer review (6), or been checked for plagiarism (7). Although the effect of predatory journals on researchers has been documented (e.g., difficulties in discriminating between predatory journals and legitimate journals) (8, 9), these journals also have potential effects on patients. As per CIHR, “patient” refers to “individuals with personal experience of a health issue and informal caregivers, including family and friends” (10). For instance, as the majority of Canadians search for health information online (11), usually before visiting their doctor (12), it is possible, and increasingly likely, that they may encounter articles from predatory journals. Consequently, patients may make health decisions based on information published in these illegitimate journals. Indeed, we have previously written about interactions that family members of our investigative team have had with predatory journals. (3, 13)

To help address this problem, an international group that included patient partners, created the consensus definition of predatory journals and agreed on several next steps. This included the creation of a one-stop-shop of resources, as well as the creation of a ‘digital journal authenticator’ tool that could be used to capture and visualize the transparency practices of academic journals. The proposed tool could support patients and other stakeholders to identify journals that do and do not align with best practices pertaining to transparency. This tool could support patients as empowered shared decision makers in their healthcare. We aimed to create a tool that is automated, meaning it would not require assessors to manually evaluate the journal. The journal authenticator tool would support users in decisions on whether to interact with the journal or not (i.e., read it, submit to it, or cite work published there), which may help reduce interaction with predatory journals and foster interaction with journals meeting transparency standards.

To develop the tool, we are employing a user-centered design that is intended to enhance the experience of people using a product, software, or service and ensure the tool meets the needs and preferences of the eventual users (14). Here we report the first of a series of research studies to identify user preferences, needs and requirements. The aim of this study was to describe how patients use the internet to obtain health information and determine their needs and preferences for the proposed digital journal authenticator tool. To do so, we conducted two studies: 1) a cross-sectional survey that obtained information on how patients in Canada use the internet to obtain health information; and 2) online focus groups to help us determine if patients in Canada would find a journal authenticator tool useful, and if so, how to design a tool that would be most relevant to them. This research is descriptive, we had no a priori hypotheses.

Patient and public involvement

This study was co-designed with two patient partners who provided important patient perspectives throughout this project (LP and MH). Each patient partner was involved in developing the funding proposal, drafting the protocols, creating and providing feedback on patient materials for the project, attending a focus group, aiding in recruitment, and revising the manuscript. This paper adheres to the GRIPP2 checklist for patient and public involvement (see Additional file 1) for completed checklist).

Study 1 – Cross-sectional Survey

The CHERRIES (15) guideline for e-surveys were used to guide reporting of our findings.

Study design

We conducted an online, cross-sectional survey of patients. The survey was completed voluntarily and anonymously. The survey opened on January 20, 2021 and closed three weeks later.

Identifying participants

To assess patients’ use of the internet to obtain health information, we recruited individuals 18 years of age or older, residing in Canada, that were fluent in English to take part in our online survey. Participants were recruited via three different methods simultaneously. A recruitment poster (see https://osf.io/7892c/) was sent out by Canada Health Infoway (an independent, federally funded, not-for-profit organization) to their patient database via their Interchange newsletter and through e-mail (16). The recruitment poster was also distributed by e-mail via The Ottawa Hospital’s Patient Services Team (17) who have a database of patients interested in partnering with researchers. We also recruited participants online through Twitter via the Ottawa Hospital Research Institute’s Centre for Journalology account (18) and through members of the research team including patient partners.

Survey administration

A link to the survey was included in the recruitment poster. Potential participants who had been e-mailed directly were given three weeks to complete the survey with reminders sent via the databases after one and two weeks from the initial invite.

Survey

Our survey was an open survey and purpose-built, administered in English, and distributed using SurveyMonkey (19). Before participants began the online survey, they were presented with a consent form. Completion of the online survey was considered as implied consent. The first part of the survey contained 14 demographic items (e.g., gender, age, location in Canada, education level) while the second part of the survey contained 11 items that focused on digital health literacy (20) (e.g., if and when patients use the internet to search for health information, if they have trouble discerning the reliability of the information they are reading, and if they have heard of predatory journals). The survey was pilot tested by two patient partners for usability and technical functionality; their feedback was incorporated into the final version. The survey contained both quantitative and qualitative (free-text) questions. For the complete survey please see https://osf.io/b6kmq/ . After participants completed the survey, they had the option to enter their e-mail address for a chance to win a $100 Amazon gift card. They were also asked if they were interested in participating in a related focus group study. E-mail addresses for both options were collected separately from the response data. This method allowed participant contact details to be stored independently from the survey responses, ensuring anonymity.

Analyses

We report the overall sample size for each item and descriptive statistics are reported using counts and percentages (Excel (Version 16), Microsoft Corporation, Redmond, USA). For qualitative text-based items, we used thematic content analysis which is a way to identify, analyze and report patterns or ‘themes’ within data (21). Two researchers (AR, KDC) independently reviewed all responses to each written response question. They independently charted the data to identify key themes and then discussed the themes iteratively until they reached agreement on the themes and subthemes for each question. Key themes are summarized and presented using a narrative synthesis with illustrative quotes and count data where relevant.

Study 2 – A Focus group study

Study design

We conducted ten virtual focus groups with patients.

Identifying participants

Survey participants interested in participating in focus groups provided their e-mail at the end of the survey. A prescreening survey (https://osf.io/7892c/) was sent to all those who expressed interest in participating. The pre-screening survey was a closed survey that was purpose-built, administered in English, and distributed using SurveyMonkey (19). Before participants began the online survey, they were presented with a consent form where completion of the online survey was considered consent. The survey contained 12 demographic items (i.e., gender, age, location in Canada, education level). The last item asked for the participant’s e-mail address so they could be contacted about the focus group.

Focus groups

Before the focus groups commenced, we ran a pilot session with two patients not otherwise involved in the study to test the functionality and usability of the focus group questions from the focus group guide (https://osf.io/8pgnj/) and determine the length of the session. Feedback from pilot participants was then incorporated into our approach.

Each participant was sent an information booklet (https://osf.io/mjs7d/) containing details about the focus groups (e.g., the purpose of the project, how the focus groups will work, background information, including videos, on the research and publication process, and predatory journals, and ground rules for effective focus group interaction). The ten focus groups were conducted online using Zoom videoconferencing software (22) and moderated by an experienced qualitative researcher (GC) and supported by a research assistant (AR). Consent was confirmed verbally at the start of each focus group. Total time contribution from each participant to review the booklet and participate in the online session was approximately two hours and compensation was provided in alignment with SPOR guidelines.

The focus group guide addressed four key topic. The topics were: 1) the content patients want included in a journal authenticator tool, 2) how the information would be visually displayed, 3) how best to share the tool within the patient community, and 4) how to track the success of the tool over time. Sessions were conducted using the nominal group technique which allowed all participants an equal chance to contribute ideas (23). Briefly, when one question was asked, the moderator allowed each participant to share their ideas, one at a time. This process was continued until there were no new responses to the question. The moderator then asked participants if there was anything else they wanted to add. The moderator took notes of participant’s statements during the focus group. These notes were shared simultaneously with participants to ensure statements were recorded and accurate.

Analyses

Notes that were taken by the moderator (GC) during each focus section were used for thematic content analysis (24) using Excel. Each of the key topic areas was analyzed independently. First, statements from participants from the ten focus group notes were coded and combined and uploaded into Excel. Codes were then assessed for inclusion based on whether or not they directly answered the discussion point (EA, AR, KC). This was done independently by one researcher with verification by two others to ensure statements were appropriate for the theme. Next, codes within each discussion point were sorted into a table and from these, themes were identified independently by two team members (EA, AR). The themes established were then discussed by team members (KC, AR, EA) and iteratively modified to reflect consensus.

Post-focus group survey

After all focus groups were completed, a follow-up survey was sent to all focus group participants. The survey was purpose-built, administered in English, and distributed using SurveyMonkey (19). The survey presented the four key topic areas discussed during the focus group along with the themes that emerged from our thematic content analysis in each area. Participants were asked to rank order the items on a scale of 1 to X, where “1” was the most important. Rank score was reported for each item (i.e., which is the weight average for each option where the top rank option is always weighed the highest and the bottom rank is weighed the lowest).

After participants completed the focus group and the survey, they were offered $100 in compensation for their time (including preparation time). Participants who did not want to be compensated had the option of donating the money to a registered Canadian health charity of their choice.

Study 1 – Cross-sectional survey

Participants

A hundred and eighty-three participants responded to the survey. Most respondents were female (N = 134, 73%). The largest proportion of participants reported to be in the 60 + range (N = 56, 31%). The majority of respondents were residents of Ontario (N = 132, 74%). Eighty-nine (48.6%) have previously participated in a health research project. Sixty-nine (38%) participants have worked in healthcare, and 48 (26%) have been a patient partner or advisor. Full demographics are summarized in Table 1.

Table 1

Participant characteristics
	N (%)
Gender	N = 183
Male	45 (24.6)
Female	134 (73.2)
Diverse	2 (1.1)
Prefer not to say	2 (1.1)
Age	N = 183
18–29	18 (9.8)
30–39	34 (18.6)
40–49	34 (18.6)
50–59	41 (22.4)
60+	56 (30.6)
Area of Canada	N = 179
British Colombia	15 (8.4)
Ontario	132 (73.7)
Quebec	8 (4.5)
Other	24 (13.4)
Rural community	N = 183
Yes	44 (24)
No	139 (76)
Highest level of education	N = 182
< High school	6 (3.3)
High school	13 (7.1)
College	37 (20.3)
Undergraduate degree	53 (29.1)
Graduate degree	67 (36.8)
Other	6 (3.3)
Identify as indigenous	N = 180
Yes	11 (6)
No	169 (92.4)
Prefer not to say	3 (1.6)
Identify as person with disability	N = 181
Yes	34 (18.6)
No	147 (80.3)
Prefer not to say	2 (1.1)
Identify as a member of an ethnic minority	N = 177
Yes	21 (11.5)
No	156 (85.3)
Prefer not to say	6 (3.3)
Patient or caregiver	N = 168
Patient	80 (47.6)
Caregiver	48 (28.6)
Both	51 (30.4)
Experience as a researcher	N = 183
Yes	52 (28.4)
No	131 (71.6)
Worked professionally in health care	N = 183
Yes	69 (37.7)
No	114 (62.3)
Participant in health research project	N = 183
Yes	89 (48.6)
No	94 (51.4)
Experience as patient advisor or partner?	N = 183
Yes	48 (26.2)
No	135 (73.8)

Digital health literacy data

The top five resources participants indicated they use when looking for health information were health professionals (N = 155, 87%), the internet (N = 146, 82%), academic articles (N = 108, 61%), books (N = 44, 25%), and friends or family (N = 57, 32%). Over one-third of respondents look up health information a few times a month (N = 59, 33%). Most participants indicated that they look for health information online after becoming concerned about a health issue (N = 146, 82%)

Reading health information online led 120 participants (68%) to book an appointment in at least one instance with their health care provider, 127 participants (72%) to make changes to their lifestyle and behaviour, and 60 participants (34%) to alter a medical treatment plan. Ten participants (6%) reported experiencing a health problem by making a change to their lifestyle/treatment as a result of reading health information online. When using the internet, most participants report using Google (N = 136, 76%), health information websites (e.g. MayoClinic or WebMD (N = 158, 88%)), and health charity or non-profit websites (e.g. Canadian Cancer Society) (N = 125, 70%).

Twenty-five respondents (14%) indicated they always read original research articles when searching for health information, 44 respondents (25%) usually read them, while the largest proportion of our participants (N = 66, 37%) indicated they sometimes read research articles. More than half of respondents (N = 92, 52%) indicated that they have difficulty knowing if the information they are reading is reliable. Eighty-six participants (49%) have never heard of a ‘predatory journal’. Table 2 provides full details on responses to health literacy items.

Table 2

Health literacy questions
	N(%)
When you have a health question, what resources do you use most often to obtain information?ⁱ	N = 178
Academic articles (i.e. research published in an academic journal)	108 (60.7)
Books (electronic or print)	44 (24.7)
Friends or family	57 (32)
Health professionals	155 (87.1)
Mainstream media (online or print) (e.g. TV, newspaper, radio, magazines)	21 (11.8)
Social media	24 (13.5)
Internet, excluding social media	146 (82)
Other (please specify)	17 (9.6)
How often do you use the internet to look up health information?	N = 177
Daily	23 (13)
A few times a week	39 (22)
A few times a month	59 (33.3)
Every few months	30 (17)
A few times a year	26 (14.7)
When do you use the Internet to find health information?ⁱ	N = 179
Before speaking with a health care provider	128 (71.5)
After speaking with a health care provider	113 (63.1)
After I become concerned about a health issue	146 (81.6)
After viewing my electronic medical record	51 (28.5)
Other (please specify)	15 (8.4)
Has reading health information you obtained online ever led you toⁱ:	N = 179
	Yes	No
Make an appointment with your health care provider	120 (67.8)	57 (32.2)
Make a change to your behaviours or lifestyle	127 (71.8)	50 (28.3)
Make a change to a medical treatment plan	60 (34.3)	115 (65.7)
	N = 179
Have you ever experienced a health problem or complication by making a change in behaviour/lifestyle/treatment plan as a result of reading health information online?	Yes	No	Unsure
	10 (5.6)	148 (82.7)	21 (11.7)
When you use the internet to search for health information, which of the following do you useⁱ	N = 179
Google	136 (76)
Wikipedia	42 (23.5)
Health information website (e.g. Mayo Clinic, WebMD)	158 (88.3)
Health charity or not-for-profit website (e.g. Canadian Cancer Society, The Arthritis Society)	125 (69.8)
Google Scholar	59 (33)
Blogs	14 (7.8)
Online discussion forums	33 (18.4)
Other (please specify)	35 (19.6)
How often do you read original research articles (i.e., publications produced by researchers in academic/scholarly journals) when searching for health information?	N = 179
Always	25 (14)
Usually	44 (24.6)
Sometimes	66 (36.9)
Rarely	36 (20.1)
Never	8 (4.5)
If so, are they helpful?	N = 169
Always	12 (7.1)
Usually	69 (40.8)
Sometimes	78 (46.2)
Rarely	9 (5.3)
Never	1 (0.6)
Do you find it difficult to know if the health information you are reading online is based on reliable research evidence?	N = 177
Always	10 (5.7)
Usually	32 (18.1)
Sometimes	92 (52)
Rarely	40 (22.6)
Never	3 (1.7)
Have you ever heard of a 'predatory journal'?	N = 176
Yes	78 (44.3)
No	86 (48.9)
Unsure	12 (6.8)
How did you first learn about predatory journals?	N = 176
From this research project	91 (51.7)
Other (please specify)	85 (48.3)
i. Participants were able to select all responses that applied and therefore N may be greater than sample size

Determining accuracy of health information found online

Table 3 (see additional file 2) provides a summary of thematic groups for how patients determine the accuracy of health information online. When asked, via free-text question, how they determine whether the health information found on the internet is accurate, a thematic analysis revealed that participants noted they 1) check the integrity of the piece (e.g., check references, check for conflicts of interest); 2) they check other sources of information (e.g., check with health professionals); 3) they use trusted sources like hospital and government websites; and 4) do not check validity and/or determine accuracy through personal experience.

Table 3

Responses to items about predatory journals and health complications experienced by some of our participants
Item	Themes	Codes	N (%)	Examples
How do you determine whether the health information found on the Internet is accurate?	Check the integrity of the piece	Check authors	14 (8.2)	"By looking at who published the information…"
		Check for biases	3 (1.8)	"…presence of bias"
		Check for conflicts of interest	3 (1.8)	"…they have no perceived conflicts of interest"
		Check references	11 (6.4)	"Look for references and go to actual reference if available."
		Check research methods	7 (4.1)	"...also look at studies to see methodology…"
		Check the publication date	5 (2.9)	"…Year of publication…"
		Validate the source	39 (22.8)	"...check the source to ensure that it appears to be reputable…"
		Check that work is peer-reviewed	14 (8.2)	"I look for peer reviewed sources…"
	Check other sources of information	Check with experts	7 (4.1)	"...ask experts for their opinion about it"
		Check with health professionals	24 (14.0)	"…then verify with my medical professionals"
		Cross check with other sources	37 (21.6)	"Looking at various sites and comparing information."
	Use trusted sources	Use government sites	6 (3.5)	"I tend to trust government websites more than any other. "
		Use reliable sources	74 (43.3)	"I usually look for verified sites (GOvt Canada CDC, major hospital…"
		Use hospital websites	4 (2.3)	"... I prefer reputable sources like hospitals and professional associations for factual info…"
		Use Mayo Clinic	11 (6.4)	"I visit reputable sites like the Mayo Clinic"
		Use nonprofit websites	7 (4.1)	"... from a creditable site such as cancer care Ontario or heart and lung associations."
		Use PubMed	3 (1.8)	"My first choice is PubMed then the appropriate association (e.g. Canadian Cardiovascular Society) "
		Use WebMD	3 (1.8)	"Look at the source–WebMD or Mayo Clinic are my go to…"
	Other	Other	34 (19.9)	"check the url to see if it looks legit…"
		Don't check validity	3 (1.8)	"I don't, I take it with a grain of salt"
		Personal experience	4 (2.3)	"...if someone's experience closely resembles mine"
	Total responses		N = 171

Learning about predatory journals

When respondents were asked ‘how did you first learn about predatory journals’, 91 respondents (52%) stated they heard the term from this survey while others have heard about them elsewhere (n = 85, 48%). A thematic analysis of open ended questions revealed that participants heard about predatory journals 1) through work and research experience (e.g., employment) and had received invitations from predatory journals, 2) through other individuals such as research experts or friends, 3) through resources like the media or internet, and 4) some participants reported they could not remember where they first heard the term. Table 4 (see additional file 3) provides a summary of responses to the items about predatory journals and a summary of health complications experienced by some of our participants who made a decision based on reading health information online.

Table 4. Responses to items about predatory journals and health complications experienced by some of our participants

Item	Themes	Codes	N (%)	Examples
How did you first learn about predatory journals?	Work and research experience	Via my employment	15 (17.6)	"Aware of them as a published author, and researcher"
		From research training experience	18 (21.2)	"Previous experience as a researcher "
		Have received e-mail invitations from predatory journals	4 (4.7)	"I receive invitations from predatory journals on a weekly basis."
	Other individuals	Friends	7 (8.2)	"In conversation with peers"
		From a health professional	1 (1.2)	"...in discussions with my doctors over the years in regards to research…"
		From a research expert	5 (5.9)	"From health scientists"
	Resources	Media	13 (15.3)	"Media coverage of predatory journals"
	Resources	Online	6 (7.1)	"Google"
	Other	Other	23 (27.1)	"Had heard of the practice… don't remember context"
				"Do not recall"
Total responses			N=85
Responses from patients who experienced health complications
Item			Responses	N (%)
Have you ever experienced a health problem or complication by making a change in your behaviour/lifestyle/treatment plan as a result of reading health information online? If yes, what type of health problem did you experience?			Increased uric acid levels	1 (12.5)
			Water retention in legs and developed hypoglycemia	1 (12.5)
			Cancer	1 (12.5)
			Liver disease	2 (25)
			Worsening of condition	1 (12.5)
			Mental health	1 (12.5)
			Reaction to suggested health aid	1 (12.5)
			Total responses	8

Study 2 – A focus group study

Participants

Twenty-nine participants took part in our focus groups; each group had 2–4 participants. The majority of participants were women (N = 18, 62%). Demographic information is provided in Table 5. [INSERT Table 5 HERE] Most of the participants had previously participated in a research study (N = 16, 55%) while 11 (38%) participants had experience as a patient advisor or partner.

Table 5

Demographics from pre-screening focus group survey
	N (%)
Gender	N = 29
Female	18
Male	11
Diverse	0
Prefer not to say	0
Age	N = 29
18–29	1
30–39	6
40–49	4
50–59	6
60+	12
Area of Canada	N = 28
British Colombia	4
Ontario	20
Other	4
Rural community	N = 28
Yes	5
No	23
Highest level of education	N = 29
< High school	0
High school	3
College	2
Undergraduate degree	10
Graduate degree	12
Other	1
Identify as indigenous	N = 29
Yes	0
No	29
Prefer not to say	0
Identify as person with disability	N = 29
Yes	5
No	24
Prefer not to say	0
Identify as a member of a visible minority	N = 29
Yes	3
No	26
Prefer not to say	0
Patient or caregiver	N = 28
Yes	25
No	3
Experience working in a healthcare or health research setting?	N = 29
Yes	12
No	17
Have you ever participated in a health research project as a participant?	N = 29
Yes	16
No	13
Have you ever been a member of a research team as a patient advisor or a patient partner?	N = 29
Yes	11
No	18

Thematic analysis of four key topic areas

Full details of the post-focus group feedback survey and thematic analyses can be found in Tables 6 [INSERT Table 6 HERE] and 7 (see additional file 4).

Table 6

Post-focus group feedback survey
What sort of features would you like a journal authenticator tool to have that could help you make a decision about whether you can trust a particular health research article or not?
Item		Total responses (N)	Rank score
1	Provide information about journal features that indicate credibility	27	6.78
2	Provide information about journal features that indicate lack of credibility	27	6.22
3	Provide information about institutions/affiliations associated with the work	27	6.11
4	Provide information about peer-review	26	6.04
5	Provide details on who the authors are	26	5.65
6	Provide information on "funders/sponsors of the research and/or any conflicts of interest"	27	5.41
7	Provide details about referencing and citations	27	4.33
8	Provide a summary of the studies methods	27	3.74
If you were to use a journal authenticator tool, how would you like it to look/feel?
Item		Total responses (N)	Rank score
1	Should have an intuitive user-friendly display	27	7.22
2	Should use non-technical language	26	6.15
3	Should provide details on how to access the tool and information	27	6.00
4	Should provide organized visual output of the results assessing a journal	26	5.50
5	Should be easily accessible	26	5.46
6	Should have the "ability to filter results"	26	5.08
7	Should be safe to access for the user	27	4.52
8	Should be visually appealing	26	4.35
How do you think we should tell people about the journal authenticator tool?
Item		Total responses (N)	Rank score
1	Through patient-centered outreach	27	7.44
2	Through media platforms	26	6.62
3	By ensuring that it is accessible to all	27	6.44
4	Through learning institutions and/or hospitals	26	6.23
5	By making it available on search engines	27	6.22
6	Through health professionals	25	5.68
7	Through health and NGO organizations	26	5.65
8	Through offering public education	26	5.46
9	Through the academic community	26	4.65
What type of feedback would help us determine if the tool is successful?
Item		Total responses (N)	Rank score
1	Through built-in user metrics	27	3.52
2	Through user feedback	27	3.37
3	Through health professionals and researcher's feedback	27	2.11
Choices participants had to rank were themes from thematic content analysis; total responses is how many people assigned a ranking to each option; rank score is the weight average of each item where the top ranked item is always weighed the highest

Table 7

Thematic content analysis from our four discussion points
Item	Themes	Codes	N (%)	Examples
How did you first learn about predatory journals?	Work and research experience	Via my employment	15 (17.6)	"Aware of them as a published author, and researcher"
		From research training experience	18 (21.2)	"Previous experience as a researcher "
		Have received e-mail invitations from predatory journals	4 (4.7)	"I receive invitations from predatory journals on a weekly basis."
	Other individuals	Friends	7 (8.2)	"In conversation with peers"
		From a health professional	1 (1.2)	"...in discussions with my doctors over the years in regards to research…"
		From a research expert	5 (5.9)	"From health scientists"
	Resources	Media	13 (15.3)	"Media coverage of predatory journals"
	Resources	Online	6 (7.1)	"Google"
	Other	Other	23 (27.1)	"Had heard of the practice… don't remember context"
				"Do not recall"
Total responses			N = 85
Responses from patients who experienced health complications
Item			Responses	N (%)
Have you ever experienced a health problem or complication by making a change in your behaviour/lifestyle/treatment plan as a result of reading health information online? If yes, what type of health problem did you experience?			Increased uric acid levels	1 (12.5)
			Water retention in legs and developed hypoglycemia	1 (12.5)
			Cancer	1 (12.5)
			Liver disease	2 (25)
			Worsening of condition	1 (12.5)
			Mental health	1 (12.5)
			Reaction to suggested health aid	1 (12.5)
			Total responses	8

What sort of information would you like to see to help you make the decision about whether you can trust it or not?
	Themes	Codes	N
	Features relating to referencing and citations	- When using the tool, a reference list that is reliable	16
		- Cross reference – does data appear elsewhere?
		- Do they have authors listed, affiliations, institutions, credentials, year of publication (more recent is better),
		- How often a journal is cited
		- Link to journal articles, excerpts
		- Links to research
		- Looking for more than one source
		- Looking source what other information they provide that might be useful
		- Multiple references/cross referencing
		- Multiple sources
		- Paying attention to website and who they are referencing
		- Rate references in article
		- References – indicator that references are unknown
		- When Links to the whole article are not provided
		- Where is the reference coming from?
		- Who are they referencing? Media, websites, or peer reviewed journals?
	Peer-review	- Peer reviewed	11
		- Peer reviewed
		- Peer review
		- Peer reviewed
		- Review process – peer reviewed? Not? Blind, single blind, double blind
		o Who the peer reviewers were – what institutions they were associated with
		- Peer reviewed
		- Peer review – how to make this understandable to lay audiences
		- How many peer reviewers? Maximum? Minimum? Ratings based on # peer reviewers
		o If institutions are credible
		- Journals that do not use peer review
	Journal features that indicate lack of credibility	- Those that are not as reputable have a lot of advertisements	23
		- Something that is odd or different from everything else, sounds like an opinion not validated
		- Sketchy website, older, ads, trying sell items
		- Advertisements – not trustworthy
		- Title of journal – related to topic?
		- No references, links, accredited institutions, authors
		- Testimonials are questionable
		- Fake impact factor
		- Look, spelling mistakes (especially of journal publisher name), quality of English language/grammar
		- Journals based in other countries that translate titles into English
		- Emails coming from journals that are aggressive/pushy
		- Date – how recent/up to date is the information
		o Pharmaceutical companies would be more skeptical
		- DOI number associated with full article vs abstract only
		o How trustworthy? Not translated well may suggest it’s not credible
		- If journal name is searchable through google
		- Ads – suggest they are trying to sell something
		- Journal article title that suggests research that is out of line with what has been published elsewhere
		- Who journals are advertising to for paper submission
		- Using titles that do not reflect reality
		- How long has the journal been publishing?
		- Location where paper appears – if it appears on public discussion boards = questionable
		o Concerns about practices
	features related to authorship features associated with named institutions/affiliations	- Is the author also an author of a book? Are they trying to sell something?	22
		- Author experience, what qualifies them as a specialist
		- Who is the publisher
		- Authors – links to other articles by same authors in other journals
		- Authors? Credentials?
		- Have there been concerns voiced before about a particular author
		- Where the authors are from – what institutions?
		- How long has journal been around? Who is contributing? What are their credentials?
		- What criteria is the journal using to accept/reject studies
		- Who started the study and why
		- Reputation of physicians
		Author credentials associated with specific health conditions
		- Where else have authors been published – other reputable places
- Symbol that indicates validated/verified to indicate credible authors
- Mismatch between authors academic background/training and the topic of the article
o Have they published work that has now been discredited?
- Flags on authors – history of disciplinary actions – e.g., Andrew Wakefield and autism study
- Skeptical of single authors
- Credentials of authors (academic institutions)
- Authors, affiliations, credentials – degrees
- Authors – links to other articles by same authors in other journals
- Author credentials
- Universities that are well-known with medical schools are more credible (e.g., Harvard, Stanford)		15
- Author specializations or institutions that specialize in specific conditions
- Institution reputation
- Research Institutions
o Easy to know when they are large institutions
o What about when they’re smaller institutions?
- Similarity to better known publishes
- Is it a reputable journal? Like BMJ, familiar, big well known journals
- Is the institution/university real, accredited?
- Will look for TOH info and from doctors
- Institutions/affiliations
- List of reliable journals or publishers
- Associated by Ottawa Hospital / institution
- Institution, accredited, specialized in health condition
- Mayoclinic, CDC, government based sites, NIH
Funders/sponsors of the research and conflicts of interest	- Soliciting donations, self-promotion, request for funds, selling products, e.g., Dr. Oz	9
	- Funding sources, conflicts of interests, associated with pharmaceutical companies
	- Funding sources – who is the journal funded by?
	- Conflicts of interest – pharmaceutical companies
	- Conflict of interests – funders – not always clear, patents, pharmaceutical companies, financial gain
	o What are physicians’ interests/conflicts?
	- Funders, associations to other organizations and institutions, conflicts of interests
	- Funders – reputable?
	o Who funds
hints/features that indicate credibility	- Credibility of sites and authors	18
	- Language used
	- What journal, publisher
	- Stage of review process
	- Hard to find information about methods, ethics
	o Indicate whether reputation meets a set of criteria
	- Do they have authors listed, affiliations, institutions, credentials, year of publication (more recent is better),
	- Outline of what the article is about, conclusion for summary
	- Endorsement or seal of approval – authenticator reviewed
	- What is the journal’s track record?
	- i.e., importance of having more than one source of information
	- Date (biggest one) and country of publication
	- Qualifications/reputations
	- Is this a credible journal? Is this a credible article? – 2 separate issues
	- Understanding the validity of research, i.e methodology and rigour of the research
	- Date of publication, dates of references – highlighting dates
	- Different appearance from typical journal
	- Often go to journal / source
Methodology	- Methods – sample size	7
	- Type of research (qualitative, quantitative), sample size, statistics (but challenging), how they arrive to the conclusion
	- Methods – sample size, objective or intent of study
	o Sometimes study of 1 can be useful information but may not be representative
	- Type of methods, how quickly were results generated
	- Has the research been replicated (done again), what is the sample size, methods (e.g., randomization)
	- Small sample sizes
Other	Not everyone will check it	15
	- How to assess journals that are based/published in languages other than English and then translated
	- What is the publication process – what is the difference between peer reviewed and predatory
	- Primary research vs systematic review or meta-analysis
	- Bias in how you word searches – how to search for different views
	- May be helpful to see where research is conducted that is published in specific journals (e.g., Lancet) i.e., what perspectives are they representing?
	- How to make sense of research studies – does the study make sense
	- How representative are the results – do they apply to others?
	- Blog, opinion pieces, biases
	- Logo or indication that it was trustworthy
	- Banner ads
	- Comparing ratings across articles
	- Comparing ratings between articles covering rare conditions with more common conditions
	- How you read the information
	o What is the process for discrediting/overturning/debunking?
Imagine using this tool at home … what would you want it to look like? How would you like to access it (e.g. cell phone, desktop)?
	Themes	Codes	N
	Visual appeal	Use of colours	15
		Yellow for caution on items
		green checklist for no ads
		Less is more
		Consistent colour scheme for ratings e.g., red = less trustworthy
		Check marks
		- Stop light – red for high alert, yellow for questionable, green go ahead
		- Red as a cue that you need to dig deeper
		- Visuals are helpful – pictures are worth 1000 words
		o could use colour scheme for journal would work
		- Infographic, checklists
		o some symbol like verify symbol would be helpful but maybe not colour
		o Visual presentation can be helpful for digesting information especially when developers are trustworthy
		- Charts, graphs, simplified information presented visually
		Green most trust worthy, yellow cautions, flags for traits to …
	How to access the tool and information	Accessible to all portable devices	28
		One central place that is credible and builds trust with public
		Having a central site that can indicate what is reliable
		Has to be accessible on all platforms
		Accessible on any device
		3A -pop up plug in for chrome, don’t have to think about it, the tool works automatically
		Pop ups super useful for those who don’t access often
		Bookmark or save a journal article – can bring to doctor later to ask if trust worthy
		Bookmark to EPI
		Extension on browser that pops up, can hover cursor and provides info
		Usable across platforms – cell phone, iPad
		- Search engine that lets you input the journal name
		- Convenient to access – e.g., plug in for Chrome or browsers
		o A tool that can be clicked on
		o If it is a pop up, needs to be easily dismissed
		- Sync across devices (tablet, phone, computer)
		- Sign in? log in? or just on in the background but not slowing down computer (not using too much memory)
		both on phone and desktop
		some will prefer to only use desktop
		- Access via desktop, fewer tools the better
		website for less frequent users
		o Correspondence between desktop and phone app – making sure they look the same – similar interface
		app for frequent users
		- Needs to be mobile friendly as well desktop friendly
		access on the desktop
		access on desktop or phone
		Google translate built in
		Accessible language, start slow and work your way up
	Non-technical language	Google translate built in	5
		Accessible language, start slow and work your way up
		Accessible
		- Has to be in laymen’s language
		Plain language, not too technical especially explanations of how to use the tool
	Visual organization of the output of the results	Google style search engine that can return results according to type (e.g., credible websites, links to journal articles)	8
		the ranking system of the output
		Organize according to most to least trustworthy
		Snapshot of traits e.g., reviewers, ratings
		- Provides checklist
		- List of credible journals? Vs list of predatory journal
		table
		- Website or app that provides references or indicates trustworthiness high, med, low
	Safe access of tool for user	- Safe search engine – like antivirus – like google search but with filter in place	2
	Safe access of tool for user	- Important to safeguard user privacy	2
	Ability to filter results	filter to narrow searches	11
		Search function with filters, topic areas, search according to traits (e.g., peer review, ratings, dates, topics)
		- Could do something similar that identifies what is of concerns and provides a rating
		- Rating system, e.g., 97% reliable
		- Maybe a menu? That can be adjusted based on criteria of interest
		- If it has a search function, it should provide top most relevant AND provide options for a deeper dive
		- Should provide option for in-depth research within tool
		- Would hopefully filter out misinformation
		Making search engine is comprehensive
		disease
		signs/symptoms
	Non-technical presentation	- Can’t be too lengthy or people won’t bother with it	5
		- Needs to be simple, provide an overview, provide a pathway into the health system
		- list of criteria being used with a good level of detail assessing journal
		- something that easily said worth looking at vs no
		Would be helpful to have apps that are specific to journals that provide layperson summaries
	Accessibility	- User interface easy, intuitive, font size, easy to ready/view for not great vision	5
		- Dark mode to reduce strain
		Accessible to all viewers, size of font, screen readers – other accessibility
		Needs to be accessible to a diverse audience
		- Needs to be accessible and inclusive so that all can benefit
	Other	Doctor or research assistant approval indicating trustworthy source	6
		- mechanism for authors or others to provide input on what metrics are being used to assess journals
		Saving own history and rating, favourites
		More in-depth information provided if users are interested
		Need quick and easy version as well as more detailed
		- Similar to mask Canada that vetted mask sources – google sheet
How do you think we should tell people about this tool?
	Themes	Codes	N
	disseminate through academic community	- Research article	6
		Journals that are deemed credible could also advertise
		Commentaries and editorials in credible journals, publications, conferences
		- Journals could promote tool – partnership
		Support from academia
		- Professional journals, licensing bodies
	By media platforms	- Social media	14
		- Radio
		- Facebook, social media, snap chat, Tik Tok
		- Social media but with caution
		- Twitter, Instagram, tiktok
		- Health journalists – Andre Picard, Julia
		- Go where people are most likely to go – chat rooms
		- Newspapers
		- Advertisements – marketing, media campaign – question is how broad?
		- News but not always trustworthy
		- Social media, YouTube videos
		- Social media, Reddit
		- Social media
		- Facebook – way of reaching those who are searching for specific health info
	Through learning institutions and/or hospitals	- Purdue University OWL site	13
		- Advertised through medical institutions
		- Tool could be part of hospital welcome package
		- Linking through TOH is helpful
		- EPIC, myChart, medical record systems
		- Page on hospital website, education tools available for use
		- On hospital website - Here are sites that we trust and here is a great tool
		- My Chart, o Could use same strategy that was used for My Chart
		- Through hospital medical records – my chart, flags
		- Academic institutions
		Hospital websites
		- Maybe an endorsement from TOH or other institutions
		- Orientation when setting up my chart opportunity to introduce tool
	Patient-centered outreach	- Patient and family engagement networks	9
		- Education awareness week – different topics throughout week
		- Science up – national campaign
		- Patient education in hospitals, can be included
		o Are already connecting with public
		- Public health offices
		- Patient partners
		- Choosing widely
		- Tool provided at the start of health journey – i.e., at time of diagnosis
	Through health professionals	- Online health services – healthcare providers recommending it would be helpful, good way to start	20
		o Could be recommended by doctors
		- Making sure healthcare providers understand tool and importance of it
		- Practitioner websites
		- Get to medical associations, through doctors
		- Circulate in institutions, through medical professionals
		- Promoted by healthcare provider – often doctors concerned you’re going to dr. Google, would be good to agree on tool to look up information
		- Can refer to tool when speaking with healthcare provider
		- Healthcare staff – important they are aware tool is available
		- Doctors offices, google searches
		- Poster in doctor’s office/waiting room
		- Doctors, doctors offices, clinics, nursing stations
		- Tool should have certification – part of that is distribution and what organizations feel comfortable promoting it – if they vet/approve it then others will trust
		- Primary healthcare providers
		- Doctors
		- Trusted people sharing it with others
		- Health care providers
		- Doctors can recommend tool to patients to encourage looking for trustworthy sources
		- Starting with healthcare providers – people already trust healthcare providers
		o Doctors would have trust the tool
	Health and NGO organizations	- Health Canada endorsements	21
- Government health agencies – Health Canada
- Medical associations
- Piggybacking on large organizations
- Patient newsletters
- BMJ blog
- PFACs across Canada
hospital PFAC
- Community health centres
- Large health organizations (e.g., Diabetes Canada)
- Organizations for those living with chronic health conditions
- Health charities as a source of information for general public i.e.,
- Partnerships with existing organizations (e.g., Cancer society)
- Rare disease organizations
- Large health condition groups e.g., Arthritis group
- Nonprofits e.g., Cancer Association, Kidney Foundation, other patient groups
- Medical associations
- Health Canada
- Prevention campaigns/organizations – already host courses/info sessions
- My pathology report – public facing how to read pathology report – tool could be posted there
- Community organizations, disease specific organizations, large non-profits like Diabetes Canada
Ensuring that it's accessible to all	- What about those with less access?	12
	- But other strategies important since not everyone has a doctor
	o Use different media for different groups
	- Brochure in different languages that could be distributed to religious groups, community organizations
	- Important to reach as many channels/people
	- Service providers, community health care centres, home care providers
	- Translated into different languages – consider making tool not too language dependent – accessible for those less familiar with the English language
	- Smaller organizations that provide online learning – Trualta – associated with organization or website – this is where caregivers go to learn
	- Community health centres, those doing outreach, Appletree clinics
	- Paper notices on bulletin boards
	- Handouts, bulletin boards
	- Community centres
Through offering education to the public	- Could be incorporated into a health literacy module	4
	- High schools – way to start early, get used to using it
	- Need to educate the public that they could benefit from this tool – why it is necessary
	- To educate public – videos like those used for this study – would need to be short and concise
Accessibility via searching	- High on the google search	6
	- First search page on google – top 4
	- Searchable through google, through google platforms
	- Connecting with PubMed
	- Caregivers need to be aware – online platforms like huddol,
Other	- Searchable through google
	- Cancer centre – symptom assessor kiosk/booth, tool could appear in interface	4
	- Maybe pop up on NGO websites to inform people
	- Maybe an ad on known sites like WebMD/Mayo
	- Browser plug in maybe helpful
If we go ahead with this project and make the tool, how will we know it is helping people?
	Themes	Codes	N
	Built-in user metrics	- Number of hits, repeats, are people coming back	20
		- May be hard to know if people accessed reliable information but CAN track clicks
		- Looking at metrics across age, demographics, patient groups
		- Number of downloads, number of users in a given time frame
		- Depends on how tool is built, - e.g., trackable fields, build in flexibility
		- Stats like usage or traffic but feedback mechanisms are necessary for details
		- Tracking whether traffic increases or drops off
		- Downloads, usage, clicks on the page
		- Whether people come back to it
		- Feedback mechanism – users can rate how useful each article was
		- Analytics, number of hits, types of hits (e.g., based on topics, other filters)
		- Number of hits/use, frequency of hits, worth cost?
		- Number of hits
		- Google analytics – use of the app – if it’s not used, not increasing in use, then suggests there is a problem
		- Site traffic, hits, repeat traffic
		o Site counts can be helpful but may not indicate use
		- If hits stay the same or go up = successful
		Do you like this? Thumbs up – but many won’t respond to this
		- Can gauge how long does it take to get to 500 responses
		o Tracking metrics
	user feedback	- Feedback comment section or survey	34
		- Feedback survey
		- Surveys in waiting room
		- Poll or feedback survey
		- Survey/interview before and after
		o Comments/feedback on article or tool
		feedback area
		- History of searches/activity
		- Feedback mechanism
		- Evaluation a year later, sharing back with networks who are promoting tool
		o Feedback of whether people will use it again
		- Pop up survey after using tool – could offer incentive/prize
		- Feedback loop
		- Or a simple “was this helpful?” Yes/no
		o Did you feel more confident in the reliability of the information you accessed after using this tool
		- Was this tool helpful to you? (first time and freq users – anytime people use tool)
		o Quick 2–3 questions
		o How it helped them
		o Regular surveys of people who use the tool – can inform updates and tweaks as they come up
		o What would you like to see more/less of?
		o Was this helpful? Build into tool
		- Reuse? Have you used this before?
		- Short evaluation at the end – was this tool helpful and how was this helpful
		- Survey with organizations for feedback
		- Can refer back to focus group participants to see if they have used it
		- Was this tool helpful to you? (first time and freq users – anytime people use tool)
		o Feedback form to submit comments re what things mean
		- Patient surveys – can include question to gauge tool awareness
		- For public, could anonymous feedback
		o Who is using it and how
		- Formal evaluation – how impactful or how helpful with people with and without healthcare backgrounds
		- Could also ask users to identify links to predatory journals
		- Some way collecting testimonies of people who use it and say whether they avoided journal/article
		- Comments section
	Through health professionals and researchers’ feedback	- Involving healthcare providers	5
		- Patient advisors – get them to use it and rate
		- Feedback from researchers, evaluation of how useful from research perspective
		- Ask nurses and doctors – are they receiving different questions?
		- Sharing back updates/tweaks with networks
	other	o May provide other ideas as to what would be helpful to track	18
		- What are users searching for?
		o How often they read articles with lots of no’s
		- Tracking what journals or articles are being searched for or inputted
- i.e., if lots of predatory journals, it’s working
- A way of keeping track of articles that may be useful in the future
- Need reminders throughout
- Providing real time data and changes – like Wikipedia
- Absence of complaints would be a good gauge
o Needs to be continually revised to keep with changing times
- Whether people come back to it
- Invite users who receive emails from predatory journals to forward them
If we see a decrease in problematic journals
- Whether people who use it avoid predatory articles
- Built in system to ask for feedback – rate how useful it was
- Follow up promotion through community groups and social media
- Could do focus groups after people used the tool
- Focus groups after users have tried it

Establishing what content participants would like to see in a journal authenticator tool

Nine themes were obtained in response to this key topic area. The top three ranked features obtained from the post-focus group survey participants would like to see in a journal authenticator tool are that it provides information about: 1) journal features that indicate credibility (e.g., date and country of publication), 2) journal features that indicate lack of credibility (e.g., sketchy websites, advertisements), and 3) institutions/affiliations associated with the work (e.g., institution’s reputation).

Other themes that emerged from the focus group discussions were that participants valued: 4) features related to referencing and citations (e.g., links to research, how often a journal is cited), 5) peer-review, 6) features related to authorship (e.g., author credentials), 7) information about the funders/sponsors of the research and any conflicts of interest, 8) information regarding the methodology and 9) other (e.g., bias in how you word searches and banner ads).

Establishing how participants want the journal authenticator tool to look and feel

Nine themes were obtained in response to this key topic area. In the post-focus group feedback survey, participants ranked the following as most important: 1) an intuitive user-friendly display (e.g. needs to be simple, provide an overview, provide a pathway into the health system), 2) use non-technical language, and 3) provide details on how to access the tool and information.

Other themes that emerged were that participants want a tool that: 4) is visually appealing, 5) displays the results in a well-organized manner, 6) is safe to access for the user (e.g., important to safe-guard user privacy), 7) can filter results, 8) is easily accessible, and 9) other (e.g., more in-depth information provided if users are interested).

Establishing how to share this tool within the community

Ten themes were obtained in response to this key topic area. The top three ranked items from the feedback survey are: 1) patient-centered outreach (e.g. patient education in hospitals can be included), 2) media platforms, and 3) by ensuring it’s accessible to all. Other themes that emerged were that participants thought the tool should be disseminated: 4) through the academic community, 5) through learning institutions and/or hospitals, 6) through health professionals, 7) by health and NGO organizations, 8) through offering education to the public, 9) accessibility via searching, and 10) other (e.g., browser plug-in may be helpful).

Establishing the successfulness of the authenticator tool over time

Four themes, based on 77 codes were obtained in response to this key topic area. The top ranked item from this key topic area was 1) built-in user metrics (e.g., Number of downloads, number of users in a given time frame). Other suggestions participants made for establishing the successfulness of the tool over time were 2) user feedback, 3) health professional/researcher feedback, 4) other (e.g., could do focus groups after people used the tool).

In the current study we considered patient needs, preferences, and requirements for designing a journal authenticator tool. We obtained preferences first through surveying patients, then through conducting focus groups drawn from survey respondents. In our survey study, the primary resource for patients when seeking health information was their health professional. Previous research has found that older adults prefer seeking health information from their doctor as opposed to the internet (25), which is consistent with our finding given the average age of our participants. We found that the internet is the second most common resource that patients use when searching for health information. Just over half of our participants indicated that they sometimes find it difficult to discern reliable health information and about half had not heard of predatory journals. These findings align with previous research showing that a large percentage of individuals are concerned about the reliability of the information they come across online (26, 27). Together they suggest a vulnerability to accessing unreliable information online, particularly with the increasing prevalence of predatory journals.

Our focus groups gave us the opportunity to explore the responses provided on the survey and to gauge participants’ need for a journal authenticator tool. Participants discussed four key topic areas and we identified a total of 32 themes across these four discussion points. These user preferences and prioritized needs will be used to inform the development of our journal authenticator tool. It is unlikely that we will be able to develop a journal authenticator tool that meets all the expressed preferences of our participants. We want the tool to be automated, but some preferences may not easily lend themselves to automation. Further, some of the preferences participants stated are not actually current norms within scholarly publishing. For example, one participant spoke about wanting information about who was responsible for funding a paper to determine if the funder held any conflicts of interest. While most scholarly journals provide funder statements, through discussion in the focus group we learned that some patients did not find the style and format of typical funder statements informative. For example, while all Canadian health researchers are likely to be familiar with ‘CIHR’ (Canadian Institutes of Health Research) our participant noted that this was meaningless to them. Instead, they would prefer to know if/if not the work was federally funded.

In subsequent studies, we intend to determine the needs and preferences of other stakeholder groups (e.g., journal publishers and researchers). We will then compare results from all three stakeholder groups and look for parallel statements across each as well as unique ones as we look to develop a journal authenticator tool to meet the needs of the broader community. We will be able to use our tool to highlight the transparency practices of journals, but also to monitor how transparency practices change over time.

This study is not without limitations. Firstly, our study was conducted in English which would have excluded any individuals who were not fluent in English, and focused only on a Canadian population. The study was also limited to those who had internet access and was only available for a limited time. Our sample is likely not representative. For instance, 37% of individuals have graduate degrees, and 75% were female. Our sample also appeared to represent a proportionally high number of patients who have been involved in research or had experience as patient partners. These groups are likely to find the journal authenticator tool particularly relevant given their active involvement in research. The relatively high proportion of these types of individuals is not surprising given out recruitment strategy in which we advertised through patient partner organizations.

This is the first part of the Centre for Journalology’s research program to develop an automated tool that could be used to check the transparency practices of academic journals. The results from this study will help inform the tool’s development to help ensure that it meets the needs of the patient community.

CHERRIES - Checklist for Reporting Results of Internet E-Surveys

CIHR – Canadian Institutes for Health Research

GRIPP2 - Guidance for Reporting Involvement of Patients and the Public

OSF – Open Science Framework

Ethics approval and consent to participate

This study obtained research ethics approval from the Ottawa Health Science Network Research Ethics Board (OHSN-REB 20200819-01H; see https://osf.io/qdv7e/). The study protocol was registered prior to data collection, using the Open Science Framework (28) (https://osf.io/fqvyr/).

Consent for publication

Not applicable.

Availability of data and materials

The datasets generated and/or analyzed during the current study are available in the Open Science Framework repository (https://osf.io/zc3s9/)

Competing interests

The authors declare that they have no competing interests.

Funding

This study was funded by The Ottawa Hospital Academic Medical Organization (TOHAMO) and Canada Health Infoway.

Authors’ contributions

Conceptualization: KDC, DM and MML; Study protocol: KDC, DM, MML, MH, LP, AG, GB and AR; Survey design: KDC, DM, MML, MH, LP, AG, GB and AR; Project administration and supervision: KDC, MML; Ethics application: KDC and AR; Participant communication: AR; Study conduct: GC, AR Analysis: KDC, EA, AR; Writing – original draft: AR, KDC, EA; Writing – reviewing and editing: All authors

Acknowledgements

The team is grateful to Donna Rubenstein for helpful discussion conceptualizing the project and for her assistance acquiring project funding.

The team is also grateful to Therese Lane and Hassan Khan who piloted our focus groups. MML is supported by The Ottawa Hospital Anesthesia Alternate Funds Association and holds a University of Ottawa Junior Research Chair in Innovative Translational Research. EA is thankful for the support of the Canadian Standard Association (CSA) as the recipient of the Canadian Standards Association Group Undergraduate Research Scholarship.

Dissemination Statement

Following the publication of this article we intend to communicate the results of the study through a presentation to our funders, dissemination of infographic results to our participants and on our social media account, and post our research findings on the Centre for Journalology website.

Lee CJ, Moher D. Promote scientific integrity via journal peer review data. Science. 2017 Jul 21;357(6348):256–7.
Prager EM, Chambers KE, Plotkin JL, McArthur DL, Bandrowski AE, Bansal N, et al. Improving transparency and scientific rigor in academic publishing. Brain Behav. 2019 Jan;9(1):e01141.
Grudniewicz A, Moher D, Cobey KD, Bryson GL, Cukier S, Allen K, et al. Predatory journals: no definition, no defence. Nature. 2019 Dec 12;576(7786):210–2.
Cobey KD, Lalu MM, Skidmore B, Ahmadzai N, Grudniewicz A, Moher D. What is a predatory journal? A scoping review. F1000Research. 2018 Aug 23;7:1001.
Cukier S, Lalu M, Bryson GL, Cobey KD, Grudniewicz A, Moher D. Defining predatory journals and responding to the threat they pose: a modified Delphi consensus process. BMJ Open. 2020 Feb;10(2):e035561.
Bohannon J. Who’s Afraid of Peer Review? Science. 2013 Oct 4;342(6154):60–5.
Owens JK, Nicoll LH. Plagiarism in Predatory Publications: A Comparative Study of Three Nursing Journals. J Nurs Scholarsh. 2019 May;51(3):356–63.
Cobey KD, Grudniewicz A, Lalu MM, Rice DB, Raffoul H, Moher D. Knowledge and motivations of researchers publishing in presumed predatory journals: a survey. BMJ Open. 2019 Mar;9(3):e026516.
Cobey K. Illegitimate journals scam even senior scientists. Nature. 2017 Sep;549(7670):7–7.
Government of Canada CI of HR. Strategy for Patient-Oriented Research - Patient Engagement Framework - CIHR [Internet]. 2014 [cited 2021 Aug 20]. Available from: https://cihr-irsc.gc.ca/e/48413.html#a4.
Daily T, Wednesday M, 25. 2011. Canadian Internet Use Survey [Internet]. [cited 2021 May 13]. Available from: https://www150.statcan.gc.ca/n1/daily-quotidien/110525/dq110525b-eng.htm.
Hesse BW, Moser RP, Rutten LJ. Surveys of Physicians and Electronic Health Information. N Engl J Med. 2010 Mar;4(9):859–60. 362(.
Fake journals in the age of fake news. the dangers of predatory publishing [Internet]. Healthy Debate. 2020 [cited 2021 Aug 6]. Available from: https://healthydebate.ca/2020/07/topic/dangers-of-predatory-publishing/.
OSF Preprints | Development and Validation of UCD-11. An 11-item Measure of User-Centered Design for Patient-Centered Tools [Internet]. [cited 2021 Apr 28]. Available from: https://osf.io/w5vnk.
Eysenbach G. Improving the Quality of Web Surveys: The Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004 Sep;29(3):e34. 6(.
Digital Health in. Canada | Canada Health Infoway [Internet]. [cited 2021 Jun 10]. Available from: https://infoway-inforoute.ca/en/.
Patient Relations. | The Ottawa Hospital [Internet]. [cited 2021 Jun 10]. Available from: https://www.ottawahospital.on.ca/en/patients-visitors/patient-relations/.
OHRI Journalology (@_Journalology). / Twitter [Internet]. Twitter. [cited 2021 Jun 11]. Available from: https://twitter.com/_Journalology.
SurveyMonkey Inc [Internet]. San Mateo, California, USA; Available from: www.surveymonkey.com.
CDC. What is health literacy? Take action. Find out. [Internet]. Centers for Disease Control and Prevention. 2021 [cited 2021 Aug 20]. Available from: https://www.cdc.gov/healthliteracy/learn/index.html.
Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006 Jan;3(2):77–101.
Zoom Video Communications Inc [Internet]. 2021. Available from: https://zoom.us/.
McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm [Internet]. 2016 Feb 5 [cited 2021 Mar 8]; Available from: http://link.springer.com/10.1007/s11096-016-0257-x.
Hsieh H-F, Shannon SE. Three Approaches to Qualitative Content Analysis. Qual Health Res. 2005 Nov;15(9):1277–88.
Chaudhuri S, Le T, White C, Thompson H, Demiris G. Examining Health Information–Seeking Behaviors of Older Adults. CIN Comput Inform Nurs. 2013 Nov;31(11):547–53.
Beck F, Richard J-B, Nguyen-Thanh V, Montagni I, Parizot I, Renahy E. Use of the Internet as a Health Information Resource Among French Young Adults: Results From a Nationally Representative Survey. J Med Internet Res. 2014 May;13(5):e128. 16(.
Murray E, Lo B, Pollack L, Donelan K, Catania J, White M, et al. The Impact of Health Information on the Internet on the Physician-Patient Relationship: Patient Perceptions. Arch Intern Med. 2003 Jul;28(14):1727. 163(.
OSF [Internet]. [cited 2021 Apr 23]. Available from: https://osf.io/.

Additionalfile1.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Establishing Patient Perceptions and Preferences for a Journal Authenticator Tool To Support Health Literacy: A Mixed-Methods Survey and Focus Group Study

Status:

Version 1

Abstract

Background

Methods

Results

Conclusions

Trial registration:

Figures

Plain Language Summary

Introduction

Methods

Patient and public involvement

Study 1 – Cross-sectional Survey

Study design

Identifying participants

Survey administration

Survey

Analyses

Study 2 – A Focus group study

Study design

Identifying participants

Focus groups

Analyses

Post-focus group survey

Results

Participants

Digital health literacy data

Determining accuracy of health information found online

Learning about predatory journals

Study 2 – A focus group study

Participants

Thematic analysis of four key topic areas

Establishing what content participants would like to see in a journal authenticator tool

Establishing how participants want the journal authenticator tool to look and feel

Establishing how to share this tool within the community

Establishing the successfulness of the authenticator tool over time

Discussion

Conclusion

List Of Abbreviations

Declarations

References

Supplementary Files

Status:

Version 1