Participants’ perceptions of quality fall into two broad categories: defining characteristics and challenges. As elaborated below, comments in the former category align with Harvey and Green’s [3,4] conceptualizations of quality as ‘perfection or consistency’ and quality as ‘transformation,’ and comments in the latter category align with Harvey and Green’s conceptualizations of quality as ‘fitness for purpose’ and ‘value for money.’ In our analysis, Harvey and Green’s conceptualization of quality as ‘exceptional’ is removed from consideration, given its focus on operationalizing high standards of academic achievement is not applicable in the production of research. Because we are focusing on the production of knowledge through research, rather than through academic processes in the classroom, we felt it warranted to exclude the dimension of ‘exceptional’ from this discussion.
Defining Characteristics
The defining characteristics of quality, as identified by our participants, were coded into the themes of ‘committing to ethical behavior,’ ‘following scientific protocol,’ ‘transparency,’ ‘knowing your field,’ and ‘research that has been deeply thought about’ (See Figure 1). In viewing these results through the lens of Harvey and Green’s [3] framework, it became clear that individuals in our study conceptualized the defining characteristics of quality research production in terms of either perfection/consistency or transformation.
Quality as Perfection or Consistency
The themes of ‘committing to ethical behavior,’ ‘following scientific protocol,’ and ‘transparency’ all align with Harvey and Green’s [3] conceptualization of quality as a process where certain specifications should be met with zero defects and where the process is gotten right the first time. In this conceptualization, quality is differentiated from standards—it is something which conforms to a specification that in itself is not “assessed against any standards” (p. 16). Specifically, there is a focus on committing zero faults at each stage of the process, rather than relying on a final inspection (such as peer review) to identify flaws. Additionally, in this conceptualization, quality becomes a relative concept, allowing flexibility for disciplines and individuals to determine their own standards of zero faults [4]. Given this conceptualization of quality, the question then turns to how it is achieved in practice, revealed in the following themes identified by participants.
Technical mechanisms. Using specific technical mechanisms to follow scientific protocol was the most frequently mentioned way of producing quality research. Some participants broadly connected quality to following scientific protocol, while others delved into specific mechanisms they associated with quality. Rigor was frequently mentioned, along with replicable methods, as ways to ensure scientific protocol was being followed, as summarized by this participant: “I realize quality is a judgment call, but to me it is about...very clear methods, very clear controls, very clear descriptions of all your methods, because ultimately in science, it's supposed to be repeated.” Other participants noted the importance of relying on peers to assist in ensuring quality before journal submission, modelling openness in the research process for junior scholars, and having honest discussion about authorship early in the research process.
Committing to ethical behavior. Committing to ethical behavior when no one is looking was discussed in a range of ways, from statements relating to values such as, “Just behaving, being a good citizen in the scholarly world and not doing bad things with regard to your own research” and “the moral value of honesty is essential for the scientific process to progress” to more concrete suggestions related to research practices, “not tampering with data” and “don't go and steal someone else's work.” Participants touched on the need for ethical behavior in every part of the research process, from going through the “IRB [Institutional Review Board] process,” to “respecting your subjects” to “present[ing] your results as it is.” Ultimately, committing to ethical behavior was best summed up by this participant who said, “you have to be dedicated to the truth, period, no matter what.”
Transparency. After ‘committing to ethical behavior’ and ‘technical mechanisms,’ a dedication to transparency was the next most frequently mentioned attribute of quality in scientific knowledge production. Some framed transparency as “being willing to show your work,” whereas others positioned it as “giving appropriate credit to the people who were involved” or mentioning specific mechanisms to increase transparency such as “preprints… because you can see how the work evolved.” The importance of transparency in the knowledge production process was best summed up by one participant who noted, “The more open you can be about [your processes], the more likely you are to pick up what went wrong, what went right, share it, build on it.”
Whereas the above conceptualizations of quality all align with processes focused on creating knowledge with zero faults or mistakes through committing to transparency, ethical behavior and using the appropriate research mechanisms, the following conceptualizations align with notions of change and transformation as part of the knowledge production process.
Quality as Transformation
The themes of ‘value mechanisms,’ ‘know your field,’ and ‘research that has been deeply thought about’ all align with Harvey and Green’s (1993) conceptualization of quality as transformation (See Figure 1). In this way of approaching quality, there is a process of change and growth through learning. Harvey and Green [3] position transformation in relation to research production through understanding that the researcher situates their work in the broader context of their field to build and transform the body of knowledge in which their work is contextualized.
Value mechanisms. The theme capturing ‘value mechanisms,’ or the human traits that can protect or increase quality in knowledge production were quite nuanced, resulting in several sub-themes. The most frequently discussed mechanism was captured in the sub-theme of ‘being critical of your own work,’ expressed through statements such as, “be more critical on yourself than anybody else is” and ask, “How can I self-evaluate more effectively?” Participants also pointed to the “moral value of honesty [as] essential for the scientific process to progress,” a sub-theme that acknowledged the conundrum of needing morals in science because “science itself depends on that moral value but morality isn't science...you can't measure morality units…there's no morality meter.” The final sub-theme discussed by our participants was ‘respect,’ which included mentions of respecting subjects along with the research environment and colleagues’ work.
Know your field. Because reproducibility is considered essential to science, our participants were adamant that a scholar who knows their field will be better equipped to produce quality research, stating “Academic research… would ask you to have both knowledge of the past but also build new things for the future.” In this sense, quality was couched as ‘fitting your study in with the rest of the field” and, “a good quality article should be able to build on some sense of some people in the past and have their own new ideas that help us to understand the phenomenon or explain the phenomenon.” All the comments presented in this theme reflect the core elements of the scientific method, highlighting the processual nature of research inquiries.
Research that has been deeply thought about. This theme emerged primarily from the researchers we interviewed, showing a dedication to embracing the internalized process of knowledge creation, where “the researcher has to be curious on something” and “also a lot to do with a kind of passion.” Additionally, there was a sense that researchers needed to take the time to produce research that “has been thought over and discussed,” and that “Ultimately [the researcher] can still stand behind what [they] would have wrote then 10 years ago” so that the research “can create a dialogue with the existing literature” and continue to build upon the existing knowledge base.
The conceptualizations of quality as perfection or transformation were embodied by activities in the knowledge production process that could be grouped as defining characteristics of quality in the knowledge production process. Conversely, the sections below identify challenges to the knowledge production process, and the markers of quality associated with those challenges.
Challenges
When participants discussed the challenges associated with producing quality research, their responses were more nuanced, resulting in a broader spread of challenge categories, including: ‘Producing knowledge in a global environment,’ ‘inconsistencies in evaluation methods,’ ‘valuing quantity over quality, pressure to publish,’ ‘getting rejected,’ ‘needing to sensationalize findings,’ and ‘willful disregard of ethical practices’ (See Figure 2). In contrast to the defining characteristics of quality addressed above, the challenges related to producing quality scientific knowledge identified by our participants align with quality as ‘fitness for purpose’ or ‘value for money’ as defined by Harvey and Green [3] and explained at the introduction of each section below. Through these challenges, as identified by our participants, we begin to see the rift between institutional perceptions of quality and individual researcher conceptualizations of quality.
Quality as Fitness for Purpose
Quality under this conceptualization is one of the most widely used in evaluating certain aspects of higher education [4, 10] and is related to the ‘purpose of the service or product’ [3]. Under this definition, quality is equated with functionality, and whether a product or service does what it was intended to do from a user’s perspective. Harvey [4] notes that fitness for purpose is a particularly troublesome measure of quality because it allows for each individual to have multiple purposes, whereas Green [10] argues that purposes change constantly and are often unclear, making it difficult to meet the quality standards under this conceptualization. This difficulty is reflected in the challenges addressed by our participants in the themes of ‘producing knowledge in a global environment,’ ‘inconsistencies in evaluation methods,’ ‘getting rejected,’ and ‘needing to sensationalize findings.’
Producing knowledge in a global environment. Within this theme, participants discussed the challenges in navigating the differences between their disciplinary expectations on national, regional, and global scales and spoke to the perception that “quality is associated with the Western world,” noting that while they were not necessarily opposed to this global view, there were other regions which were producing high quality work too but were not being recognized because “they don't meet the kind of processes that the Western publication industry is familiar with.”
Values related to international standards of methodology were also questioned by one participant who stated, “international publishing means Western publishing, and I'm very critical against this stuff” and then went on to give the example of Latin America, a region this participant suggested has:
…their own research cultures, and they are very good in building their own databases and own publication networks… in Latin America, they developed a very good and working network… we can speak about Latin American research integrity because these articles meet their standards. When you are not Latin American, it will be different for you.
‘Resources’ emerged as a significant sub-theme, capturing the challenges faced by entire disciplines or scholars within a specific country. This notion of inequities between Western and non-Western scholars’ resources for contributing to the global sphere of knowledge production was also addressed in terms of how a supposedly global organization like the International Communication Association (ICA) perpetuates biases.
Inconsistencies in evaluation methods. Academia, by its very nature, encompasses all fields, meaning that one approach that works for a certain field may not work for another. But a bigger problem identified by several scholars from various parts of the world, expressed succinctly by a participant from Eastern Europe, was that “there is some assessment system, but nobody cares.” Another scholar from Southeast Asia noted their national system of ranking publications “only look[ed] at the ratings,” before continuing to describe how this has lowered the production of quality research because so much focus has shifted on playing the journal rankings game.
Some participants also discussed the intricacies of their national standards and the influence of the government in scholarly research practices. For example, one scholar in a STEM field who has worked in Africa and Asia, compared the influence of government policies in two countries, noting that in the African country there was an increase in publications in predatory journals after a new policy was enacted, emphasizing the importance of the quantity of publications for researchers, whereas in the Asian country, “they have a ranking…, which is developed by [the National Academy of Science]…So just publishing in a predatory journal, your articles is not going to be recognized … So, people here avoid predatory journals because of that.”
Getting rejected. This theme captures the acknowledgement of rejection as a constant occurrence in academic life and thus its impact on how participants conduct research. Although it may seem like this fear of rejection would encourage researchers to emphasize quality in knowledge production, one participant noted it may have the opposite effect, as it could lead desperate scholars to turn away from traditional, reputable publishers, and instead, “just rely on predatory journal.” Getting rejected unfairly was a particularly salient issue for scholars working in non-Western locales, a trend summarized by a scholar who stated, “when you submit your paper from Nigeria, and the same paper is submitted claiming that it is from Harvard, the referees will extremely overvalue the paper [from Harvard].”
Yet there was also an acknowledgement of a positive side of rejections: “even if we didn't publish at that journal that rejected us, we used that feedback to make the article better and better,” a viewpoint agreed upon by another participant, although they took more time to note the emotional difficulty of moving past rejection, acknowledging that despite the emotional difficulty of rejection, reviewer comments often help improve the manuscript.
Needing to sensationalize findings. This theme captures the potential of over-sensationalizing findings as a challenge to producing quality science and a “preponderance of people just wanting cool findings that you just throw at whatever high-impact journal.” One senior scholar spoke of the importance of instilling a sense of personal responsibility in junior scholars “not to oversell their studies, to be honest about what their study really showed, not the things that they wished it would have shown you” and not succumb to “temptations to maybe make the story a bit more streamlined…, or to make it a bit more simple, a bit easier to understand.”
But the trend of only publishing research that supports hypotheses or that is statistically significant is not just an issue for authors to address. In the sub-theme ‘importance of null results,’ participants spoke of the additional pressure from journals to only publish work that supported hypotheses, rather than understanding that “sometimes negative results can be very important as well,” and that “a lot of times an advance comes off something negative, not off something positive.” One participant summed up the issue of feeling pressured to sensationalize their work and not publicize negative results as “the most harmful thing probably for scientific progress… Because a null finding can be much more revealing than a finding.”
Willful disregard of ethical practices. Aside from the disciplinary and institutional pressure expressed by participants in our study, an added complexity that a handful of individuals discussed was the reasons why scholars engage in unethical research practices. Some participants tried to couch unethical behavior in softer terms, saying “people make mistakes. But there's a lot of … intentionally unethical behavior, because there's so much pressure to chase the money, and so if you're under a lot of pressure to chase money, yeah, you cook your data.” Another participant chalked it up to “laziness,” sharing the practice of “faculty members here in [my country], they just pick their student's project work and modify it as a journal article and submit” noting that “sometimes they put the name of the student along, sometimes they don't.” One participant shared a story of a colleague who knew they were submitting to a predatory journal, but did it anyway, ending the anecdote by saying:
My impression is that perhaps at that moment he was so unhappy with the system. He was so unhappy with the way reviewers were dealing with his submissions. …I think that when he found that somebody was appreciating his work …he simply was happy to have those ideas spreads through that journal. I think that sometimes they know that there is something wrong, but the fact that somebody accepted the work, … people like to have their work published.
Quality as Value for Money
Quality as value for money is conceptualized as customer-centered (e.g., tuition-paying students) and embraces competition as a driving factor for effective use of resources [3, 39]. Effectiveness, in turn, is gauged in terms of research assessment, control mechanisms, and quantifiable outcomes [3]. For our participants, quality as value for money aligns with the codes of ‘valuing quantity over quality’ and ‘pressure to publish’ (See Figure 2) due to the emphasis on quality being something that can be quantified and brings value to the institution.
Valuing quantity over quality. The complicated interaction between personal ethical values and institutional pressures was especially salient as participants discussed the trend in academia to value the number of publications, rather than the social benefit, practical application, or other less quantifiable markers of quality. This concept was addressed from a range of angles, including scholars noting cultures of publishing in which the publication venue does not matter: “in some areas, unfortunately, they're mainly counting publications not thinking about quality and so you can just publish a lot and you're considered productive.” Another scholar shared arguments from his students, noting when he tried to guide them towards publishing in reputable journals, “they say, ‘Oh, why should I take 100 or 1000 hours for developing a good paper, when I can just write something in low quality, but my university will accept it as international? Why should I care?’”
Others spoke about the focus on quantity from a resource perspective, noting “there are really perverse incentives in academia...with this idea of just maximizing a very poor proxy for quality, just because it's quantifiable,” an idea they continued with by saying, “quality has become…a question of numbers...the number of publications per academic.”
Pressure to publish. The valuation of the number of publications, rather than social benefit or practical application of fewer articles has led to a culture where scholars constantly feel they are in a race to “fling [manuscripts] to a journal... rather than taking the time and the effort to really prove yourself wrong.” This pressure to publish was especially salient to some PhD students, where their national systems require them to publish before their degree can be conferred (e.g., India and Cameroon), placing publications as “kind of the currency of the field.”
In terms of systemic challenges, the so-called ‘replication crisis’ emerged as a powerful sub-theme, with both researchers and publishers noting that placing more value on replication studies could increase the quality of research. One participant noted researchers’ wasted time because “people just keep doing the same stuff because they only have the articles published that did work.” A participant in the publishing industry looked at the responsibility of journals to be more open to publishing replication studies, a perception balanced by another participant who noted the difficulty journals face in providing all the materials scholars need to reproduce studies.