Knowledge exchange activities between science and policy are driven by a need to address practical issues [1]. Several studies have highlighted barriers to effective translation from scientific evidence to policy [2],[3]. In particular, scientists and policy professionals often have different motivations and goals, which limit their collaboration [4]. Whilst the prospects for bringing these two communities’ motivations and goals into complete alignment are poor, scientists might reasonably gain a greater understanding of the motivations and goals of policy professionals––and how the evidence they generate feeds into, or helps to achieve, them. Examining the questions which policy professionals pose to scientists is instrumental to achieving this greater understanding. The style of a question provides valuable indicators of what the asker is motivated to know, and what they might use that knowledge for [5], [6]. Indeed, science-policy exchanges can often involve framing policy issues through a particular style of inquiry that articulates professionals’ policy goals and the means of achieving them [7]. In light of this, the aim of this study is to examine the structure of questions which policy professionals pose to scientists, in order to expose any underlying patterns in these evidence requests. It is hoped that an understanding of these underlying patterns will improve science-policy exchanges. Policy professionals will find it easier to improve the articulation of their policy questions. Scientists will find it easier to adapt their communication, in order to focus on the evidence that policy audiences want from them.
Before considering the substantive lessons to be drawn from the literature on questions, a terminological confusion must be addressed. Within this domain, a universal agreement regarding the meanings of certain relevant terms has not yet been reached. This can cause confusion, as the same term can be deployed to refer to distinct features of questions and/or answers. For example, Pomerantz [5] uses the term ‘content’ to refer solely to the subject matter that a question/answer concerns. By contrast––as noted by Pomerantz [5]––Graesser, McMahen, and Johnson [8] use the term ‘content’ to refer both to a question’s subject matter and its ‘style’–––specifically, the structure of the information sought (see below). When using antecedent studies to evidence the arguments in this paper, such terminological issues are disregarded in favour of the underlying point being made.
To begin, it is useful to consider two contrasting features of questions: subject matter and style. The subject matter of a sincere question indicates the information that the inquirer is interested in attaining, thereby indicating the kind of content which would be appropriate for a sincere answer [5]. For example, sincerely asking “what is net zero?” implies that one wants to know about the net-zero emissions goal. The style of a question––the structure of the information sought––indicates the understanding of the inquirer [9], and what kind of answer is expected [5],[10],[11]. Asking a sincere question implies that the inquirer has enough of an understanding of the issue from which to build the question and interpret the answer, but does not know enough to make seeking the answer superfluous [9]. For instance, sincerely asking “what is net-zero?” requires that the inquirer has at least heard of the term ‘net-zero’ but does not have a complete understanding of its referent. Furthermore, the style of this question indicates that sincere answers should be structured as definitions. By contrast, sincerely asking “what do we need to do to achieve net-zero?” requires that the inquirer has a basic understanding of what net-zero is but not a complete understanding of how to bring it about. Moreover, the style of this question indicates that sincere answers should outline the procedure(s) which will bring about net-zero.
Earlier work in psychology [8],[12],[13], linguistics [14], and information science [5] provided the foundation for the types of analyzes that have been used to develop taxonomies of questions. It is important to highlight here that question stems––such as “why…?”, “how…?”, “when…?”, and “what…?” ––have not been the basis on which a taxonomy is developed, because they are polysemous [10],[15]. The ambiguity of question stems makes their application highly context-specific, hence why question classification systems have generally focused on question styles [10].
The most practical approach to taxonomizing questions has been to classify questions according to their style. Lehnert [11] was the originator of this approach. Graesser, Person, and Huber [10] later generated a simpler taxonomy of questions by style (the ‘GPH Taxonomy’)––these questions were later grouped by the length of the expected answer by Graesser, McMahen, and Johnson [8] (see Table 1). For example, “what does X mean?” was given as part of the abstract characterization (“abstract specification”) of the ‘definition’ style of question. Definition questions invite answers which specify the details––usually as long descriptions––that characterize a phenomenon or event. In contrast, “what caused some event to occur?” was given as the abstract characterization of the ‘causal antecedent’ style of question. Causal antecedent questions invite answers which outline the factors that brought an event about. Taxonomies of questions can be used to investigate applied scientific problems. In order to improve outcomes in a variety of domains, such taxonomies are used to understand how subjects approach the task of structuring a problem or dilemma, what types of solutions they are expecting, and how their inquiries could be improved [6].
The GPH Taxonomy (see Table 1) has proved fairly popular. It has been successfully applied within the education sector [6],[8],[16],[17]. It has also been used as a foundation of, and supplement for, arguments made by other education researchers [18],[19],[20]. Moreover, it has played an applied, foundational, and/or supplemental role in studies analyzing web search strategies [21],[22],[23], consumer health-related inquiries [24], interpersonal exchanges [25], and interview settings [26].
Through an analysis of the frequency of questions generated, this taxonomy has been used to determine what types of questions are most likely to appear in a particular domain. Such information feeds into proposals regarding what improvements are necessary to support an effective evidence exchange process. In the education domain––where the GPH Taxonomy has been used most often––it has aided in identifying the types of inquires made by students, so that they can then be encouraged to formulate different styles of questions which enable a more substantive understanding of a topic [6]. For instance, often the efforts have been to shift students away from verification-style questions (“did X occur?”) to analytical questions––such as causal consequence-style questions (“why did X occur?”)––to develop deeper understanding.
Table 1
The GPH Taxonomy | Taxonomy of Policy Questions |
Super ordinate category | Class | Abstract specification | Super ordinate category | Class | Abstract specification |
Short Answer | Verification | Did X occur? | Bounded Answers | Verification/ Qualification | Is it the case that X is here? Did X event occur? Are Xs more inclined towards y? Is X a viable version of Y? |
Disjunctive | Is X or Y the case? | Comparison | What are the strengths and weaknesses of X? What are the costs and benefits of implementing X? |
Concept Completion | Who? What? What is the referent of a noun argument slot? | Forecasting | Which areas would you foresee improving in the next 10 years? How likely is it that X will be popular in the future? |
Feature Specification | What attributes does X have? |
Quantification | How may are there of X? |
Long Answer | Definition | What does X mean? | Unbounded Answers | Example/ Explanation | Which X is more like Y? What would be a case where Y is like X? How does X work? |
Example | What is an example of X? |
Comparison | How is X similar to Y? | |
Interpretation | What concept can be inferred from X? |
Causal Antecedent | What event caused X? | Casual Analysis (Antecedents or Consequences) | What are the barriers that will prevent X from occurring? What are the effects of X if it is implemented now? |
Causal Consequence | What are the consequences of X? |
Goal Orientation | What are the motives behind X’s actions? | Instrumental/ Procedural/ Enablement | How can we use X to make Y better? What would need to be incorporated to ensure that X is achieved? In what way can we measure X so that it can later be used to support y? |
Instrumental/ Procedural | What plan can allow for X to be achieved? |
Enablement | What resource allows X to perform their action? |
Expectational | Why did X event not occur? | |
Judgement | What value does the responder place on X? | Explaining/ Asserting Value judgments | How should the infrastructure available be used to produce x? How should X respond to y? |
Assertion | The inquirer makes a statement indicating lack of knowledge | |
Request/Directive | The inquirer wants the responder to perform an action |
This table sets out the GPH Taxonomy on the left, and the Taxonomy of Policy Questions on the right. The Taxonomy of Policy Questions was generated by applying the GPH Taxonomy to the dataset, and moderating it––as explained in the methods section––in order to better capture the data.
The theoretical underpinning of work analyzing the quality of questions has largely been informed by the ‘Grasser-Person-Huber (GPH) Scheme’ [6], [10]. It proposes that there are three dimensions on which a question needs to be assessed. Firstly, style (“content”): the structure of the information sought. Secondly, question-generation mechanism: the psychological processes––goals, plans, and knowledge––which bring about a question. The GPH Scheme lists four question-generation mechanisms: reducing, or correcting, a knowledge deficit; monitoring common ground; social coordination of action; and control of conversation and attention. The scheme holds that these categories are orthogonal to the style categories, since––in theory––a style of question might be motivated by any question-generation mechanism. For example, an inquirer might ask “what are the consequences of academic freedom?” to address a deficit in their knowledge. Alternatively, the same question might be asked to monitor the extent to which they share common ground with the responder. The GPH Scheme’s final dimension of assessment is ‘degree of specification’: the extent to which the information sought is made clear. A highly specific question is clear regarding what information is sought. Whereas, an under-specified question requires that the responder make inferences about what details are relevant to the inquirer.
Within cognitive psychology, associations have been made between the effective generation of questions and problem-solving ability, as well as the learning of complex material [6],[27],[28],[29],[30]. Within social psychology, improvements in ability regarding interpersonal exchanges has also been shown to be the result of asking good questions [31], that can increase one’s likability [25],[32]. Many of the efforts to improve cognitive functions (e.g. problem solving, critical thinking, memory, and text comprehension), by improving questioning, are based on two factors. Firstly, increasing the specificity of the question to ensure that the responder has the best chance of providing answers that are directly applicable. Secondly, encouraging ‘deep-reasoning questions’: those which direct the inquirer to ask questions that invite a causal analysis [6]. In essence this involves considering the cause-effect relationships between variables to start examining the underlying structures that enable inferences to be made about what brings about observable outcomes [33],[34].
To date, there has been no empirical work examining the styles of questions that policy professionals pose to scientific experts––including the types of questions that are asked, and the frequency by which they are asked. Once this is understood, it can be used to improve science-policy exchanges. Improvements can be made to the articulation of policy questions so that the value of the answers provided is maximized. Furthermore, scientists might find it easier to adapt their communication, in order to focus on the evidence that policy audiences want from them. To address this deficit, the present study analyzed policy questions that have been compiled by the Centre for Science and Policy (CSaP), at the University of Cambridge. CSaP is a knowledge brokerage which creates opportunities for public policy professionals and academics––primarily scientists––to learn from each other. This is achieved through CSaP’s policy fellowship programs, as well as workshops, seminars, conferences, and professional development activities. Regarding the main policy fellowship program, public policy professionals––as well as those from the private sector––initially submit a set of questions (typically between five to seven) to academics that shape the engagement with them. This is done to develop an evidence base which informs the types of policy issues that they aim to address. Thus, CSaP has accumulated a database of policy questions submitted by over 400 policy fellows over 10 years.
The database was used to examine two properties of policy questions: 1) What frequent styles of questions are posed to expert scientists? 2) Is there a relationship between the subject matter and style of questions posed to expert scientists? By answering these questions, it is possible to build up a profile of what evidence policy professionals invite scientific experts to provide, as well as what that evidence is applied to.