Incidents of re-application and reactions to feedback
The incident of reapplication among survey participants (n=233) is shown in Figure 2A. Of the responses, 47% were successful at T0 but did not reapply at T1. Reapplications were future grant applications to any funding agency, not just a reapplication to a Wellcome funding call. Irrespective of the outcome at T0, 66% (n=126) reapplied for funding at T1. Specifically, of those unsuccessful at T0, 62 (50%) of respondents went on to reapply at T1. A total of 35% of respondents were successful at T1 following an unsuccessful outcome at T0.
A diminishing pattern of resubmission was found for subsequent rounds of resubmission (T1+x), as respondents were successful at earlier stages and therefore did not need to reapply for funding, or else switched to a different research topic and/or career strategy. At T2, 27 respondents reported reapplying which represented 68% of those who were unsuccessful at T1 with this decrease continuing with subsequent reapplications. Although these numbers show the number of individuals reapplying decreasing with each application stage, there is no clear pattern of reapplication suggesting a role for other motivating factors.
In this peer review model, written reviewer comments are provided directly to applicants as feedback after the final grant decision making process is complete. The amount of feedback applicants receive is dependent on whether the panel considered that the applicant merited the commissioning of external written reviews. Survey respondents were asked about to indicate their opinions of feedback received, however not all applicants received feedback (7% did not receive feedback, Figure 2B).
Survey results show, a higher proportion of successful applicants rated the feedback as extremely valuable (51%) than those who were unsuccessful (10%). This may represent a differential perspective shadowed by a successful versus an unsuccessful outcome. Successful applicants were more likely to respond positively to the feedback as the outcome has already ‘signalled’ their value. In addition, a large proportion of unsuccessful applicants reported the feedback as having ‘no value’. It is, however, not clear how applicants used this feedback (if at all) in their subsequent research and career decisions.
Characterising positive feedback in reviewer reports
Targeted feedback
Targeted feedback was considered feedback that was directly relevant to the proposal and were not based on hypothesised assumptions of capability, ability or feasibility. Instead, targeted feedback was introduced by the reviewer in a tone that indicated suitable expertise through the use of words that were absent of uncertainty and assumption. This meant identifying specific sections of the applications to fix or change; identifying new experiments required prior to a resubmission; or else consideration of new arguments or literature to use in this resubmission. For example,
There has been some evidence that obesity is associated with variations in brain volume and in grey matter concentration or volume but these are not consistent. The PI might wish to consider whether an examination of brain grey matter differences is relevant to or essential for interpretation of the neuroimaging data: recognising that attempts to control for grey matter or volume differences may introduce additional confounds or affect the interpretability of the data. Reviewer comment (Targeted)
In this excerpt, the information about the additional variables to be considered by the applicant during resubmission that may ‘affect the interpretability of the data’, is offered in a tone that communicates suitable expertise and authority. In contrast, the use of words in the feedback below communicates uncertainty - such as ‘seems’ and ‘I think’ but did not always come as part of negative comments; “ I would describe the timeline as ambitious. However, I think it is achievable”. In cases when the doubt was associated with harmful feedback, the reviewers use of qualifying statements to absolve themselves of excessively harsh commentary, or else use opinion to make claims they cannot support otherwise. However, in some cases there were direct admissions from the reviewer of their inability comment on certain aspects of the proposal, due to their lack of expertise.
In relation to scientific rigour, the candidate has failed to justify the use of (only) 3 cases studies. Is this sample scientifically significant? I will leave this question to social scientists and simply raise it here. Reviewer comment (untargeted)
From the panel’s point of view, to diminutize this expertise, not only reflects negatively on the applicant indirectly, burdens the panel to navigate this uncertainty independently without access to the necessary expertise to make this assessment. From the applicants’ point of view, untargeted or vague comments were where the applicant does not immediately get a sense of what needs to be fixed to gain approval, such as ‘the methods are ill-defined’ or ‘there should be more detail in this section’. Other untargeted or vague feedback included the reviewer’s insertion of themselves into the feedback. Here, the reviewer’s use of ‘I’, ‘me’ or ‘we’ in the feedback often led to the commentary associated with the reviewer’s opinion instead of a detached assessment and assumptions about the proposal and applicant based on information not included in the proposal. For example,
The various pathways led me to wonder whether or not the applicant was distracted or unfocused and ponder whether or not the form was untidy and poorly constructed because of lack of focus or other distractions. (Vague)
This self-insertion, often led to feedback that implied prestige signalling to an individual (example; ‘this journal doesn’t seem prestigious enough for me’ or ‘I’m surprised they don’t do [suggestion], which I have found integral to my studies’) and both reflected a personal value of prestige and opinion of the review and imposing it as an unworkable (see Fair below) standard to the applicant.
Actionable feedback
Actionable feedback envisioned a future for the applicant and application by providing constructive comments to be altered future applications. In this way, actionable feedback was targeted feedback in the presence of judgment free steps aimed to encourage resubmission and increase the likelihood of success at resubmission. By doing so, actionable comments focused on what could be used to improve the proposal content and/or further articulating the ideas therein. The characterisation of these comments echoed Carless and Boud (2018) focus on the provision of feedback that is useful and highlights opportunities for the recipient to take action while supporting greater assessment literacy. These comments were goal-related assessments of the proposal that were also neutral and free from a judgement on where the proposal was going, and avenues for how to support to proposal to reach its goal. For the sake of the analysis, comments were only considered actionable if they could reasonably be completed within 12 months, allowing for enforcing necessary boundaries around reviewer comments. Actionable comments were often comments that recommended changes to methods or perspectives that, in the reviewers’ opinion, would improve the proposal but would not require a complete reconstruction of the project. For example, the below review provides two avenues for the applicant to consider in future iterations either change the title, or consider a shorter time focus;
I believe however that there is a problem with the design of the project and that the candidate should focus on a shorter period of time as the focus on XXX and his surroundings and the larger timeframe suggested in the title are inconsistent. Reviewer comment (Actionable)
In the above excerpt there is no judgement on which avenue the applicant should choose, but promotes the applicant’s learning by indicating proposal improvement scenarios (Hattie & Timperley, 2007). However, as previously outlined, feedback is only effective if it is acted upon (Sutton, 2012; Winstone et al., 2017) and some, more light-touch actionable comments are more easily adopted in future iterations than others. These included alternations such as adding references, new directions for thought, clarity of wording, or additional comments on the exclusion or inclusion of reviewer-offered suggestions. In these suggestions is the potential for bias to frame the interpretation of the feedback by the applicant, especially if it is assumed that suggestions from the literature stem not from a desire for the reviewer to improve future proposals, but through self-interested driven desires for inclusions and representation in future research. For example, in the above quote, the blinded nature of the peer review process means that it is difficult to interpret the motive for suggesting literature inclusions. It is therefore not clear how including this literature will improve the application or else act to virtue-signal to potential reviewers in the future;
Since a fellowship such as this offers the opportunity to work with the most appropriate groups to address a given problem. I would expect that the candidate should seek to bridge this gap by studying data from experimental epilepsy models and engaging with experts in this field…….other studies indicate diverse distributions for laminar states indicating different underlying mechanisms, see e.g. Suffczynski et al, 2006). Reviewer comment (Actionable)
In contrast to actionable comments, unconstructive or vague feedback was also found. This included feedback that was considered without utility or, comments that included unobtainable or impossible thresholds for improvement (e.g., suggests changing a fundamental aim of the project). Specifcallym unactionable comments were those that required a change based on achieving a goal of a judgement of professional improvement for the applicant, and/or would take more than a year to achieve. Examples here included a change in the host institution or supervisor; “the lack of a senior scientist with expertise in functional imaging as a supervisor or collaborator is a concern”; developing a stronger publication record; or else a major redesign of the project as it was stated in the application which would have included a core element of the project. In these situations, these goals were technically possible but feedback was made without the element of the process of achievement, or were only possible within a very long timeframe. Unactionable feedback was also vague in that they provided a judgement without a suggestion of the process of rectification for the applicant to consider ahead of any resubmission.
With more care and attention to detail, especially regarding questions of access and method, this appealing anthropological collaboration could become a fundable proposition." Reviewer comment (Unactionable)
These comments also betrayed a goal of the proposal, or applicant, becoming ‘fundable’ but again in the absence of meaningful steps needed to achieve this status.
Unfair feedback
Not all unsuccessful applications are associated with unfair feedback. For the purpose of this research, Unfair feedback was identified through a choice of words, or else commentary that was determined by the research team as to reasonably generate an emotional response by applicants. For example, the use of language indicating tentativeness right before the focal phrase, distorting it otherwise positive emotional impact, such as the following; “I assume the lack of detail was due to word limits of the project” Reviewer comment (Unfair). Here, the comment could be interpreted as positive or negative depending on how the applicant would receive, interpret and then constructively implement this feedback (Carless & Boud, 2018; Sutton, 2012) but the illusions to ‘lack of detail’ suggest a reviewer motive that overlooks the emotional toll to applicants in receipt of the comment.
There were also relationships between comments that were coded as Unfair but also unactionable which included suggestions for changes that were unworkable for an individual (e.g. change institutions, country or mentor) and through these alluded to a bias in their suggestions about what an individual was, or was not, capable of working considering their ECR status. As the below excerpt demonstrates, these comments included normative suggestions on what a “Fellow at this career stage” should do in order to present and more fundable (ibid) application. The suggestion of being “embedded in a group of imagers” or including more “senior researchers”, was a normative assumption of capabilities of the individual and therefore considered unfair;
However, there are no senior researchers in cognitive neuroscience involved in this application which is not ideal for a Fellow at this career stage who would be better placed being embedded in a group of imagers (even if they were working on different areas). Reviewer feedback (unfair)
However, the majority of unfair comments were related to judgements of quality (ex-post) or potential (ex-ante) of the applicant based on thresholds that were not relevant to the ECR-stage. These included direct statements of a ‘poor track record’ that betrayed a misunderstanding of common restrictions placed on ECRs in pursuit of research independence and job security. For example, the comment; “the most recent of these publications came out seven years ago. This is a poor track record, even taking into account her stage of career”, does not consider why such a gap for this (female) researcher exists, but instead makes a judgement on a threshold of productivity above and beyond an appreciation for the individual circumstances that underpin such a gap. Other unfair comments within this theme of places unrealistic or unfair thresholds of research performance or excellence on ECR candidates, were considered unfair because they seemed to be applying thresholds of excellence that were more akin to an assessment of more established academics, and were ill-suited for ECRs, e.g. “It is of some concern that he has not yet published a first author paper from his current lab”. This also led to complications between the reviewers’ sense of competitivity through performance, and establishing a level of career independence. Such comments overlooked the fact that the objective of ECR-followships at the Wellcome Trust was to provide the opportunity to develop research independence, and that having already established independence was not a fair pre-requisite for successful, and therefore fundable applications.
“The candidate has a strong track record, but the degree of independence is unclear” Reviewer feedback (Unfair)
Such comments were considered unfair. In addition, in the below excerpt the reviewer is applying a performance threshold that is divorced from the reasonable performance expectations of applicants that are, as per the eligibility guidelines, up to 5 years post PhD. In addition, the reference to “top journals” implies socially constructed terminology use that is embedded into the culture of which an ECR, by merit of being an ECR, is yet to become accustomed.
"I have some reservations about the suitability of the candidate. For someone who has been working in the field with a PhD since 2009, the publication rate is extremely slow and the venues, with one exception, are not the top journals." Reviewer feedback (Unfair)
Such comments suggest that for reviewers there is a tendency to apply to same standards of peer review practice to apply peer reviews with little consideration of the developmental needs of applicants that are also relative to the career stage and/or the funding call objective.
Associating feedback received with professional decisions at T0
During the interviews, participants were re-introduced to the feedback they received for their unsuccessful application at T0. They were then asked to reflect how they interpreted the feedback, felt that it was relevant, and how it inspired and/or influenced their next career choices. The use of feedback as vignette’s allowed a more robust reflection of how feedback inspired proceeding career steps, with the (temporal) distance for these steps to be reflected upon free of the emotive reaction of any injustice perceived by applicants due to the unsuccessful application.
Interview analysis identified two applicant typologies based on the characteristics of participant’s career/research choices and likelihood that they would resubmit their T0 proposal in a similar form[1] at T1. These were themed as “Switchers” and “Persisters”. Persisters were identified as applicants who reapplied at T1 after an unsuccessful submission at T0; whereas, Switchers were those who did not reapply at T1, or who had left academia after an unsuccessful submission at T0. How each typology responded to the characteristics of feedback described above received at T0 was explored in the interview analysis. In particular, how feedback sends ‘signals’ to applicants and, through these signals motivates behaviours for applicants to switch or persist was considered.
Signals to Persist
That participants were sensitive to signals received through reviewer-comments-as-feedback, compromises the consideration that competitive funding peer review is solely a process of selection. Participants reported how review comments their emotional state and, by extension, how they employed their own agency and feedback literacy (Carless & Boud, 2018; Winston et al, 2017). Participants also made assertions of reviewer motivations and temperament in receipt of the feedback as a meaning-making exercise. This was particularly the case when the feedback was negative. Applicants worked to personify the feedback given in a way as to reflect on the personal, and humanist motivations of the reviewer themselves;
So I'm not sure whether the reviewer himself or herself knew the impact of their review; [that it means not], that it doesn't mean that we are given the chance a chance to respond and improve the proposal, but it decides whether the proposal continues on or not. So in a way, yes the, the review is minimalist so probably in itself not negative, just asking for clarifications in defence, but in the context of the rejection ultimately, it feels like the negative review.
The sensitivity by which participants absorb these signals from feedback, and how these signals were internalised and then implemented in future research and career decisions, is an important reflection on the behaviour modification capability, and therefore the developmental potential of peer review. Participants primarily used theses signals in feedback as a tool to decide whether to resubmit (persist) at T1, or not (switch).
The decision to persist (resubmit at T1) was motivated not only on the overall impression of positivity in the feedback where the majority of applicants felt that the reviewer knew them individually, or the work. It was also motivated by a positive orientation towards their own agency to implement to suggestions within the reviewer comments. This also acted to alter their perception that the peer review was conducted fairly, with appropriate reviewer expertise (targeted).
Yeah, there was one that was very positive, and I could also tell from the wording he or she knew me before because they made a comment that wouldn't have made sense otherwise.
A signal to persist was embedded in how ‘useful’ the feedback was perceived towards resubmission at T1 (actionable), but also in how possible it was for the participant to act on that feedback. Since for ECRs, a resubmission at T1 is time-sensitive due to the eligibility of the applicant or else the novelty of the idea, feedback utility was related to how quickly and easily it would be incorporated into a renewed proposal. When feedback was used to stimulate learning, either about how to improve ‘grantmanship’ (McAlpine, 2020) (“learn how to make applications”) then this was seen as a signal by applicants to ‘persist’.
I let some time pass [after failure at T0] and then for my next applications I always reviewed the reviews. I always took into consideration what they said because it’s a process, one learns how to make applications
Despite a lack of success at T0, the applicants described how the feedback served not only as a tool to strengthen the application ready for resubmission (T1), but also is considered an important learning tool for the applicant at the ECR stage. For applicants who received feedback signals to persist, there was a sense that the feedback was considered useful, despite the negative outcome, if applicants retained a level of control over how it was to be implemented.
The feedback wasn’t particularly disturbing because I thought I can do something about the things which they appear to be most obviously criticizing.
If applicants felt in control of the suggestions made in the feedback, despite the decision not to fund the project, then this was interpreted as a signal to persist. In contrast, where this sense of control was lacking or else the feedback received did not stimulate learning was perceived as sending negative signals.
I didn't feel it was useful. That's for sure because it didn't teach me anything. It didn't help me in any way to see flaws in the study design and improve the study for the future - so in terms of usefulness – No.
These signals were amplified by emotions caused by the strong signal sent through a lack of success (failure), and hence did not encourage the applicant to persist.Further signals associated with the applicants’ decisions to switch are outlined in more detail below.
Signals to Switch
The decision of an applicant to “switch” is not associated solely with the decision to leave academia or to never again apply for funding. The decision to leave academia, was an extreme form of switching, but was still a career choice exercised by a small number of interviewees (n=3). More commonly, switching is associated with an applicant’s decision to change topic, discipline, to undergo retraining, or else to resubmit the T0 proposal in smaller, seemingly more competitive projects. The decision to switch, therefore, includes a range of behaviours and choices that result in either: the lack of an (re-application at T1; the lack of presence/visibility as an author in publications following the year of T0; and/or the presence of an application at T1 as a co-investigator only.
For applicants, receiving the news of an unsuccessful grant application was a strong signal to switch. This was amplified in situations where the news of a lack of success of an application was sent without any accompanying feedback. It should be noted here that for Wellcome, feedback is sent independently to the grant-decision. In the absence of reviewer comments, applicants felt frustrated not at the absence of feedback but by a sense of a lack of control that was explored in the previous section as a signal to persist; “Yeah, it's in a way, it doesn't matter how negative each is in itself, when the outcome is that you don't continue.”. Applicants who did eventually receive feedback, reported that the absence of accompanying reviewers report at the time of receiving the funding outcome heightened the desire to switch. Indeed, applicants felt that at this time their ability to learn from the feedback, and therefore continue to improve their portfolio or the proposal, and therefore the prospect of persisting, was limited. The absence of feedback – positive or negative – meant that for the applicant the only signal received was the blunt ‘reject’ from the outcome, and therefore to switch; ; “By that point you know that this means that you're rejected”. For participants, not all applicants received feedback and subsequently did not participate in the interview aspect of this research.
Feedback that solely communicate the rationale for the decision not to award funding, was disheartening again associated with a decision to switch. Such commentary was seen as devoid of relevance and as a tool of the funding agency and, by extension the reviewers, to promote authority irrespective of the characteristics of the applicant or their proposal. This reasserted an applicant’s perception of a lack of fit between themselves, their ideas and the potential for a future in the academy. In the excerpt below, as an example, the absence of feedback only acted to embed an impression of a lack of personal capability and sense of belonging;
I needed a while to see but they were saying, they were not saying, okay you are a fraud. They said ‘okay, we see that you have a background as a researcher and you can make some good stuff but this is not good enough for us and because this and that I’m sorry’ [you can’t be funded]
Other signals in the feedback associated with the decision to switch were more nuanced. As with the signals influencing a decision to persist, the level of perceived control an individual has over the future of the proposal, and its potential for eventual success is also important. As one participant explains, a successful project “it can’t just be like an incremental research project, it needs to be transformatory. It has to be transformatory.” Feedback that was seen as lacking in envisioning a future for the proposal was also associated with a decision to switch. This was not related to simply the absence of ‘actionable’ feedback, but also feedback that contained actionable characteristics that were beyond the capabilities of the applicant to implement. Indeed, feedback provided in the absence of a consideration of whether it was within the capabilities, or even morally reasonable to assume an applicant would confirm in order to increase the likelihood of future success, also sent a signal to switch. For example, comments related to mobility and the perception that greater mobility would increase the competitivity of the proposal. These comments were actionable, but they were also unfair, and would signal the applicant to switch.
This is not reasonable to ask people to relocate, and no amount of family relocation costs can change people's mental state when it comes to moving country.
Another common characteristic of actionable but unfair feedback was related to the assessment of the performance of ECRs at the stage of application. Feedback here concentrated on suggestions of how to increase performance but is also betraying the expectation that an increased performance would result in a more competitive application both at T0 and T1. Above all else the type of feedback imposed unfair thresholds on ECR performance as well as unfair expectations of future academic performance
So you know someone said to me like… ‘they need to publish ten more articles’ then we’d have a problem because I honestly can’t publish ten more articles in the space of a year, you know at least not good ones.
Likewise, if feedback was not actionable in that it was unclear or confusing; “It’s also very vague you know, it’s not easy to move it”, then this was also interpreted by applicants as a decision to switch.
[1] ‘a similar form’ relates to the understanding that a proposal at T0 is not always exactly the same proposal at T1. Applications at T1 can be: (1) the same; (2) a smaller pilot form of the project (less time, or less funding requested); or (3) in the same theme but still distinct in its methodological approach.