A review and meta-analysis of collaborative research prioritization studies in ecology, biodiversity conservation and environmental science

1. Prioritizing scientific inquiry: a fundamental task

Given the vast scope of scientific uncertainty and the limited availability of scientific resources, prioritizing research is a necessary challenge for the scientific community. Research prioritization occurs at many levels; from the broad-scale science planning of governments that allocate funding among regions and scientific fields, to the day-to-day decisions of individual research groups related to which topics, systems and techniques they choose to focus on. Prioritizing scientific focus on particular topics serves to provide a mechanism for the distribution of scientific funding and attention, and can also coordinate research in a way that may provide synergistic benefits and increase the rate of scientific discovery [1]. However, the process of identifying priority research topics is not trivial. Ideally, scientists would focus on research that will advance scientific knowledge and provide value to society. Yet, it is often not clear what research will lead to important scientific insights, and there are subjective judgements involved in assessing the value of one scientific advance over another.

In many applied research fields, there is an apparent priority to conduct research that will be useful to practitioners and policymakers and will contribute to evidence-based decision-making. Yet, many researchers have lamented the apparent lack of effect that scientific research has on policy and management, and this ‘science–policy gap’ warrants an examination into how topics for scientific investigation are prioritized. In particular, some authors have noted that the broad, holistic questions relevant to policy formation are not informed by the reductionist scientific approaches adopted by most researchers (i.e. scope mismatch; [2]). In addition, the value of particular research outputs to practitioners has largely been assumed by the scientific community [3], and a mismatch between the topics that researchers are studying and those that actually hinder evidence-based decision-making (i.e. topic mismatch) may also be contributing to the science–policy gap.

2. Collaborative research prioritization

Studies employing collaborative research prioritization (hereafter ‘CRP’) methods seek to address some of these issues and have become increasingly popular since the publication of the influential ‘One hundred questions of importance to the conservation of global biological diversity’ [4] and its predecessor ‘The identification of 100 ecological questions of high policy relevance in the UK’ [5]. By bringing together a diverse group of stakeholders, and using a democratic process to create a list of research priorities, these methods purport to identify topics that will both meet the needs of science users and are answerable through the scientific method [6]. In general, these studies follow four main steps (Box 1), including (i) the solicitation of a large pool of candidate topics, (ii) collation and processing of candidate topics, (iii) selection or ranking of topics by democratic methods, and (iv) dissemination of priority research topics in a list.

CRP methods have some commonalities with other established research synthesis methods. Narrative reviews, systematic reviews, bibliometric analyses and systematic mapping studies often include suggestions for future research that are derived from their analysis of what scientific knowledge is (and is not) already available. However, CRP studies are unique in their singular focus on bridging the ‘is-ought problem’ [8] by assigning value and rankings to future research topics. Indeed, implicit in the CRP methodology is the assumption that a group of experts collectively knows the state of a field and is a reliable source of estimating what the costs and benefits of future research would be. Certainly, consulting and considering the viewpoints of clients is thought to be beneficial in a wide variety of sectors (e.g. industry and government) and could help match scientific products to the needs of users. In addition, directly involving science users in prioritization processes should strengthen the relationship between researchers and science users, and may increase the transmission and use of scientific information regardless of changes in the quality of scientific knowledge being generated [9].

For all their apparent benefits, there is skepticism related to the use of CRP studies [3,9,10]. In particular, some authors have suggested that the priorities outlined in CRP studies may be too broad or too narrow to effectively inform research planning. As a result, there is a risk that the lists of priority topics will be inappropriately used to motivate research and distribute research funding [10], despite the fact that research topics not included in a CRP list may provide a more value to society in some circumstances. Conversely, the evidence that CRP studies actually motivate new research on the identified priorities has been thus far assumed and not directly examined; it may be that such studies are useful in generating publicity around a particular issue, but do not meaningfully alter the research trajectory of a field. CRP studies can also be time-consuming because of their reliance on a large group of contributors; if the needs of science users are rapidly shifting, then the research priorities identified through CRP methods may be out of date by the time a CRP study is completed. Furthermore, large teams may produce less innovative science [11], and therefore, CRP studies could hinder the transformative shifts that may be required to solve complex challenges such as the biodiversity crisis.

Nonetheless, there has been a proliferation of CRP studies in the fields of ecology, biodiversity conservation and environmental science (figure 1). In this paper, we review these studies and consider their role in determining the direction of research. In doing so, we examine the scope, methodology and impact of CRP studies to date, and assemble a database of all research priorities identified through these studies, which we hope will support researchers, practitioners and science funders.

• Download figure
• Open in new tab
• Download PowerPoint

3. A review of collaborative research prioritization studies

Since the first CRP study was published in 2006 [5], there has been a substantial increase in the number of publications that seek to apply these methods to a diverse set of ecological and environmental issues (figure 1a). In this review, we consider 41 such publications (table 1; electronic supplementary material, figure S1) that employ the CRP methods outlined in Box 1. These studies demonstrate a wide range in scope from studies prioritizing research on single species in a given region [38,43], to those identifying research priorities for the conservation of global biodiversity as a whole [6], and have focused on a wide diversity of topics (table 1). Some studies focused on issues that were a subset of issues covered in other CRP studies, for example Kaiser et al. [43] prioritized knowledge needs for the conservation of seabed habitat in the face of bottom-trawling, while a previous CRP study by Parsons et al. [38] examined research needs for the conservation of all marine biodiversity. Similarly, many studies are topically similar but vary in their regional focus, including the prioritization of research for national biodiversity conservation and management activities in Canada [16], Hungary [32], India [37], Israel [42], Switzerland [49], the UK [5] and the USA [50].

An explicit goal of CRP studies is the involvement of a large and diverse group of participants [6]. We compared group size across studies using three measures: the number of individuals involved in candidate topic contribution (Box 1—Step (i)), the number of individuals involved in prioritization (Box 1—Step (iii)) and the number of authors of the published manuscript (Box 1—Step (iv)). In many studies, individuals participated in multiple steps of the CRP process, and there was not always sufficient description to determine the number of individuals involved in each step. However, for studies in which the number of participants was explicit, candidate topic contribution was typically performed by a greater number of individuals than topic prioritization, with a median of 73.5 topic contributors (range 13–893, n = 30) and 38 topic prioritization participants (range 13–352, n = 30). The authorship of published manuscripts was typically composed of the study organizers and all, or a subset, of individuals involved in topic prioritization. This resulted in a range in size of author groups from 2 to 205, with a median of 29 (n = 41). Across studies, there was a weak relationship between the number of topic contributors and the number of individuals involved in prioritization (log–log R² = 0.16), while there was no meaningful relationship between the number of individuals involved in prioritization and the number of study authors (log–log R² = 0.02).

Unfortunately, most studies provided limited information related to the selection and composition of their participant groups which hindered the quantification of professional diversity (figure 2b). In the 19 papers for which we were able to extract comparable information, participants from academia (19 out of 19), government agencies (18 out of 19) and non-governmental organizations (16 out of 19) were typically involved in at least one component of the study. Relatively fewer studies involved resource users (7 out of 19), and Indigenous organizations were rarely included (1 out of 19). The median number of sectors represented per study was 3, with a range of 2–5.

• Download figure
• Open in new tab
• Download PowerPoint

Similarly, most studies did not report demographic data (e.g. age, gender and experience) related to study participants. However, using name-based gender assignment tools, we estimated the gender diversity of author sets from 33 studies (figure 2c) as one measure of demographic diversity. In contrast to our analysis of professional diversity, we found that there was generally low gender diversity in the authorship of most CRP studies. In 58% of studies for which data were available, the estimated ratio of male-to-female authors was greater than 2 : 1 (i.e. > 66% male authors), and the median percentage of male authors across all studies was 73% (figure 2c).

4. What are the scientific priorities for ecology, biodiversity conservation and environmental science?

The 41 CRP studies included in this review identified 2031 research priorities. To aid researchers, science planners and funding agencies, we provide a database of all research priorities identified in the studies we reviewed, including importance scores or ranks where available (electronic supplementary material). Identified research priorities ranged from highly focused questions addressable through targeted studies (e.g. ‘what is the total estimated population of Pilbara leaf-nosed bats in the region?’ [31]) to broad questions that may never be met with a scientific consensus (e.g. ‘what are the impacts on biodiversity of shifting patterns and trends in human demography, economic activity, consumption and technology?’ [4]).

Using word collocations, we explored common terms in the research priorities identified in CRP studies. Ten collocated terms repeatedly occurred across multiple CRP studies, with ‘climate change’, ‘protected areas’ and ‘ecosystem services’ being the most common (figure 3a). Notably, climate change, which was a feature of more than 15% of the research priorities identified in the seminal ‘One hundred questions of importance to the conservation of global biological diversity’ [4], decreased in prevalence over time (figure 3b). Additionally, the frequency of research priorities including the term ‘protected areas’ was highest in studies published in 2019 (figure 3b), despite the sole CRP study focusing on protected areas being published in 2013 [21]. Such a pattern could reflect an increased global focus on area-based conservation perhaps related to the Convention for Biological Diversity 2020 Aichi Targets [51] and associated national protected area strategies.

• Download figure
• Open in new tab
• Download PowerPoint

Given the large total number of identified priorities, and the persistence of some terms across published CRP studies (see above), we evaluated whether similar research priorities have been repeatedly identified over time (i.e. across multiple studies). Using cosine text similarity analysis [52], we produced scores ranging from 0 (totally dissimilar) to 1 (identical) for pairs of research priorities in our database. We found a trend for individual research priorities to become less unique (i.e. more similar to a previous priority) over time (beta-regression: β_year = 0.052, s.e. = 0.0038, p < 0.001; figure 3c), with an estimated effect size equivalent to a 5% increase in similarity score per year. Research priorities that had high commonality (table 2) focused on issues related to functional biodiversity, climate change, protected areas, fishing impacts and agriculture.

We found some evidence that the characteristics of the participant groups affected the content of the identified research priorities. For each study, we calculated the lexical diversity (moving-average type-token ratio [53], a score that ranges from 0 to 1 and indicates diversity in word use) and mean research priority length (words per priority) for the identified research priorities. While we found no clear relationship between participant group characteristics and research priority length (electronic supplementary material, table S3), we did find a pattern for increased lexical diversity in studies involving more individuals in topic prioritization and in studies with more authors, with estimated effect sizes suggesting a 68% increase in a lexical diversity score for every 10-fold increase in the size of the prioritization group and a 49% increase in lexical diversity for every 10-fold increase in the number of study authors. While this analysis was limited by poor reporting of the characteristics of CRP participant groups (see above), it is broadly consistent with other research demonstrating a link between group composition and the content of co-authored texts (e.g. [54]).

5. Evaluating the impact of collaborative research prioritization studies

One of the primary goals of CRP studies is to influence the research trajectory of a field by outlining topics that would be particularly beneficial should they be addressed by researchers. Yet, despite their increasing popularity (figure 1), there has been no analysis of the impact of CRP studies on the scientific literature. As such, we analysed citation patterns of CRP studies as a measure of their impact. We found that the total number of citations of CRP studies has increased over time (figure 4a), suggesting that the corpus of CRP studies are having a growing impact on the scientific literature. The citation rates of individual papers appear to be related to their scope, with papers with a broader scope (i.e. those focused at a regional or scientific discipline level) being cited at a higher rate than those with a narrower scope (figure 4b).

• Download figure
• Open in new tab
• Download PowerPoint

However, we note that CRP studies may be cited for many reasons beyond directly influencing the design of a citing study. For example, citations of CRP studies may be in reference to the methods used rather than the list of research priorities they produce, or a post hoc link may be made between a study and a research priority outlined in a CRP study. As a result, citation patterns are not direct evidence of the influence of CRP studies on research trajectories per se. Indeed, many CRP studies are cited within the first 2 years after being published (figure 4b). Given that a typical study in the field of ecology, biodiversity conservation or environmental science would take longer than 2 years to be designed, funded, conducted, peer-reviewed and published, we suggest that the presence of citations in the first 2 years post publication demonstrates that CRP studies are cited for reasons beyond being direct study motivators. In support of this idea, a recent analysis of papers citing Sutherland et al.'s [4] global CRP study suggested that 69% of citations were for reasons other than a direct justification of research on an identified priority ([17]; electronic supplementary material, figure S12). Additionally, we found that greater than 29% of studies citing CRP papers were secondary research items (i.e. reviews and other syntheses), and that citations of CRP studies had a higher ratio of secondary to primary research than the background publication rate (electronic supplementary material, figure S2). Taken together, these data suggest that much of the apparent impact of CRP is unrelated to their direct motivation of primary studies that address the outlined research priorities.

A recent study [11] presents an alternative metric of research impact using co-citation patterns to determine if studies were developmental (i.e. expanded on previous work) or disruptive (i.e. presented new ideas or solutions). We calculated this disruption score, which ranges from −1 (developing) to 1 (disruptive), for CRP articles in our review as a measure of their impact on the trajectory of a field. Contrary to Wu et al.'s demonstration that large teams typically produce more developmental science [11], we found that some CRP studies (which typically involve large teams) have been highly disruptive (electronic supplementary material, figure S3). Five studies included in our review [4,5,26,29,33] had disruption scores that would place them in the top 20% of the 12 million papers analysed by Wu et al. [11], with the most disruptive CRP study being Sutherland et al.'s [4] identification of important questions for global biodiversity conservation (D = 0.013, 94th percentile). However, the median disruptive score for CRP studies in our review (D = −3.7 × 10⁻⁴, 57th percentile) was similar to the median score across all articles (D = −7.9 × 10⁻⁴, 50th percentile), suggesting that CRP studies are no more disruptive or developmental than a typical article.

6. Has collaborative research prioritization helped bridge the science–policy gap?

CRP studies have emerged as a popular tool for the identification of important knowledge gaps in ecology, biodiversity conservation and environmental science. However, if the goal of CRP studies is to motivate future research, it is worth considering the capacity of the scientific community to deliver on these research priorities. Across the 41 CRP studies included in this review, we found a total of 2031 identified priority research topics, and additional research priorities related to ecology, biodiversity conservation and environmental science have been identified in other CRP studies not reviewed here. How long would it take to answer these questions to a level that is sufficient for science users? A recent review [55] of progress on Sutherland et al.'s [4] ‘100 questions of importance for the conservation of global biological diversity’ demonstrated a considerable variation in effort directed towards each question, but did not suggest that any of those questions had been sufficiently addressed in the intervening decade, such that further research was no longer needed. It may be that CRP studies have been too broad in their definition of priority research topics, or too vague in the formulation of those priorities [56], and are therefore at risk of diluting focus on the most important research questions. Certainly, reflecting on the adage ‘when everything is a priority, nothing is a priority’ seems warranted, and if CRP studies are to be useful guides for future research, then it will be important to balance the rate of identification of new research priorities to the rate of scientific discovery.

One of the primary reasons for researchers to be involved in the CRP process is that they should be able to temper the expectations of science users, and help to cast priority knowledge gaps as questions that are addressable by the scientific method. Most of the studies included in our review were led by academics and included a large proportion of academics during topic contribution and prioritization. However, there appears to be little evidence that the involvement of researchers has led to the creation of research priorities that are actually addressable. Indeed, demonstrating the value of involving researchers in CRP studies is crucial, as CRP studies have been criticized by those who argue that practitioner-led research agendas would be a better tool to address evidence complacency by academics and bridge the science–policy gap [3]. In addition, we suggest that identified research priorities may be more likely to be addressed if science funding organizations had more involvement in CRP processes. Including science funding representatives could help to cast priorities in the context of constraints on research funding and may also motivate science funders to enhance support on the identified research priorities. Science funding organizations necessarily prioritize research topics as part of their core activities and could benefit from the rigorous, democratic and transparent CRP procedures that are now being widely used.

Alternatively, perhaps, CRP studies should not be viewed as tools to directly motivate research on the identified priorities. Rudd [9] suggests that much of the impact of CRP studies is in their ability to increase the clarity of policy issues and strengthen the science–policy interface. Certainly, some CRP studies have been highly publicized in the mainstream media and other online outlets, and have perhaps helped to advance the discussion on ecological and environmental policy through these means. Furthermore, the direct facilitation of interactions between researchers, science users and other stakeholders during CRP studies is likely to enhance knowledge transfer and build appreciation for the roles and limitations of different parties. Models of knowledge co-production often emphasize the long-term and iterative nature of successful transdisciplinary collaboration [57], and CRP studies could be helpful in building relationships among individuals from different sectors, especially when CRP groups are professionally diverse.

The user focused framework of CRP studies represents a step forward in bridging the science–policy gap. However, CRP studies to date have involved a large proportion of academics and have rarely involved resource users and indigenous organizations. Furthermore, these studies have been predominantly authored by males, which could be a reflection of existing gender biases in academia [58]. Involving more diverse representation in CRP studies should help in the creation of more valuable and actionable research priorities, and could encourage the formation of long-term, cross-sector relationships. Towards this end, when possible, reporting on the demographic and professional composition of participants at all stages of the CRP process will help us to understand the impact of participant diversity on project outcomes. Furthermore, the development of standardized methods to evaluate progress on the identified research priorities would support analyses of the value of CRP studies and the methods that are particularly beneficial. CRP studies in the fields of ecology, biodiversity conservation and environmental science have identified 2031 research priorities since 2006. While there are certainly important research questions that have not yet been identified, this corpus provides a rich base to identify issues thought to be important by a number of expert groups. Whether the scientific community can address these research priorities, and whether they will meaningfully contributed to policy and management, remains to be seen.

Data accessibility

The database of research priorities and summary article-level for the 41 studies included in this review is provided as the electronic supplementary material, associated with this article. The R code used to conduct the analysis and produce that the figures have also been supplied. Full data required to conduct the analysis can be requested from Cody Dey ([email protected]).

Authors' contributions

M.A.K., J.D.M. and C.J.D. designed the study. A.I.R. and C.J.D. collected the data. C.J.D. and A.I.R. wrote the manuscript and conducted the analysis with assistance from M.A.K. and J.D.M.

Competing interests

We declare we have no competing interests.

Funding

This study was funded by Fisheries and Oceans Canada's Freshwater Habitat Science Initiative.