Overview
The methods used for the systematic literature review and evidence synthesis are discussed briefly here and detailed in the following sections. We conducted keyword searches in Scopus and Web of Science to identify potentially relevant studies. The searches and initial removal of duplicate results from the two databases yielded 299,502 articles. We then further de-duped and filtered to articles that were published in 2010 or later and referred to a specific city or region in their abstracts, using the ClimActor harmonized dataset and R package. This initial processing resulted in a set of 51,562 articles. Articles were then screened for eligibility, first on abstracts and then on retrievable full text articles, using automated rules-based filtering and n-gram analysis in R. To be eligible for inclusion, each study must: (1) report numeric impacts to GHG gas emissions, (2) evaluate a strategy intended to reduce GHG emissions, (3) report impacts within the jurisdiction of a subnational (city or regional) government, and (4) report on the results of primary research. The automated screening process resulted in a set of 1,362 articles, which were then manually screened by members of the research team, resulting in a final set of 234 eligible subnational mitigation case studies. From this final set of articles, we extracted and coded 1,413 emissions reduction impacts and supporting meta-data.
In order to synthesize the extracted evidence, the emissions reduction impacts needed to be standardized. The effect statistic for this analysis was emissions reductions in annualized, per-capita metric tons of CO2 equivalent (metric tons CO2e/year/capita). We were able to standardize impacts from approximately 58% studies (137 studies) and 55% of the data (779 impacts), while the remaining data did not have the elements needed to standardize to the units of the effect statistic. We used clustered, non-parametric bootstrapping to estimate the average emissions reduction impact and confidence interval for each mitigation strategy category18,36,37.
Database Search
The literature search strategy expanded off of methods previously used to identify urban case studies15,16. Keyword searches in Scopus and Web of Science were used to identify studies that had both a subnational term and a mitigation term in their title, abstract or keywords. The search strings used in Scopus and Web of Science are shown in SI Table 4.1, with subnational terms and mitigation terms joined by “AND”. We did not limit by document type, in order to include both peer-reviewed and “grey” literature, such as books and conference papers. This keyword search yielded 149,875 results in Scopus as of January 2022 and 340,687 results in Web of Science as of March 2022. The article results were downloaded from Scopus and Web of Science, then de-duped by article title, journal, and year. In the initial data processing, the research team also removed duplicates of articles that were identified in both Scopus and Web of Science. After these initial processing steps were complete the dataset included 299,502 articles, with 149,016 from Scopus, 216,746 from Web of Science, and 66,260 articles that appeared in both the Scopus and Web of Science searches.
Article Screening
After identifying a set of potentially relevant articles based on keyword searches, we filtered the dataset to a subset of articles that specifically mentioned the name of a subnational actor in the title or abstract. This approach also aligns with the literature search process used in Lamb et al. (2019) and Sethi et al. (2020), who used the GeoNames database to filter down to articles that mentioned a city or urban location name with at least 15,000 inhabitants in the title or abstract. Since we wanted to capture other subnational government actors (such as states and regions) and not just cities, the dataset was filtered using the ClimActor dataset and R package. ClimActor is the largest harmonized dataset of city and regional governments that have participated in climate action networks, such as the Global Covenant of Mayors for Climate and Energy and C40 Cities initiative38. Furthermore, since the ClimActor dataset is intended to be used for data harmonization, it includes multiple possible names for over 27,000 subnational actors. This prevents us from excluding articles where the authors may not have used the most common name for a specific subnational actor. After filtering for articles that mentioned a subnational actor name in the title or abstract, there were 79,572 articles remaining. During this processing step, we observed that there were still a number of duplicates in the dataset and performed a second round of de-duping, based on article title only. We also removed any articles that had erroneously flagged a publisher name or location as a subnational actor by removing articles that had flagged a subnational actor name after the copyright symbol in the abstract and removed studies that were published prior to 2010. After these steps, 51,562 articles remained.
The articles were then screened using pre-determined eligibility criteria, first on abstracts and then on the full article texts. To identify subnational climate change mitigation studies with quantitative emissions reduction impacts, we established the following eligibility criteria:
-
The study must include quantitative emissions reduction impacts to greenhouse gas (GHG) emissions, reported in numeric form.
-
The reported emissions impacts must be associated with an intervention or strategy intended to reduce GHG emissions.
-
The reported impacts must be associated with a specific subnational government (city or regional) context.
-
The reported impacts must be the result of original research conducted in the study.
To screen article abstracts, three members of the research team screened a sample of 50 articles for eligibility based on their abstracts. We developed a list of “topical” keywords that indicated relevance to greenhouse gas emissions and “positive” keywords that indicated impacts to GHG emissions. We also established a list of stopwords that signified ineligibility and identified broad subnational terms from ClimActor that did not refer to specific actor names. These four sets of terms are listed in SI Table 4.2. First, we excluded articles with stopwords terms in the abstract or title (18,305 articles), then we removed articles that had been flagged with a broad subnational term (928 articles). We used N-gram analysis to screen the remaining articles. First, we generated bigrams from the abstracts of all remaining articles. The list of bigrams was filtered using the topical terms and then the positive terms, so that the remaining bigrams included one topical and one positive term. The final list of 593 bigrams and the frequency with which they appeared in the article abstracts is included in SI Table 4.3. We used this list of bigrams to filter the articles, aiming to identify articles that discussed impacts to emissions in their abstracts. This filtering process yielded 7,790 articles. From this set of articles, we were able to retrieve 5,380 full text PDF documents. Full text articles were then screened using a similar, rules-based strategy. Two random samples of 50 articles were selected for manual screening – one to develop a screening strategy and the other to test the performance of the final strategy. The final strategy identified articles with at least 2 references to “co2” that were located within 40 characters of a number, aiming to identify articles reporting quantitative emissions impacts. This strategy was then applied to all articles, resulting in a final set of 1,362 articles. The remaining articles were screened manually for eligibility by members of the research team, resulting in a final set of 234 articles.
Data Extraction and Standardization
After identifying the final set of subnational mitigation case studies, we extracted the emissions reduction impacts and meta-data from the articles. Each emissions reduction impact was recorded as an individual observation, either exactly as it was reported in the study or by calculating the difference between reported baseline and scenario emissions. We collected data related to the impact, the mitigation intervention, the subject of the intervention, the study methodology, and the subnational context. We also coded additional fields, such as the sector, mitigation strategy, and subnational actor type. A full list of the variables that were collected is reported in SI Table 4.4. For each impact, we captured details about the specific climate change mitigation intervention and categorized it into one of six sectors and one of 38 mitigation strategies, based on classifications used in similar studies.15,16 The list and definitions of these mitigation strategies are in SI Table 4.5. In the full dataset, 36 of 38 possible mitigation strategies appeared. The two strategies that did not appear in any study were “parking management-expansion, park & ride” and “walkability & pedestrianization”. The sector-mitigation strategy pairings were assigned to one of 13 mitigation strategy categories by sector (SI Table 4.6). For articles that discussed government actions and policies, a separate data extraction was conducted to record data on all the actions described in the article, even if they were not associated with a specific emissions reduction impact. We hosted two in-person workshops and one asynchronous workshop where student volunteers assisted in extracting data from the articles. Student volunteers assisted in collecting data from 20 articles, with each article being screened by two students to ensure data quality. We extracted 1,413 emissions reduction impacts from the 234 studies, with 6.04 impacts reported per study on average.
To synthesize the results, we needed to convert the extracted impacts to a standard metric, which we defined as emissions reductions in annualized, per-capita tons of CO2 equivalent (tonsCO2e/capita/year). Positive values indicate reduced emissions, while negative values indicate that emissions actually increased under a given strategy. Although all studies in the final dataset reported quantitative emissions reduction estimates, not all impacts were able to be standardized. Observations fell into one of four categories, based on two characteristics. The first characteristic related to the units of the reported impacts – about three quarters of studies reported per-time impacts, meaning they reported emissions reductions over a certain time frame, which were simple to annualize using the time frame of the impact. The remaining quarter of studies reported per-unit impacts, where the emissions reductions are reported in relation to another quantity, such as building area (ex. tons CO2/m^2) or waste production (ex. tons CO2/ton of municipal solid waste). It was possible to convert the per-unit impacts to per-time impacts if: (1) the study reported details on the type, value, and units of the subject of the study, such as total building area or the amount of municipal solid waste produced in the year, and (2) the units of the emissions reduction impact aligned with the study subject units (ex. the study reported emissions reductions in tons CO2/m^2/year and reported the total building area in m^2). Studies that did not report the information needed to convert the per-unit impacts to per-time impacts were excluded from the standardization. The second characteristic was related to the scope of the impact related to its subnational context, since the standard metric was reported per-capita. We chose to include observations where the mitigation strategy was applied to the entire subnational context or a specified portion of the subnational context (nearly 60% of observations) to reflect the perspective that a subnational actor might take when considering potential mitigation strategies. We excluded observations where it was unclear how the scope of the study related to the subnational actor. By this distinction, we would include an observation where a strategy was applied to 10% of all residential buildings, but exclude an observation where a strategy was applied to 10 buildings. This distinction ensures that the impacts of the included strategies were comparable in scale, could be attributed to the whole subnational population (as a per-capita impact), and were intended to represent the types of strategies that might be considered and implemented by a subnational actor. Note that this does not mean that all observed impacts in the synthesis dataset are attributable to subnational government actions (this is analyzed separately in the actions dataset). Eligible observations were standardized using the report impact units, time frame, subnational actor population, and study subject values and units (where necessary). Subnational actor populations were taken from the ClimActor dataset and supplemented with desk research as needed.
We were able to standardize 779 observations (55%) from 137 studies (59%), which comprise the synthesis dataset and are used in the meta-analysis. As a validity check, we compared the standardized emissions reduction impacts to the subnational actor’s country-level per-capita emissions. Of 779 total observations, just 10 (1.3%) had standardized emissions reduction impacts higher than country-level per-capita emissions. All 10 observations were in a city context – which tend to have higher per-capita emissions – and were projecting potential emissions reductions in future years (2025 or later), under assumptions that emissions would be higher in the future. Summary statistics for the standard metric, annualized, per-capita tons of CO2 equivalent, are reported in SI Table 4.7.
Synthesis and Analyses
We used non-parametric, cluster bootstrapping to derive the mean value and distribution of standardized emissions reductions across 12 mitigation strategy categories. Non-parametric bootstrapping is a flexible statistical technique for estimating uncertainty around a point estimate that does not rely on distributional assumptions and can be applied to finite samples, making it a popular option for meta-analyses 18,39,40. The standard non-parametric bootstrap method assumes each observation is independent. In our dataset, we cannot make this assumption, since many of the studies report multiple emissions reduction impacts that are likely dependent due to similar methods, context, and assumptions within each study. To account for this dependence at the study level, we selected the cluster bootstrapping technique, also known as the pairs cluster bootstrap, to account for within-group dependence at the study level37. For each mitigation strategy category, we resampled studies with replacement up to the original number of studies for that category, keeping all observations for each study that is selected. Then we compute the mean value of emissions reductions across all observations in the resample and repeat this process 10,000 times to derive a bootstrap sampling distribution for each category. The reported mean value and confidence intervals are drawn from the bootstrap sampling distribution. Using the standard deviations from the bootstrapping, we also estimated the coefficient of variation (CV) for each strategy category to facilitate comparisons across categories. The coefficient of variation is a relative measure of data dispersion, estimated as the standard deviation divided by the bootstrap mean for each category41,42.