Limitations in historical satellite archives bias SDG monitoring

doi:10.21203/rs.3.rs-2949558/v1

Download PDF

Article

Limitations in historical satellite archives bias SDG monitoring

https://doi.org/10.21203/rs.3.rs-2949558/v1

This work is licensed under a CC BY 4.0 License

Version 1

posted

You are reading this latest preprint version

Satellite remote sensing is vital to monitoring, research, and policy addressing sustainability challenges from climate and ecosystem changes to food and water security. Here, Landsat satellite data play a crucial role, thanks to their unique global, long-term, and high-resolution coverage. Yet, disregarded biases in the Landsat data archive threaten the validity of their applications. Here, we demonstrate that global Landsat data are spatiotemporally highly uneven, frequently interrupted, and have seasonally incomplete coverage and quality. We show that these limitations are inherited in prominent global time-series products, leading to biased perceptions of changes in forests, croplands, and water resources that impair reliable assessments of related sustainability issues. Several data limitations and their biasing effects disproportionately affect lower-income countries. We provide global data-quality information to support their explicit consideration in future mapping efforts. Our results call for better data-bias reporting and control in satellite-based sustainability monitoring and analyses.

Earth and environmental sciences/Environmental sciences

Scientific community and society/Developing world

193 countries committed to 17 Sustainable Development Goals (SDGs)¹ to comprehensively address the environmental and social impacts of economic development. Yet, nearing the target year 2030, we are still far from meeting these goals². Widespread and rapid land alterations³ cause biodiversity loss⁴, accelerate climate change⁵ and threaten regional food and water security⁶.

The global 2030 Agenda for Sustainable Development foresees regular progress monitoring and reporting towards these SDGs, as a basis for their periodic recalibration to regional development differences⁷, and to identify national responsibilities for sustainability issues and secure practical commitments⁸. Satellite remote sensing allows monitoring many SDG indicators at multiple spatial and temporal scales⁹ and, thanks to open-data policies of key satellite archives¹⁰, (geo)computational advances¹¹, and investments in technical capacity-building¹², has become a primary tool for countries to meet their reporting obligations⁹. In particular, the Landsat program¹³ fulfils a vital role for continuous land-surface monitoring, due to its unique combination of long historical coverage (Fig. 1a) and relatively high spatial and temporal image resolution (30-m since 1982, typically every 16-days). Thanks to the program’s longevity, Landsat data are key to evaluating long-term environmental changes against historical baselines or in response to human interventions¹⁴. Moreover, Landsat data play an important role in providing forward-looking policy support by enabling the construction of credible future scenario projections that help anticipate emergent development issues and their possible responses to policies¹⁵. Correspondingly, Landsat data have become an integral part of a growing number of mapping applications¹⁶, including many designed specifically for tracking progress towards SDGs^17–19.

However, the archive of historical Landsat satellite images contains extensive data gaps (Fig. 1b), and existing images often have limited quality due to cloud cover or data degradation²⁰. These gaps and quality limitations can lead to misclassifications of land-surface features in derived time-series products, and thus to misinterpretations of land changes (Fig. 1c). Regional and temporal variations in the magnitude and prevalence of these satellite data limitations can hamper efforts to reliably map change patterns and to establish common baselines across countries. For example, interruptions in the annual continuity of satellite observations may lead us to misinterpret the timing and magnitude of long-term changes in forest cover²¹. Changes in time intervals between usable images, in turn, affect our ability to detect periodic, abrupt vegetation changes that often distinguish agricultural from other vegetated lands²². Similarly, changes in within-year image availabilities affect our ability to capture dynamics of seasonally occurring land resources²³.

Although many remote sensing experts are aware of such issues²⁰, their maps and time-series products do not generally control for the different data limitations. Moreover, their products ultimately reach a much broader, non-expert user community of resource managers, policy-makers, and scientists from different disciplines. Additionally, open-access policies and cloud-computing technologies have mainstreamed remote sensing and enabled non-experts to generate their own products²⁴. While these developments helped accelerate scientific progress²⁵, many dataset developers, as well as data users, are now largely unfamiliar with the limitations of satellite archives²⁶. Moreover, as data choices become vast, more users rely on the ‘Landsat’ reputation as an indicator of data quality²⁶. Most literature on satellite data limitations is highly technical and directed at remote sensing experts²⁰, whereas literature bridging remote sensing and other fields mostly focuses on promoting different remote-sensing methods or applications²⁷ and disregards their limitations. Frank discussions of the magnitude and implications of different types of satellite data limitations targeted at non-expert data developers and data users are largely lacking.

To tackle this gap, we expose the magnitude of spatial and temporal variations in different types of data limitations in the global, historical Landsat archive since the 1982 launch of the first 30-meter-resolution sensor. To this end, we map different dimensions of data limitations in the archive, quantifying the between- and within-year frequency, recurrence, and quality of daytime Landsat images. Additionally, we demonstrate how spatial and temporal fluctuations in image coverage and quality affect our perception of global changes in forest cover, seasonal water availability, and arable-land extents as mapped by state-of-the-art monitoring products.

An uneven history of satellite data limitations

Between 1982 and 2022, the Landsat archive contains 17,553,123 daytime images (per-pixel avg.: 1618, ± 1599; Extended Data Fig. 1a), meaning that 33.1% of images that would be expected under a 16-day revisit frequency are missing (see Methods). Moreover, the average pixel effectively lost an additional 44.8% (± 17.6%) of available observations due to cloud cover or degradation of Landsat images (Extended Data Fig. 1b/d), leaving an average quality-weighted number of 1,429 (± 887.4) images per pixel across the 40 years (Extended Data Fig. 1c). Data gaps are spatially and temporally highly uneven and particularly prevalent in the Tropics, Arctic, and Antarctic (Fig. 1b) and in earlier years (Fig. 2a). Notwithstanding the regionally important role of cloud cover²⁸, extensive data gaps primarily relate to the historical development of the Landsat program²⁰.

Global coverage of Landsat data evolved only gradually (Fig. 2a, Extended Data Fig. 3) with the support of a global network of receiving stations established in all continents except Antarctica (Extended Data Fig. 2a). Whereas the archive grew by an average of 118,362 (± 67,104) images per year during the 1980s, this rate increased nearly five-fold by the 2010s (to 575,169, ± 159,924; Extended Data Fig. 3). However, improvements were not uniform. Many countries lacked (or still lack) the infrastructure and know-how to continuously collect and preserve data from overpassing satellites²⁰ (Extended Data Fig. 2a-b). As a result, many world regions, particularly at low and very high latitudes, have lagged behind in developing a dense Landsat data record (Fig. 2a). Improvements happened particularly late over islands, with, for example, 65.4% of Oceanian Island areas never observed until the 1990s.

Beyond these gradual increases, changes in Landsat satellite technologies caused several abrupt global changes in data coverage. Since the launch of Landsat 7 in 1999, Landsat satellites include on-board data storage that reduced the reliance on global networks of receiving stations for assuring data collection and archiving²⁰. Combined with improving data transmission and warehousing, this helped the Landsat archive expand rapidly during the 21st century. Since 2003, however, mechanical issues in Landsat 7 degraded as much as 25% of pixels per image²⁹. With the end of Landsat 5 in 2010, the quality of available images decreased until the launch of Landsat 8 in 2013, where data coverage improved dramatically (Fig. 2a). The average proportion of countries’ lands with a full-year coverage increased from 17.5% (± 30.8%) for the years before 2013 to 89.4% (± 26.4%) thereafter. Additionally, interoperability between Landsat 8 and earlier missions is hindered by differences in the sensed spectral information³⁰, a challenge not addressed explicitly in state-of-the-art mapping applications.

With the exception of much of North America, nearly all land areas have continuous gaps lasting ≥ 1 year in their post-1982 Landsat data record (Fig. 2b). 36.2% of those lands have ≥ 1-year interruptions after their first data coverage, including most islands, most of Africa, Mesoamerica and north-eastern South America, northern Beringia, Patagonia, and Antarctica (Extended Data Fig. 4a). Oftentimes, these interruptions persisted over extended periods (averaging 5.4 years, ± 3.7), with several Central African regions not having a single usable Landsat image during > 10 consecutive years (Extended Data Fig. 4b). These archive interruptions often reflect short-lived or inconsistent receiving and storage capacities (Extended Data Fig. 2b), and are particularly severe during the mid-1990s²⁰.

When data are available, their frequency and quality vary (Fig. 2c-d). Whereas much of the world has a large number of any Landsat images (Extended Data Fig. 1a), consistent high coverage with high-quality images only exists for dryland regions of the Middle East, North Africa, Australia, and North America (Extended Data Fig. 1c). By contrast, for most equatorial forest regions of the world, less than half of existing images are usable, due to persistent cloud cover²⁸ (Extended Data Fig. 1b). In some areas such as the Sahel belt and much of Central Asia, annual fluctuations in quality-weighted image numbers exceed annual averages (Fig. 2c-d). In similar regions, seasonal data coverage is highly incomplete, often with less than three months covered with data in a given year, and fluctuations of a similar magnitude between years (Fig. 2e-f).

Data limitations bias perceived changes in SDG indicators

Unless carefully accounted for, the described spatial and temporal differences in data coverage and quality can introduce biases into the derived time-series products used for monitoring SDGs. For example, discontinuous historical satellite-data coverage impairs our perception of the timing of change events, while varying frequencies of quality images affect our ability to perceive time-sensitive changes, and fluctuations in seasonal data completeness limit our ability to distinguish multi-year changes from seasonal fluctuations. These biases in monitoring can lead to poor policy-making. For example, biased information on trends in forest and wetland areas may lead to ill-conceived protection and restoration goals as Nationally Determined Contributions under the Paris Agreement. In the following paragraphs, we will highlight how different types of Landsat data limitations indeed bias perceptions of changes in different SDG indicators derived from state-of-the-art monitoring products.

Gaps in quality data bias perceived timings of deforestation events. Changes in forest areas affect multiple SDGs and are the focus of SDG indicator 15.1.1. Tropical moist forests accounted for > 90% of global deforestation since 2000³¹, and are particularly important for capturing and storing carbon (SDG 13), preserving and restoring biodiversity (SDG 15), and providing billions of people with income, food, and/or medicine from forest products (SDGs 1, 2, and 3). Accordingly, the recently published Landsat-based Tropical Moist Forest product (TMF)¹⁹, which maps the onset of deforestation since 1982, is poised to play prominent role in global monitoring under diverse policy frameworks, including the Paris Agreement, the Post-2020 Global Biodiversity Framework, or EU regulations on deforestation-free supply chains. The TMF addresses temporal gaps in the Landsat archive by preserving the last-recorded classes into the gap periods and only mapping any class changes once new satellite images confirm those¹⁹. The resulting annual time-series thus hide uncertainties regarding the true timing of forest-change events. This means that the inferred deforestation years and perceived change trajectories may commonly be biased by unaccounted gaps and quality differences in the Landsat archive.

We examined the TMF and, indeed, found strong indications of such biases. Globally, perceived annual deforestation affected disproportionately large portions of the tropical moist forest biome during two periods since 1990, both of which mark periods of particularly rapid improvements in Landsat satellite-data coverage and quality (Fig. 3a). For example, from 1999 to 2000, right after the launch of Landsat 7, the World seemingly experienced an increase in deforested areas of 65.2%, more than twice the maximum year-over-year increase (31.9%) registered anywhere between 1990 and 1999. Similarly, the 2012–2013 deforestation increase of 60.5% coincides with increased image frequencies following the launch of Landsat 8 and is nearly twice the recorded maximum over the 2001–2012 period (30.8%), in which Landsat 7 was the sole data source. Regions experiencing ≥ 1-year periods with either no data or potentially unusable, low-quality data show disproportionately higher deforestation rates during years immediately following those periods, compared to their smoothed trend line (Fig. 3a; see Methods). This results in 67,329,203 ha of globally deforested areas that are potentially allocated to the wrong year (Extended Data Fig. 5a; see Methods), corresponding to 58.4% of total gross deforestation mapped since 1990.

We wanted to know whether gaps in Landsat data actually bias the perceived timing of deforestation. To this end, we used a formal causal analysis technique developed for detecting causal relationships between two time-series called Convergent Cross Mapping^32,33 (CCM; see Methods). Based on the results of these analyses, we attribute deforestation anomalies to anomalies in maximum annual image quality in preceding periods in 68.8% of tropical-moist-forest countries (Fig. 3b; see Methods).

Resulting biases in perceived deforestation years may bias any timing-sensitive applications related to achieving SDGs, including modelling of carbon emissions⁵, restoration prioritization to mitigate extinction debts³⁴, or attributions of forest changes to changing socio-political conditions³⁵. For example, deforestation inside the Luo Scientific Reserve (Democratic Republic of Congo) that reportedly happened during the first Congo war (1996–1997) due to human displacement³⁵ would be falsely attributed to processes in the immediate post-war period, for which Landsat data are again available (Extended Data Fig. 5b). These biases may also cast unfair perceptions of national progress in curbing deforestation. Twelve countries indicated by the TMF as having increasing deforestation rates around the Landsat 8 launch – when data improvements were particularly strong (Fig. 2a) – in fact reported decreases in the Forest Resource Assessments (relative to the previous reporting period)³⁶, including countries with successful restoration and conservation programs over that period (e.g., Cuba, India, Vietnam, Thailand)³¹.

Increasing frequencies of quality data miss regional arable-land losses. Accurately capturing dynamics in arable-land extents is a critical component of measuring SDG indicator 2.4.1 on agricultural lands under sustainable use, and is also closely linked to indicators aimed at avoiding deforestation (15.2.1) and loss of water-related ecosystems (6.6.1)⁶.

Mapping arable-land requires temporally dense satellite observations to capture phenological land-surface changes driven by crop planting and harvesting, and as such is highly sensitive to cloud-related gaps in Landsat data³⁷. To tackle this, a recently developed global product (GLAD)¹⁸ maps arable-land in four-year epochs, exploiting the highest-quality images of an entire epoch (aggregated into an annualized 16-day time-series) for more accurate detections of cropping-related phenological patterns¹⁸. Yet, this approach is not immune to increasing densities of high-quality images in the Landsat archive over time (Extended Data Fig. 3), which may lead to overestimations of arable-land gains by reducing the likelihood of missing existing arable-lands, compared to earlier time periods. Simultaneously, changes in newer Landsat sensors relative to earlier missions (e.g., different bad spectral ranges)³⁰ are likely to misinform classification algorithms fed mainly with data from earlier periods.

In fact, we identified 123 countries where the GLAD mapped gains despite reported losses in national statistics (Extended Data Fig. 6), casting doubts on 74,975,550 ha of arable-land expansion, an area larger than the total of all arable-lands across all Amazonian countries in 2021³⁸ (Extended Data Fig. 5c). Most (80.0%) of these positive disagreements relative to statistics are associated with improvements in the frequency of quality Landsat images (Fig. 3c). These disagreements peak between the 2008–2011 and 2012–2015 epochs (36.0% of cases), coinciding with the 2013 launch of Landsat 8 which massively increased quality-image numbers (Fig. 2a). Doubtful arable-land gains concentrate in Southern Asia (19.5% of doubtful gains), South America (18.8%), and Western Africa (18.5%, Extended Data Fig. 5c), with countries such as Ghana in Nepal consistently experiencing positive disagreements between all epochs. Our causal analysis using CCM attribute the former to the latter in 48.4% of countries (Fig. 3d; see Methods).

These biases in perceived arable-land changes can severely bias perceptions of global food security issues. The 54 countries with moderate to high bias-causing effects include top food-producing countries (e.g., China, Russia, France) and together accounted for 38.4% of global cereal production in 2021³⁹. However, they also include many food-insecure countries (e.g., Central African Republic, Niger, Somalia, South Sudan, Yemen, Zimbabwe), where misinterpreting losses of arable-lands for gains bears risks that policy-makers might fail to recognize emerging crises.

Overestimated arable-land gains can also lead to unfair evaluations of progress towards SDG target 2.4 (sustainable food production) that exaggerate conflicts of food security with ecosystem protection and climate-change mitigation. For example, two recent studies^40,41 using GLAD data reported extensive cropland expansion into global protected areas, with massively accelerating expansion rates between the mid-2000s and mid-2010s. The above-described data biases associated with the 2013 launch of Landsat 8 (Fig. 2a, Fig. 3c), however, may render these assessments unreliable. This is illustrated in India, where sudden changes in Landsat data led the GLAD mapping algorithm to falsely re-classify an entire protected Ramsar wetland of > 3,000 ha into arable-land (Extended Data Fig. 5d).

Improving seasonal data completeness exaggerates water gains. SDG Indicator 6.6.1 tracks changes in surface water bodies, such as lakes, rivers, and reservoirs, and is informed by the Landsat-based Global Surface Water product (GSW)¹⁷. Particularly in many dryland regions of the world, seasonal water bodies that only exist for a few months per year play a crucial role for water security, both as seasonal sources of drinking water and water for livestock and cropping⁴², as well as for filling aquifers that sustain water supplies during dry seasons. Even outside drylands, seasonal flooding of river plains affects both natural nutrient inputs in, and leaching from, major agricultural production regions⁴³. The GSW maps seasonal (as well as permanent) surface water extents annually based on monthly classifications of water occurrences.

These data show a nearly 5-fold increase in global seasonal water areas between 1984 and 2020, with increasing trends over 90.1% of the maximum seasonal-water extent. However, because the GSW maps water if as little as 43.5% of expected images per year are available, seasonally biased distributions of those images could either entirely miss seasonal water occurrences or misclassify seasonal for permanent water (if only covering the dry or wet season, respectively). Therefore, long-term increases in seasonal completeness could be falsely mapped as increasing seasonal-water extents⁴⁴.

We found that, indeed, global seasonal surface water gains correlate with improvement in seasonal data completeness (number of months with usable data; r² = 0.80; Fig. 3e, Extended Data Fig. 5e), which are largely unsupported by local discharge measurements (64.5% of gauge stations show disagreements, Extended Data Fig. 7; see methods). Our causal analysis using CCM found moderate to high bias-causing effects in 144 countries (Fig. 3f), including several with severe water stress (e.g. Yemen, Sudan)³⁹, mischaracterizing persistent and expanding water scarcity issues driven by increasing drought frequencies⁴⁵ (e.g., in Somalia⁶; Extended Data Fig. 5f).

Biases disproportionately affect lower-income countries

We found that Landsat data limitations, as well as the resulting biases in perceptions of land changes, occur disproportionately often in countries with lower financial capacity to sustain remote sensing monitoring programs. Specifically, biasing effects on perceived arable-land and seasonal water trends were significantly more frequent in lower-income than in higher-income countries (McNemar’s tests, arable-land: 51.9% of lower-income vs. 46.1% of higher-income, p-value = 0.00; water: 89.7% vs. 73.1%, p = 0.00; note there was a near-significant difference in deforestation bias in the opposite direction among the respective income groupings of tropical-moist-forest countries; 51.9% vs. 46.1%, p = 0.07; details in Methods). Unless ensuing biases in SDG indicators are accounted for, misperceptions of progress in food- and water-security goals in those countries may hamper adequate international support and timely policy interventions.

Similarly, we found higher average frequencies of years without any usable data in lower-income countries (Wilcoxon test, p = 0.0, avg. of 4.9% [± 2.9] vs. 3.7% [± 4.0] for higher-income countries), affecting 43.3% of their combined area, compared to only 17.2% of the combined area of higher income countries (mainly high-latitude and offshore territories). Similarly, we found that pixels in lower-income countries were more frequently affected by fluctuations in usable-data frequencies exceeding the expected frequencies under a 16-day recurrence (60.2%, vs. 40.6% for upper-middle-/high-income countries; p = 0.00), and also by fluctuations in usable-data months exceeding a typical climate-season length (88.2% for lower-income vs. 70.73% for higher-income countries; p = 0.0).

Future needs: bias corrections, fair product validations, and support to users

While this paper focuses on Landsat data as the most important resource for long-term, global land-change monitoring, all satellite data archives are affected by uneven data coverage and quality^46,47. Given the importance of satellite-based land-change observations for sustainability policy, monitoring, and related scientific fields, addressing the highlighted biases caused by limitations in global satellite archives becomes imperative. This will require more rigorous bias-control and more honest validations by data developers, as well as better support for (and commitment by) data users for detecting and addressing remaining uncertainties.

Firstly, expert communities developing remote-sensing-based time-series products should raise standards for correcting for satellite data limitations before applying classification algorithms. An increasing array of sophisticated approaches can fill gaps in satellite archives^48,49, for example, by fusing sparse Landsat with coarser-resolution but less incomplete data from the MODIS and AVHRR satellite missions to generate global, seamless data cubes⁵⁰, but such approaches remain rarely applied in operational land-surface monitoring. To further improve their performance, information on data coverage and quality, as provided here for Landsat (see Data availability and Code availability sections), could be made available for all sensor systems, enabling its explicit use by gap-filling models for correcting satellite data to desired, high-quality levels.

Secondly, we need higher standards for assessing uncertainties in the derived land-change products. All three products scrutinized here were, in fact, extensively validated by their developers. Yet, validation samples were mostly generated by visually interpreting Landsat images^17–19 – as is true for nearly all global time-series, especially for pre-2000 periods, where few alternative sources of validation data exist⁵⁰. For accuracy assessments to be meaningful, however, accuracies must be comparable between validated and non-validated pixels and years⁵¹. In reality, the selection of validation samples and their correct visual interpretations are both biased away from the most data-limited regions and periods, which is also where the classification algorithms are most likely to fail. This likely results in exaggerated accuracy scores and hence unwarranted trust in Landsat-based monitoring products. To be honest towards data users, accuracy tests should directly incorporate information on limitations in both satellite and validation data. Again, models could be used to generate seamless predictions of class-confusion probabilities in between existing samples that are representative of all pixels and years, including those with limited data. Much more than allowing 'corrections’ of all pixel values⁵¹, this should allow mapping remaining uncertainties in ways that enable their due propagation into change assessments and indicators⁵², for example, in form of probability-mass functions of alternative class sequences for each pixel.

Such higher standards would imply more time needed for the development and quality-assurance of time-series products, and thus fewer, more transparent products that pass peer-review and enter the market every year. This would be desirable from the perspective of data users, who are already overwhelmed by too many products to choose from with little guidance on which products they should trust²⁶. Many data users may be similarly overwhelmed by fewer but more voluminous products with rich, pixel-level uncertainty information, as they lack the technical capacity to effectively use them. Thus, we additionally need easy-to-use tools helping with their use, as well as with selecting the most fit-for-purpose product for a given desired application. For example, software packages could support easier incorporation of data uncertainties into, and propagation between, different types of applications (mapping, change assessment, causal analysis, etc.). Similarly, cloud-based tools could automatically test where within a user-specified region and period a given product could plausibly support the desired application, given the product’s uncertainties and/or underlying satellite data limitations.

At the same time, data users should acknowledge that remote-sensing “data” on land changes are not facts, but model-based interpretations of (satellite) data that often inherit large uncertainties. Ultimately, data users carry the burden of validating their original results on land changes, even when they rely on existing products. Easy-to-use and freely available webtools for exploring historical time-series of high-resolution images (e.g., Google Earth Pro) empower them to do so.

By highlighting data limitations and biases in perceived land changes, and offering data-quality layers and suggestions for addressing these, we provide an essential first step. We hope that this may serve as a starting point for the needed collaborative actions, to ensure that satellite-based data can reliably guide progress towards a sustainable future.

Quantification of satellite data limitations

Landsat quality metrics. We developed a suite of quality metrics that characterise the quality and coverage of Landsat satellite observations. The data-coverage aspects considered are annual and year-to-year frequency and recurrence of images, weighted by the quality of individual observations. The quality of individual images reflects the reported geometric and spectral image quality⁵³, as well as limitations in image usability caused by cloud cover²⁸. We calculated these quality metrics for each year and each descending tile drawn in the World Reference System 2 (WRS-2), which is used to partition Landsat data into publicly available chunks⁵³. We then combined the tile-specific metrics by averaging them into global grids with a 1-km resolution.

We calculated the quality metrics base on the available metadata for all images acquired between 1982 and 2022⁵⁴ with Landsat 4, 5, 7, 8, and 9. We thus disregarded images obtained with Landsat’s Multispectral Scanner System (MSS) that, while available since 1972, provide data with a lower spatial resolution (60 to 90-m) compared to more recent sensors (30-m), the latter resolution providing more adequate spatial detail for historical environmental monitoring⁵⁵. MSS data also have a coarser spectral resolution, making them unsuitable for many remote sensing applications^56,57.

Image quality. We calculated the quality of each Landsat image j (QI), for a given year y and tile t, as\({Q}_{y,t,j}/9\times \left(100- {CCover}_{y,t,j}\right)\). This results in a normalized metric between 0 (worst) and 1 (best) based on cloud-cover percentage (CCover) and the spectral and geometric ‘image quality’ (Q). The latter is as qualitative metric between 0 and 9 that grades the number of bad scans in an image⁵³, which we normalized by the maximum possible grade. As cloud cover compromises the reliability of individual pixels⁵⁸, QI is 1 (best) if Q is 1 and CCover is 0, and decreases as CCover increases. Similarly, QI is lower when Q is lower, such as for images acquired by Landsat 7 post 2003 due to the degradation of the sensor.

Within-year variations in data quality. Using QI, we measured the within-year frequency and completeness of the Landsat archive, and identified between-year interruptions in data availability. Here, we only considered images with QI > 0, because images with a QI of zero imply the absence of any data usable for downstream land-surface monitoring applications, such as those discussed in this paper.

First, we counted the number of years without data since 1982 (i.e., the first year when Landsat 4 was operational), capturing inconsistencies in data collection efforts. Second, we summed the QI of all images collected in each year, which weighs down the annual image frequency by the proportion of usable pixels in each image (referred to as ‘quality-weighted frequency’ in this paper), as not all images provide usable data due to clouds or otherwise obstructed visibility or data degradation. Increases in quality-weighted data frequencies over time increase the chance of misinterpreting the long-term changes, such as those related to land use (see ‘Increasing frequencies of quality data miss regional arable-land losses’). Third, we counted the number of months with usable images, which informs on our ability to perceive seasonal change dynamics, such as those driven by regional climate forcing (see ‘Improving seasonal data completeness exaggerates water gains’). Fourth, we estimated the maximum within-year QI across all image acquisitions, which depicts the likelihood of detecting persistent changes between in comparison to previous years (used in ‘Gaps in quality data bias perceived timings of deforestation events’).

Estimating the completeness of the Landsat archive. We quantified the ‘completeness’ of the Landsat archive as the proportion of usable images relative to the number of expectable images when assuming one image every 16 days, or 23 per year, per active sensor. The number of usable images corresponds to the sum of annual, quality-weighted image frequencies.

Analyses of effects of satellite data limitations on perceived land changes

Causal analysis approach. We tested for biasing effects of data-quality on different Landsat-inferred land changes for three case studies, corresponding to three different types of data limitations and three different types of land changes that are expected to be sensitive to those (see below). We tested for causal effects using Convergent Cross Mapping (CCM)³², a technique that can identify and quantify causal links between two variables (time-series), even if the variables are not separable and if links are very weak or non-linear. To achieve this, CCM first constructs co-called “shadow manifolds” of the two variables, which summarize their past behavior over time. CCM then establishes whether there is a causal relationship between the variables by finding corresponding points in the shadow manifolds of these variables (via “cross-mapping”), and testing whether the shadow manifolds “converge”, i.e., whether information from the causal variable has been embedded in the effect variable.

Given that the Landsat products analyzed in this study have varying, and sometimes very short time-series lengths (the GLAD has a length of 5, whereas the TMF and the GSW have lengths of ≥ 30 years), we used a further development of the original CCM method³³, that uses dewdrop regression to combine information from multiple short time-series from similar systems, leveraging time-series information from multiple pixels to identify causal effects. To this end, we ran the CCM analyses for each case study (i.e., the different data limitation biasing perceptions in different land changes) on a country-by-country basis. We could country-level aggregations as this is the typical scale of SDG progress reporting⁵⁹.

Deforestation year analyses. Our case study on annual deforestation patterns is based on per-pixel deforestation and degradation years mapped by the Tropical Moist Forest dataset¹⁹. According to the developers, these data inform on the first change year between 1982 and 2021. To assure comparability with our quality metrics, we aggregated these data from their native 30-m to 1-km resolution, by summing corresponding pixel area.

We then evaluated the ability of the TMF dataset to map deforestation trends as reported by national statistics³¹. Because statistics are reported in 5 to 10-year intervals, we matched the TMF to the same temporal resolution. Then, we quantified changes between consecutive years, and then compared subsequent change magnitudes. Here, if the TMF accurately depicts the evolution of forest change, we would expect that decreases in change magnitudes reported by statistics are followed by similar decreases in the TMF. Using these data, we identified disagreements in the direction of change.

Additionally, we used formal causal analyses (using CCM, see previous section) to analyse effects of variation in the maximum annual Landsat data quality on perceived deforestation timings. Our hypothesis was that abrupt improvements in data quality after extended periods without quality data cause disproportionately high perceived deforestation rates in following years, due to the backlog of deforestation events that accumulated over the data-gap year(s) that suddenly became perceivable. We thus tested for effects of positive/negative outliers in data quality on positive/ negative forest-change outliers, using the data of annual forest-change areas and annual maximum image qualities. For this, we calculated magnitude of differences between annual values and multi-annual trendlines, as derived by Locally Weighted Scatterplot Smoothing (LOWESS)⁶⁰ using a span of 0.4. We focused these analyses on pixel time-series that experienced ≥ 1 instance of forest-cover changes between 1990 and 2020, as well as one or more years without usable data.

Arable-land change analyses. Our case study on arable-land changes is based on a global dataset on the Global Cropland Extent dataset (GLAD)¹⁸, which maps arable-land extents in subsequent four-year epochs between 2000 and 2019. To match these data to our quality metrics, we aggregated the 30-m GLAD data to a 1-km resolution time-series of per-pixel areas of arable-land, and recalculated the quality-weighted image frequencies for the same four-year epochs.

We evaluated the ability of the GLAD product to detect arable-land gains and losses as reported in FAO national statistics on the area of arable-land⁶¹. Specifically, we adjusted the FAO-reported areas of arable-land by subtracting the reported areas of temporary pastures and meadows, following recent recommendations⁶² by FAO data experts that this adjusted metric most closely corresponds to the “cropland” definition adopted by the GLAD product. To compare the FAO statistics to the GLAD data, we first aggregated the annual FAO values to per-epoch maximum areas for each country. We chose maxima because the GLAD product maps any arable-land occurrences perceivable at any points within a given epoch, making GLAD data sensitive to maximum extents. We averaged the quality-weighted frequencies over each epoch, and over each country to match the national scale of the FAO data, focusing only on pixels where the GLAD product maps some arable-land at any point during the full observation period.

Using these data, we tested our hypothesis that differences in quality-weighted Landsat data frequencies cause bias in perceived changes in arable-land extents in the Landsat-based GLAD product, causing disagreements with changes inferred from FAO data. Specifically, we used CCM to test whether national-scale qualitative disagreements between GLAD and FAO data in the directions of arable-land changes between subsequent epochs are causally linked to changes in the quality-weighted image frequencies. To this end, we focused on pixels in each epoch and country where the GLAD product mapped arable-land gains – the dominant source of disagreement between the GLAD product and national statistics (Fig. 3c) – and where we recorded increases in quality-weighted image frequencies.

Surface-water change analyses. We based our case study on seasonal surface-water dynamics on the Global Surface Water dataset (GSW)¹⁷, which classifies annual occurrences of permanent and seasonal water. These data are derived from monthly maps on water occurrences, and seasonal water is distinguished from permanent whenever the presence of water is intermittent. Given the dependency of these data on monthly observations, we compared long-term changes in surface water to annual variations in the number of months per year with ≥ 1 usable Landsat observation (Fig. 3e).

Additionally, we evaluated the plausibility long-term trends in surface-water changes mapped by the GSW dataset relative to those depicted by data on water discharge recorded by gauge stations⁶³. Because upstream surface water limits discharge⁶⁴, long-term changes in discharge are likely to be correlated with true surface-water changes in upstream areas. To allow for comparisons with annual surface water extents, we derived the annual maximum values of discharge for each data year in each gauge station. The maximum annual value conceptually fits the GSW data, which also maps maximum surface water extents.

We quantified surface water extents in each year with available discharge measurements over the hydrological basins upstream of the respective gauge stations, using the spatial information on (sub-)basin extents and their water-flow connections from the HydroSHEDS dataset⁶⁵. Specifically, we quantified the proportions of upstream hydrological (sub-)basin extents that, according to the GSW, were covered with seasonal surface water in each year. Here, we excluded gauge stations placed downstream of dams⁶⁶, which may disrupt the natural links between upstream surface-water extents and downstream river flows. For each of the remaining 2,561 stations (Extended Data Fig. 7a), we then smoothed the discharge and upstream surface water time-series using LOWESS. We then applied Mann-Kendall tests to determine if both surface water and discharge experienced significant change trends (p < 0.05). These tests are sensitive to abrupt changes in variables that are related to data gaps. Therefore, when extracting the surface-water proportions for each (sub-)basin and year with gauge-station data, we excluded any pixels without any usable observations in the respective year.

We used CCM to test out hypothesis that changes in seasonal Landsat data completeness bias perceptions of changes in seasonal surface-water extents, as observing the land surface during more months of a given year increases the likelihood of detecting short-lived seasonal water occurrences and of accurately distinguishing longer-lived seasonal from permanent water bodies. Specifically, we tested for effects of positive/negative outliers in annual numbers of months with usable Landsat data on positive/negative outliers in seasonal surface areas. We calculated the magnitude of differences between annual values and trendline derived with LOWESS. When applying this test for each country, we focused on pixels with seasonal surface water observed at any point during the observation period.

Comparisons between lower- and higher-income countries. We evaluated whether there are asymmetries in both data-quality limitations and in the related biases in perceived land changes between higher- and lower-income regions of the world, as identified in World Bank data⁶⁷. For simplicity, we defined only two income groups of countries, aggregating countries classified as either ‘low-income’ or ‘lower-middle-income’ into a combined ‘lower-income’ group, and those classified as either ‘upper-middle-income’ or ‘high-income’ into a combined ‘higher-income’ group. In all income-groups comparisons, we used a threshold of p < 0.05 to identify significant differences.

For the income-group comparisons of different quality metrics, we compared average per-pixel values across each group’s entire land area (i.e., not distinguishing countries within the respective groups). We report area-weighted mean values for compared quality metrics. Here, we used the Wilcoxon-Signed-Rank⁶⁸ test to evaluate differences in means among each income group for each quality issue. To account for spatial autocorrelation, we matched pixels from the lower-income regions to the spatially closest pixels of the higher-income regions (with replacement) using a nearest neighbour approach⁶⁹, and excluded all non-matched pixels from these tests. Given this matching step, these comparisons consistently involved 167,494,453 and 53,134,895 numbers of pixels in the lower- and high-income groups, respectively.

For the income-group comparisons of bias-causing effects of data limitations on perceived land changes (based on the country-level results of the CCM analyses). Specifically, we used McNemar's tests⁷⁰ (a paired, non-parametric Chi-square test for categorical samples) to evaluate whether the prevalence of Landsat-driven land-change biases identified at country-level differed between lower-income and higher-income groups of countries. To this end, we first classified country-level p-values of the corresponding CCM analyses as ‘significant’ (p < 0.05) or ‘non-significant’ (p > 0.05). As with the previous tests, we used matching to account for spatial autocorrelation, but here matching each higher-income country (with a population size of 133) to its spatially closest neighbour lower-income country (with a population size of 83 countries) based on country centroids. Given this matching step, these income-group comparisons were consistently performed with a sample size of 133, except for our example on forests. This is because the TMF dataset is restricted to the moist tropics. Here, higher-income countries had a population size of 36, compared to 41 for lower-income countries.

Data availability

All data underlying the findings of this manuscript are available through the associated Figshare repository, with the exception of the original Landsat quality metrics, which are provided through a dedicated repository (https://doi.org/10.5281/zenodo.7901148).

Code availability

The code used in the causal analysis of each example dataset is provided within the supplementary material. In turn, the code used to calculate Landsat quality metrics is available through a dedicated GitHub repository (https://github.com/RRemelgado/ltqa).

Acknowledgements

We developed this study using the High-Performance Computing (HPC) Cluster EVE, a joint effort by the Helmholtz Centre for Environmental Research (UFZ) and the German Centre for Integrative Biodiversity Research (iDiv) Halle-Jena-Leipzig. CM acknowledges funding from the Volkswagen Foundation through a Freigeist Fellowship (A118199), with additional support by iDiv through its Senior Scientist program (FZT-118, DFG). We are grateful to Andrea Perino for helpful comments which helped improve the manuscript.

Author contributions

RR and CM designed the study. RR developed the analysis with support from CM and CC. RR developed the data underlying the study and ran the analysis. RR and CM designed the figures. RR, CM, and CC interpreted the result and wrote the paper.

UN general Assembly. Transforming our world: the 2030 Agenda for Sustainable Development. https://www.refworld.org/docid/57b6e3e44.html (2015).
Moyer, J. D. & Hedden, S. Are we on the right path to achieve the sustainable development goals? World Dev. 127, 104749 (2020).
Winkler, K., Fuchs, R., Rounsevell, M. & Herold, M. Global land use changes are four times greater than previously estimated. Nat. Commun. 12, 2501 (2021).
IPBES. Global Assessment Report on Biodiversity and Ecosystem Services of the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services. (IPBES secretariat, 2019).
IPCC et al. Climate Change and Land: an IPCC special report on climate change, desertification, land degradation, sustainable land management, food security, and greenhouse gas fluxes in terrestrial ecosystems. https://wedocs.unep.org/20.500.11822/29261 (2019).
UN DESA. The Sustainable Development Goals Report 2022. https://unstats.un.org/sdgs/report/2022/ (2022).
Xu, Z. et al. Assessing progress towards sustainable development over space and time. Nature 577, 74–78 (2020).
Perino, A. et al. Biodiversity post-2020: Closing the gap between global targets and national-level implementation. Conserv. Lett. 15, e12848 (2022).
Anderson, K., Ryan, B., Sonntag, W., Kavvada, A. & Friedl, L. Earth observation in service of the 2030 Agenda for Sustainable Development. Geo-Spat. Inf. Sci. 20, 77–96 (2017).
Wulder, M. A. & Coops, N. C. Satellites: Make Earth observations open access. Nature 513, 30–31 (2014).
Cracknell, A. P. The development of remote sensing in the last 40 years. Int. J. Remote Sens. 39, 8387–8427 (2018).
Mora & Wijaya, A. Capacity development in national forest monitoring: Experiences and progress for REDD+. https://www.cifor.org/publications/pdf_files/Books/BWijaya1201.pdf (2012).
Zhu, Z. et al. Benefits of the free and open Landsat data policy. Remote Sens. Environ. 224, 382–385 (2019).
Schmidt-Traub, G., Kroll, C., Teksoz, K., Durand-Delacre, D. & Sachs, J. D. National baselines for the Sustainable Development Goals assessed in the SDG Index and Dashboards. Nat. Geosci. 10, 547–555 (2017).
Gregory, S., Hulse, D., Bertrand, M. & Oetter, D. The Role of Remotely Sensed Data in Future Scenario Analyses at a Regional Scale. in Fluvial Remote Sensing for Science and Management 271–297 (John Wiley & Sons, Ltd, 2012). doi:https://doi.org/10.1002/9781119940791.ch12.
Hemati, M., Hasanlou, M., Mahdianpari, M. & Mohammadimanesh, F. A Systematic Review of Landsat Data for Change Detection Applications: 50 Years of Monitoring the Earth. Remote Sens. 13, (2021).
Pekel, J.-F., Cottam, A., Gorelick, N. & Belward, A. S. High-resolution mapping of global surface water and its long-term changes. Nature 540, 418–422 (2016).
Potapov, P. et al. Global maps of cropland extent and change show accelerated cropland expansion in the twenty-first century. Nat. Food 3, 19–28 (2022).
Vancutsem, C. et al. Long-term (1990-2019) monitoring of forest cover changes in the humid tropics. Sci. Adv. 7, eabe1603 (2021).
Wulder, M. A. et al. The global Landsat archive: Status, consolidation, and direction. Remote Sens. Environ. 185, 271–283 (2016).
Sales, V. G., Strobl, E. & Elliott, R. J. R. Cloud cover and its impact on Brazil’s deforestation satellite monitoring program: Evidence from the cerrado biome of the Brazilian Legal Amazon. Appl. Geogr. 140, 102651 (2022).
Prishchepov, A. V., Radeloff, V. C., Dubinin, M. & Alcantara, C. The effect of Landsat ETM/ETM+ image acquisition dates on the detection of agricultural land abandonment in Eastern Europe. Remote Sens. Environ. 126, 195–209 (2012).
Mas, J.-F. & Soares de Araújo, F. Assessing Landsat Images Availability and Its Effects on Phenological Metrics. Forests 12, (2021).
Gorelick, N. et al. Google Earth Engine: Planetary-scale geospatial analysis for everyone. Remote Sens. Environ. 202, 18–27 (2017).
Nagaraj, A., Shears, E. & Vaan, M. de. Improving data access democratizes and diversifies science. Proc. Natl. Acad. Sci. 117, 23490–23498 (2020).
Molder, E. B., Schenkein, S. F., McConnell, A. E., Benedict, K. K. & Straub, C. L. Landsat Data Ecosystem Case Study: Actor Perceptions of the Use and Value of Landsat. Front. Environ. Sci. 9, (2022).
Pasquarella, V. J., Holden, C. E., Kaufman, L. & Woodcock, C. E. From imagery to ecology: leveraging time series of all available Landsat observations to map and monitor ecosystem state and dynamics. Remote Sens. Ecol. Conserv. 2, 152–170 (2016).
Ju, J. & Roy, D. P. The availability of cloud-free Landsat ETM+ data over the conterminous United States and globally. Remote Sens. Environ. 112, 1196–1211 (2008).
USGS. Ramifications of the Landsat 7 Failure: Short-Term Funding Strategies. (2004).
Roy, D. P. et al. Characterization of Landsat-7 to Landsat-8 reflective wavelength and normalized difference vegetation index continuity. Remote Sens. Environ. 185, 57–70 (2016).
FAO. Global Forest Resources Assessment 2020. https://doi.org/10.4060/ca8753en (2020).
Sugihara, G. et al. Detecting Causality in Complex Ecosystems. Science 338, 496–500 (2012).
Clark, A. T. et al. Spatial convergent cross mapping to detect causal relationships from short time series. Ecology 96, 1174–1181 (2015).
Figueiredo, L., Krauss, J., Steffan-Dewenter, I. & Sarmento Cabral, J. Understanding extinction debts: spatio–temporal scales, mechanisms and a roadmap for future research. Ecography 42, 1973–1990 (2019).
Nackoney, J. et al. Impacts of civil conflict on primary forest habitat in northern Democratic Republic of the Congo, 1990–2010. Biol. Conserv. 170, 321–328 (2014).
Keenan, R. J. et al. Dynamics of global forest area: Results from the FAO Global Forest Resources Assessment 2015. For. Ecol. Manag. 352, 9–20 (2015).
Fan, L. et al. The effects of Landsat image acquisition date on winter wheat classification in the North China Plain. ISPRS J. Photogramm. Remote Sens. 187, 1–13 (2022).
FAO. FAOSTAT statistical database. https://www.fao.org/faostat/ (2023).
World Bank. World Development Indicators. World Bank database https://data.worldbank.org/ (2023).
Meng, Z. et al. Post-2020 biodiversity framework challenged by cropland expansion in protected areas. Nat. Sustain. (2023) doi:10.1038/s41893-023-01093-w.
Wang, L., Wei, F. & Svenning, J.-C. Accelerated cropland expansion into high integrity forests and protected areas globally in the 21st century. iScience 26, 106450 (2023).
Michalak, A. M. et al. The frontiers of water and sanitation. Nat. Water 1, 10–18 (2023).
Talbot, C. J. et al. The impact of flooding on aquatic ecosystem services. Biogeochemistry 141, 439–461 (2018).
Yamazaki, D. & Trigg, M. A. The dynamics of Earth’s surface water. Nature 540, 348–349 (2016).
Osima, S. et al. Projected climate over the Greater Horn of Africa under 1.5 °C and 2 °C global warming. Environ. Res. Lett. 13, 065004 (2018).
Justice, C. O. et al. An overview of MODIS Land data processing and product status. Remote Sens. Environ. 83, 3–15 (2002).
Sudmanns, M., Tiede, D., Augustin, H. & Lang, S. Assessing global Sentinel-2 coverage dynamics and data availability for operational Earth observation (EO) applications using the EO-Compass. Int. J. Digit. Earth 13, 768–784 (2020).
Yin, G., Mariethoz, G., Sun, Y. & McCabe, M. F. A comparison of gap-filling approaches for Landsat-7 satellite data. Int. J. Remote Sens. 38, 6653–6679 (2017).
Asare, Y. M., Forkuo, E. K., Forkuor, G. & Thiel, M. Evaluation of gap-filling methods for Landsat 7 ETM+ SLC-off image for LULC classification in a heterogeneous landscape of West Africa. Int. J. Remote Sens. 41, 2544–2564 (2020).
Liu, H. et al. Production of global daily seamless data cubes and quantification of global land cover change from 1985 to 2020 - iMap World 1.0. Remote Sens. Environ. 258, 112364 (2021).
Olofsson, P. et al. Good practices for estimating area and assessing accuracy of land change. Remote Sens. Environ. 148, 42–57 (2014).
Povey, A. C. & Grainger, R. G. Known and unknown unknowns: uncertainty estimation in satellite remote sensing. Atmospheric Meas. Tech. 8, 4699–4718 (2015).
Landsat 7 science data users handbook. http://pubs.er.usgs.gov/publication/7000070 (1998) doi:10.3133/7000070.
United States Geological Survey. Landsat Bulk Metadata Service. https://www.usgs.gov/landsat-missions/bulk-metadata-service.
Wulder, M. A. et al. Current status of Landsat program, science, and applications. Remote Sens. Environ. 225, 127–147 (2019).
Herold, M., Gardner, M. E. & Roberts, D. A. Spectral resolution requirements for mapping urban areas. IEEE Trans. Geosci. Remote Sens. 41, 1907–1919 (2003).
Selkowitz, D. J. A comparison of multi-spectral, multi-angular, and multi-temporal remote sensing datasets for fractional shrub canopy mapping in Arctic Alaska. Remote Sens. Environ. 114, 1338–1352 (2010).
Eberhardt, I. D. R. et al. Cloud Cover Assessment for Operational Crop Monitoring Systems in Tropical Areas. Remote Sens. 8, (2016).
UNFCCC. Adoption of the Paris Agreement, 21st Conference of the Parties. (2015).
Seabold, S. & Perktold, J. statsmodels: Econometric and statistical modeling with python. in 9th Python in Science Conference (2010).
FAOSTAT. Agricultural production statistics: 2000-2020.
Tubiello, F. N. et al. Measuring the world’s cropland area. Nat. Food 4, 30–32 (2023).
Global Runoff Data Centre of WMO. GRDC: Long-Term Statistics and Annual Characteristics of GRDC Time Series Data. https://www.bafg.de/GRDC/EN/02_srvcs/21_tmsrs/210_prtl/prtl_node.html (2022).
Duan, K., Sun, G., Caldwell, P. V., McNulty, S. G. & Zhang, Y. Implications of Upstream Flow Availability for Watershed Surface Water Supply across the Conterminous United States. JAWRA J. Am. Water Resour. Assoc. 54, 694–707 (2018).
Lehner, B., Verdin, K. & Jarvis, A. New Global Hydrography Derived From Spaceborne Elevation Data. Eos Trans. Am. Geophys. Union 89, 93–94 (2008).
Mulligan, M., van Soesbergen, A. & Sáenz, L. GOODD, a global dataset of more than 38,000 georeferenced dams. Sci. Data 7, 31 (2020).
Fantom, N. J. The World Bank’s classification of countries by income. http://documents.worldbank.org/curated/en/408581467988942234/The-World-Banks-classification-of-countries-by-income (2016).
Rey, D. & Neuhäuser, M. Wilcoxon-Signed-Rank Test. in International Encyclopedia of Statistical Science (ed. Lovric, M.) 1658–1659 (Springer Berlin Heidelberg, 2011). doi:10.1007/978-3-642-04898-2_616.
Rubin, D. B. Matching to Remove Bias in Observational Studies. Biometrics 29, 159–183 (1973).
McNemar, Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12, 153–157 (1947).

There is NO Competing Interest.

Supplementarymaterial.docx

Download PDF

Version 1

posted

You are reading this latest preprint version

Limitations in historical satellite archives bias SDG monitoring

Status:

Version 1

Abstract

Figures

Main

Results and Discussion

An uneven history of satellite data limitations

Data limitations bias perceived changes in SDG indicators

Biases disproportionately affect lower-income countries

Future needs: bias corrections, fair product validations, and support to users

Methods

Quantification of satellite data limitations

Analyses of effects of satellite data limitations on perceived land changes

Declarations

References

Additional Declarations

Supplementary Files

Status:

Version 1