Impact Evaluations in Data Poor Settings: The Case of Stress-Tolerant Rice Varieties in Bangladesh
Jeffrey D. Michler Dewan Abdullah Al Rafi Jonathan Giezendanner Anna Josephson Valerien O. Pede Elizabeth Tellman




1]Jeffrey D. Michler
2]Dewan Abdullah Al Rafi
3]Jonathan Giezendanner
1]Anna Josephson
4]Valerien O. Pede
5]Elizabeth Tellman
	[1]Department of Agricultural and Resource Economics, University of Arizona
	[2]Department of Agricultural and Applied Economics, University of Georgia
	[3]International Rice Research Institute (IRRI)
	[4]Earth Intelligence Lab, Massachusetts Institute of Technology (MIT)
	[5]School of Geography, Development, and Environment, University of Arizona


Impact Evaluations in Data Poor Settings: The Case of Stress-Tolerant Rice Varieties in BangladeshCorrespondence to mailto:jdmichler@arizona.edujdmichler@arizona.edu. A pre-analysis plan for this research has been filed with Open Science Framework (OSF): https://doi.org/10.17605/OSF.IO/YE7PVhttps://osf.io/ye7pv. We gratefully acknowledge funding from the Standing Panel on Impact Assessment (SPIA), the Bill and Melinda Gates Foundation (BMGF), and the CGIAR Research Program on Rice. We are especially grateful to Aileen Maunahan, Jorrel Aunario, Pavan Yeggina, and Renaud Mathieu for their work on the early stages of EO data generation, model building, and creating training data sets. We also greatly appreciate the work on Donald Villanueva and Humnath Bhandari in the 2022 data collection effort as well as Donald, Rose San Valentin, and Rowell Dikitanan for initial data cleaning and database construction. This paper has been shaped by conversations with participants at the Center for Environmental Economics and Sustainability Policy (CEESP) seminar at Arizona State University, the 7^th African Conference of Agricultural Economists in Durban, the 6^th International Rice Congress in Manila, and the the 32^nd International Conference of Agricultural Economists in New Delhi. An earlier version of this paper was presented at the AAEA annual meeting in Anaheim. 
 
    [
    August 2024
=================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================



empty


§ ABSTRACT 
     Impact evaluations (IEs) of new technologies are critical to improving investment in national and international development goals. Yet many technologies are introduced at times or in places that lack the necessary data to conduct a well identified IE. We present a new method that combines Earth observation (EO) data, advances in machine learning, and survey data so as to allow researchers to conduct IEs when traditional economic data is missing. To demonstrate our approach, we study stress tolerant rice varieties (STRVs) introduced more than a decade ago. Using 20 years of EO data on rice production and flooding, we fail to replicate existing evidence of STRV effectiveness. We validate this failure to replicate with household panel data and through Monte Carlo simulations that demonstrate the sensitivity of past evidence to mismeasurement. Our findings speak to the challenges and promises of using EO data to conduct IEs in data poor settings.
	

	
	
	JEL Classification: C51, C81, D83, O13, Q12, Q54 

	Keywords: Remote Sensing, Earth Observation, Machine Learning, Rice, Flooding, Bangladesh









§ INTRODUCTION
 

Getting one's hands on a sufficient amount of relevant data has always been a challenge in answering economic questions, whether theoretical or empirical. Karl Marx struggled to obtain firm-level production data for use in developing his theory of value <cit.>. John Maynard Keynes lamented the lack of sufficient data to definitively disprove Arthur Pigou's theory of unemployment <cit.>. Simon Kuznets, in part, won his Nobel Prize for creating standardized and regularly collected data on national accounts <cit.>. One reason why the earliest advances in econometrics were made in the sub-field of agricultural economics was because George F. Warren convinced the U.S. Census Bureau to conduct a separate national census of agriculture at more frequent intervals than the decennial census <cit.>. This meant that agricultural economists had much more data for empirical research than those working on other macro or micro questions.[Nearly as important as having any data is having sufficient data. Well know empirical findings, such as the Philips Curve <cit.> and the deterrence effect of capital punishment <cit.>, have proven much less robust than initially thought once additional data were added to the analysis <cit.>.]

The last quarter century has seen an explosion in the availability of data, economic and otherwise. Data, in combination with the personal computer, the internet, mobile information, and communication technology, has created a data rich world. This plethora of data has allowed economists to answer a host of new questions on anything from the historic effects of colonial institutions <cit.> to the current effects of misinformation <cit.> to the future effects of climate change <cit.>. Economists have also become adept at integrating traditional socioeconomic data with new types of data from a variety of sources. In particular, remotely sensed Earth observation (EO) data has become a favorite for use in economic research <cit.>. Economists have used EO data to examine questions about economic growth, agricultural productivity, land use, population growth, poverty, child mortality, and governance <cit.>.[A cottage industry exists of researchers using EO weather data to predict everything from human capital formation <cit.> to labor markets <cit.> to conflict and institutions <cit.> to agricultural production <cit.> to intra-household bargaining power <cit.> to technology adoption <cit.>.]

Yet the recency of so much of this data means that there is at least one place that remains relatively data poor: the past. As <cit.> show, the increase in the spatial and temporal resolution of EO data has primarily occurred since 2010, with the most substantial improvements (<10m, daily) occurring in the last five years. While EO data has been used to answer many research questions about recent or contemporaneous events, its usefulness in answering economic questions about the (relatively recent) past has, thus far, been limited. The requirements for ground truth or training data to improve the accuracy of EO derived outcomes also places perceived limits on where and when EO data can be used to answer a research question.

In this paper, we develop a methodology to overcome the recency bias in high quality EO data so that it can be used to answer economic research questions in environments that remain - and are likely to remain - data poor. We do this in the context of a long-term, large-scale impact evaluation on the effectiveness of stress tolerant rice varieties (STRVs) in mitigating yield loss to flooding in Bangladesh. The flood-tolerant rice variety, Swarna-Sub1, was first released in Bangladesh in 2010 and other varieties quickly followed <cit.>. While a randomized control trial (RCT) was conducted in 128 villages in Odisha, India in 2011 <cit.> and household panel data on varietal-specific rice production exists in South Asia starting in 2014 <cit.>, no data exist that would allow for a before/after comparison of the impacts of adoption on a large-scale. The relatively long time since the introduction of varieties with the Sub1 gene means that the high spatial and temporal resolution EO data now being used in impact evaluations <cit.> also does not exist for use as a baseline.

In order to conduct our impact evaluation in this data poor environment we combine data from a variety of sources (EO, administrative, household survey), develop new approaches to generating ground truth data, and leverage recent advances in machine learning (ML). Our first task is to build maps for where rice is grown in Bangladesh. The standard approach is to collect GPS data on rice and non-rice plots and use that as training data for an ML algorithm to predict rice area at a larger scale <cit.>. While we have this type of ground truth data for 2021 and 2022, it does not exist for previous years. To overcome this lack of ground truth data, we hand-annotate thousands of Google Earth images to build a training data set going as far back as 2002. Having identified rice area, we next build maps of where rice was flooded. The traditional method for identifying floods is to use an existing databases of large-scale floods based on optical sensors that might be blocked by clouds <cit.> or to handcraft flood measures using EO data <cit.>. Both methods miss many small flood events or have cloud blockage and thus under-measure the extent of flooding <cit.>. Substantial improvements in accuracy can be achieved using ML methods similar to those used in rice mapping but here again the problem is a lack of ground truth data. To overcome this, we use radar data from Sentinel-1, a satellite with high spatial and temporal resolution data not blocked by clouds, launched in 2014, as training data for a Convolutional Neural Network -  Long Short-Term Memory (CNN-LSTM) model that then predicts past flooding in lower resolution historical data <cit.>.

Our first approach to estimating the impacts of STRVs combines these rice and flood maps with the EO derived enhanced vegetative index (EVI), a common proxy for yields, and administrative data on STRV seed release. This produces a panel data set that covers the entire country for the 20-year period 2002-2021. Using a variety of econometric methods (event study, difference-in-difference, two-way fixed effects), each with its own strengths and weaknesses regarding identifying assumptions, we find little evidence that, after the introduction of STRVs, EVI in flooded areas is higher than it was in those same areas before the introduction of STRVs.

Our failure to replicate the positive impacts of STRVs on yields during floods, which have been documented in numerous agronomic trials and in an RCT, sheds doubt on the effectiveness of our EO-driven approach. To assuage this doubt, we validate our approach using household panel survey data that covers the years 2014, 2017, and 2022 and is representative of rice growing regions in Bangladesh.[Prior to the collection of the new round of survey data, we pre-specified our analysis and archived a pre-analysis plan (PAP) publicly on Open Science Foundation (OSF) <cit.>.] We first use a two-way fixed effects (TWFE) model to estimate the impacts of household adoption of STRVs on household rice yields. That the household data all comes post-introduction of STRV means we have no baseline data in which adoption was completely exogenous (as the seed did not yet exist). Recognizing this, we implement a TWFE model with instrumental variables (IV) in which past flood events derived from our flood maps are used to predict STRV adoption. Results from our household-level analysis comport with the evidence from our EO-driven approach: there is little evidence that STRV adoption results in higher yields for the household during flooding relative to similarly severe floods experienced by the same household. Analysis using cross-sectional plot-level data corroborates the household and EO results.

Why then, using three different levels of analysis and a variety of econometric specifications, do we fail to find the large, positive impacts of STRVs on yield that exist in the agronomic and RCT data? One potential reason is that, unlike high yielding seed varieties which create unconditional yield effects, STRVs are a stochastic technology that generates higher order treatment effects. Based on agronomic field trials, the specific technology we study generates measurable treatment effects only when flooding is greater than seven days but less than 17 days. Outside that window, STRVs have yields statistically indistinguishable from non-STRVs. To test this conjecture, we conduct exploratory (not pre-specified) analysis using EO flood data on the exact number of days a region experienced flooding. We also conduct Monte Carlo simulations using the RCT data from <cit.> and <cit.> to examine the sensitivity of those results to injecting noise into the data. This exploratory analysis demonstrates that measuring a statistically significant treatment effect for STRVs is highly sensitive to mismeasurement or mis-classification, be it in the EO or RCT data.

We make three contributions to the literature. First, we contribute to the growing set of geospatial impact evaluations. A geospatial impact evaluation uses georeferenced EO data to measure either the intervention, the outcome, or both <cit.>. This geospatial data can be used as part of an RCT, as in <cit.>, <cit.>, and <cit.>, but is more frequently used as part of a quasi-experimental approach to establishing the causal effects of an intervention. Geospatial impact evaluations have been used to study the impacts of governance, anti-poverty programs, and conditional cash transfers on deforestation <cit.>, and the impact of insecticide treated bednets on child mortality <cit.>, to name a few. Many of these geospatial impact evaluations combine EO data with survey data on specific individual outcomes in specific areas. However, a growing proportion of studies, like <cit.> and <cit.>, rely solely on EO and administrative data. In this paper, we combine both of these approaches, relying solely on EO and administrative data in our first set of analyses and relying on a combination of EO and household and plot survey data in the second set. We believe this multi-pronged approach provides increased reliability in our estimates by providing evidence from a variety of data sources at a variety of scales.

Second, we make a methodological contribution to the broader literature that looks to combine EO data with socioeconomic data. Both the availability and quality of EO data is changing rapidly. So too is the sophistication of model building to use both optical and radar EO data to predict specific outcomes of interest. Early economic studies that integrated EO data typically relied on what <cit.> term “handcrafted features”, such as calculating ratios of reflectances at different wavelength, for use in shallow predictive models. One commonly used example of a handcrafted metric is the Modified Normalized Difference Water Index (MNDWI) <cit.>. The MNDWI is simply a ratio of the differences in surface reflectance such that positive values are inferred to be water and negative values are inferred to be land <cit.>. An example of a shallow predictive model is using a simple linear regression to predict values of pixels that might be missing or obscured by clouds <cit.>. We leverage recently developed techniques from ML to fuse optical and radar data, including models that use the spatial and temporal structure of the data, to improve predictive ability and reduce noise in our estimates of where rice is grown and where flooding occurs. 

Finally, we add to the literature on the impact of agricultural technology by studying a stochastic technology that generates higher order treatment effects. The aim of the first-generation of Green Revolution technologies was to improve on the low yields of landrace varieties <cit.>. Newer generations of high yielding varieties now also embody genetics to allow the plant to tolerate biotic and abiotic stresses <cit.>. This creates a challenge for the researcher trying to measure impact, as the effectiveness of STRVs manifests only within a very specific window and whether an observations fits within that window is often a function of a stochastic event. As our results show, this “Goldilocks Problem” places a limit on the effectiveness of geospatial impact evaluations, particularly in data poor settings. However, as our exploratory analysis and Monte Carlo simulations show, this sensitivity to misallocation, mismeasurement, or mis-classification is not limited to EO data but applies to survey and experimental data as well - a point made by <cit.>, <cit.>, and <cit.> that our paper reinforces.







§ STUDY CONTEXT
 

Green Revolution technologies played a critical role in improving food security and reducing poverty in many developing countries <cit.>. Early in the Green Revolution, many technologies were developed for and disseminated in agronomically favorable environments, bypassing environments where abiotic stresses resulted in low or uncertain yields <cit.>. These earlier technologies often embodied unconditional yield effects. Simply planting them resulted in high yields under existing growing conditions and farming practices, though even greater yields could be achieved when high yielding seeds were bundled with other modern inputs.

Stress tolerant seed varieties, including stress tolerant rice varieties (STRVs), were developed to address the rising challenges of climate change-induced stresses, such as salinity, extreme temperature, drought, or submergence <cit.>. In South and South-East Asia, where seasonal flooding is critical to rice cultivation, farmers face a substantial risk of crop loss due to early and sustained flooding <cit.>. To address this, researchers isolated submergence tolerant traits (the Sub1 gene) in wild rice <cit.> to create Swarna-Sub1 <cit.>. 

Swarna-Sub1 has no yield penalty in the absence of flooding. If and when flooding occurs, Swarna-Sub1 can survive complete submergence of between seven and 17 days, in up to 25 cm of water, with little yield penalty. But, while Swarna and other Sub1 varieties outperform non-STRVs, the yield effect is not unconditional. Figure <ref> traces the yield effect in relation to the duration of flooding. There is a clear “sweet spot“ in which Sub1 and other STRVs produce large yield gains. The effectiveness of STRVs is a function of an essentially random event (the length and intensity of flooding) making STRVs a stochastic technology and meaning attempts to quantify their impact requires measuring a higher order treatment effect. If flooding is outside the 10 day window (less than seven or more than 17 days), then we would expect STRVs to have a treatment effect equal to zero.[Additional details on Sub1 varieties is available in Online Appendix <ref>.]

Recent data collection efforts, including an RCT <cit.> and the Rice Monitoring Survey (RMS) <cit.>, have provided useful information on STRVs, but they each provide a circumscribed picture of STRVs that is limited by the purpose and methodology of each program. While the RCT has strong internal validity for measuring the impact of STRVs, it has weak external validity as it focused on a small set of villages in a single Indian state and as treated farmers were given the seeds. And, while the RMS provides strong external validity for the breadth of adoption, it has weak internal validity as it started collecting observational data four years after the initial introduction of STRVs. Neither source of data can - nor were they designed to - provide evidence on the long-term, large-scale impacts of IRRI's (and other's) investment in developing STRVs nor BMGF's investment in producing and disseminating STRVs.

As such, we are operating in a data poor environment. No historical data exist that provide the necessary information: 1) a panel, 2) observations before 2010 and well after 2010 (to allow time for dissemination and adoption), 3) representative at the national level or at least of rice cultivation, and 4) have data on rice varieties grown and yields.[There are a number of nation-wide panel surveys of households in Bangladesh that might appear to be good candidates for filling this data gap. In Online Appendix <ref> we discuss a variety of available data sets and why they are insufficient to answer our research question.] And current or future data collection efforts will not be useful as STRVs were introduced over a decade ago, meaning no relevant baseline can be constructed.

As an alternative, we use EO data to reconstruct the past. EO data provide a repeated time series for a given location (panel data), provide observations long before and after 2010, and provide coverage of an entire country. What EO data lack are information of varieties grown, which we can proxy for with administrative data on dissemination efforts, and yields, which we can proxy for with EVI. We can also enrich and validate the EO-based analysis using existing - if incomplete - survey data, such as the RMS, which has data of varieties and yields, the very items that the EO data is missing. Although, there is no way to generate the data one would want if one were planning a prospective impact evaluation of a new agricultural technology, our method provides a way for researchers to answer important questions about the impact of technology adoption in data poor settings.

As a test case for our method, we focus on the adoption of Swarna-Sub1 and subsequent submergence tolerant varieties (which we collectively term STRVs) in flood-prone rice-growing regions of Bangladesh during the Aman season. In the Aman season, farmers face more crop loss due to natural disasters like flood <cit.>. The intuition behind our identification strategy is simple: for a given rice-growing location, if no flooding occurs, EVI provides a consistent signal of crop growth up until harvest. For that location, prior to 2010, flooding of x amount would cause a consistent signal response in which EVI values abruptly drop. Post introduction of STRVs, if there is no flooding, there would be no change in the EVI signal relative to before 2010. And, if there is flooding of x amount and no adoption of STRVs, there would again be no change in the signal response relative to before 2010. But, if there is a change in the signal response to flooding, the only explanation would be the adoption of STRVs, as no other technology to reduce the negative impact of flooding on rice was concurrently introduced.







§ DATA


Data availability is the key challenge to establishing the long-term, large-scale impacts of agricultural technology in international development. As <cit.> point out, the relative infrequency of in-person surveys in many developing countries can potentially be overcome using remotely sensed EO. In the case of STRVs in South Asia, where no comprehensive baseline data was collected, EO data allows us to look back into the past and reconstruct outcomes before and after the technology was disseminated.

In this study we combine data from various sources to provide evidence on the impact of STRV adoption from a variety of angles. For our national-level analysis, we combine data from MODIS with two types of ground truth data to build rice area maps. We also use MODIS to extract the enhanced vegetation index (EVI), which is our primary proxy for rice yields. To this we layer on flood maps built through a fusion model developed by <cit.> that combines data from Sentinel-1 and MODIS. Finally, we combine the EO data with administrative data at the district-level on the production and dissemination of STRV seeds by both public and private entities. For our household-level analysis, we combine three rounds of panel data collected as part of the Rice Monitoring Survey (RMS) with historic flood data from the fusion model. We supplement this with cross-sectional, plot-level data also drawn from the RMS.

The combination of these various sources of data provide us with a picture of the impacts of STRV adoption at a variety of spatial and temporal resolutions. While we view no single source of data or econometric approach to analysis as definitive in establishing the causal impacts of adoption, the sum of these various sources and methods provides a clear picture and a robust understanding of the impacts of this technology.






 §.§ Remote Sensing EO Data
 

Our construction of the EO data set starts by identifying where rice was cultivated in Bangladesh. From the time series of rice maps, we extract EVI values. We then construct flood maps for areas where rice was grown. All of this pixel-level data is aggregated to the district-level (there are 64 districts in Bangladesh) and then combined with administrative data on seed dissemination. In this section, we provide a non-technical, intuitive summary of the methods used to generate the EO data. Online Appendix <ref> provides the technical details sufficient to implement our procedures in alternative contexts.





  §.§.§ Rice Area Mapping


As early as <cit.>, rice pixels have been identified by detecting surface water followed by a sudden growth of vegetation. The detection of surface water is critical for identification as it represents agronomic flooding, which is unique in rice cultivation and the most common practice for cultivating rice in Bangladesh.[In recent years there has been a push by IRRI and other development organizations to promote alternate wetting and drying (AWD) and/or direct seeded rice (DSR), both of which would substantially reduce or completely eliminate the need for agronomic flooding. However, promotion of both methods has mainly focused on use during the Boro (dry) season when groundwater irrigation is necessary for cultivation. To date, there is a dearth of reliable data to measure adoption of either technology in Bangladesh. See <cit.> for a recent overview of AWD in Bangladesh.] In early applications of remote sensing, which relied on handcrafted, indice-based features, agronomic flooding is signaled when the value of vegetation indices (e.g., the Enhanced Vegetation Index [EVI]), are lower than the value of water indices (e.g., Land Surface Water index [LSWI]). The sudden growth of vegetation is determined if vegetation index reaches half of the maximum value within 40 days immediately after agronomic flooding.

If our goal was to simply construct rice area maps that were contemporaneous with the RMS household data collection efforts (2014, 2017, and 2022), then we could follow a process similar to <cit.> in validating the maps. This would involve taking GPS measures of the area of a number of rice and non-rice plots and using them as ground truth data for training a machine learning classification algorithm. The algorithm could then use the ground truth training data to predict rice and non-rice areas for the rest of the country for that year. However, our goal is to build rice area maps for the 20 year period of 2002-2021, providing us with a large t in both the pre- and post-STRV release period. No traditional ground truth data exists for predicting where rice is grown during years when no RMS data was collected.

To address this data deficit, we take two approaches designed to complement one other. Our first approach involves collecting traditional ground truth data. In 2021, we conducted a small survey of RMS households to collect plot GPS coordinates, crop varieties, and planting dates (n = 787). There was no socioeconomic component of this survey as it was solely focused on building a ground truth data set for training. This effort was augmented in 2022 by the third round of RMS data collection, which included household socioeconomic information plus plot production information and GPS locations (n = 3,219). All plot location data was collected following guidance in <cit.> and allows for creating rice area maps in 2021 and 2022 following standard procedures as in <cit.>.

Our second approach re-constructs “ground truth” through visual inspection of high resolution Google Earth images. This visual reconstruction using optical imagery is commonly used in the remote sensing literature and is particularly useful for capturing difficult to measure activities, like illegal fishing <cit.>. In our case, it is difficult to rice cultivation, spanning back two decades. To build our historical ground truth data set, we selected three districts in Bangladesh (Barisal, Kurigam, and Rajshahi) that experience different hydrological characteristics and thus different distributions of riceland. We derived grids from the MODIS pixels and selected a stratified random sample of 150 points (75 points each for rice and non-rice areas) in each district for a total of 450 points (see Figure <ref>). This was completed for nine years for which Google Earth imagery exists: 2002, 04, 06, 09, 15, 16, 18, 19 and 21. The MODIS grids were overlaid on Google Earth imagery as the basis for visually determining the land cover of each grid for the particular season being validated. MODIS pixels were categorized as rice if, in the Google Earth imagery, more than 70% of the grid was visually determined to be rice. Pixels were categorized as non-rice otherwise.

We then classify rice areas for all years for the entire country by training a random forest (RF) algorithm on the ground truth data using a leave-one-out cross validation scheme. The input data to the model is a combination of surface reflectance and vegetation indices (specifically EVI) extracted from the MODIS eight- and sixteen-day composites at 250m resolution, as well as topographical indicators (elevation and slope, derived from the Digital Elevation Model (DEM) product FABDEM <cit.>). The RF model consists of 1,000 trees each with five leaves. We use a standard leave-one-out cross validation scheme in which we rotate the districts used as training, leaving one district out each time. This ensures generalizability of the model over space. To ensure generalizability over time, we reserve 2020 as an out-of-sample test set. Our model is accurate at predicting rice and non-rice pixels in 2020 82% of the time.[Online Appendix <ref> contains more details on the methods, including results of accuracy tests for the model.] The end results is a time series of rice and non-rice pixels for the entire country for all years spanning 2002 to 2022.





  §.§.§ Flood Area Mapping


As with the rice maps, building reliable flood maps that look into the past raises data availability challenges. Until recently, two common approaches to using EO data for generating flood maps was either to rely on the historic flood data base curated by the Dartmouth Flood Observatory <cit.> or to use MODIS to construct the Modified Normalized Difference Water Index (MNDWI) <cit.>. 

The archive from the Dartmouth Flood Observatory, which extends back to 1985, has been highly cited but suffers from several well-know data deficiencies. Most importantly, it only includes large flood events that members of the observatory were able to identify in news stories and governmental sources. Many small floods, that might destroy crops for some farmers but result in no deaths and generally do not make news, are not covered in the archive. Recently, <cit.> develop the Global Flood Database (GFD), which uses 12,719 scenes from Terra and Aqua MODIS sensors to produce daily images of flooding at 250m resolution from 2000 to 2018. The GFD builds on the flood event reported in the Dartmouth data and is validated on 30m resolution Landsat data. While the GFD is based on MODIS 250m, it provides only a binary flood/not-flood indicator at that resolution.

In terms of using MNDWI and MODIS, the MNDWI is a threshold method for identifying flooding developed by <cit.>. Water and non-water are identified based on surface reflectance and a pixel is defined as water if MNDWI is above zero. This handcrafted feature has been popular because it requires no ground truth data and is simple to construct. However, as <cit.> point out, methods that combine EO data with ground truth data and methods from deep learning can make substantial improvements to accuracy relative to previous approaches, like EO-only determined thresholds. Furthermore, using MODIS or any optical sensor only to map floods can cause large underestimations in inundation due to cloud cover.

Sentinel-1 provides a numbers of improvements over MODIS and the Dartmouth data that address these two issues (low resolution and underestimation). Specifically, Sentinel-1 comes at a spatial resolution of 10m and uses a synthetic aperture radar (SAR) sensor, which allows it to penetrateins cloud cover. The limitation to Sentinel-1, for our purposes, is that data exists only back to 2017. To overcome the lack of historic Sentinel-1 data, we train a Convolutional Neural – Long Short-Term Memory Network (CNN–LSTM) on Sentinel-1 fractional flood data to predict historic MODIS satellite data and then project back into the past, prior to the launch of Sentinel-1. The method is described in detail in <cit.> and the workflow is summarized in Figure <ref>. 

To detect water in Sentinel-1 data we use a dynamic thresholding algorithm tailored to Bangladesh by <cit.>, which produces a binary flood indicator for the 10m pixel. This is upscaled to create a nearly continuous fractional index at 500m resolution, which matches the resolution in MODIS's eight-day composite images. We then feed this Sentinel-1 derived fractional index, along with hydrologically relevant indices (elevation, slope, and height above nearest drainage), and MODIS 500m eight-day composite data into a CNN-LSTM to predict fractional flood area back to 2002. Relative to other deep learning frameworks, the CNN-LSTM is able to exploit the spatial and temporal structure of the data. The data (features) from a single time step are feed into a first CNN which produces a single pixel of spatially contextualized data. This is repeated for each time step (t) to produce a time series of length T for that pixel. We then feed a time series of length T-1 into the LSTM to produce a single temporal value. The output of the LSTM is then combined with the output of the CNN for time T and feed into a second CNN to produce a fractional value for the extent of flooding in a given pixel for a given time step. As with the RF model for rice area, the CNN-LSTM is trained using leave-one-out cross validation, where a time step is withheld for validation one at a time. <cit.> shows that this method produces accuracy levels comparable to existing flood detection methods using MODIS data and previous methods like MNDWI. And, in our case, our model is accurate 90% of the time.[Online Appendix <ref> provides more details about the fusion model and tests of the model against traditional methods.] The result is weekly data on the fraction of a 500m pixel that was flood, covering all of Bangladesh from 2002 to 2022. 





  §.§.§ Creation of District-Level EO Data Set


The creation of the district-level panel data follows the workflow summarized in Figure <ref>. We start by using the rice area maps for Aman (wet) season for each year to mask the flood maps, so that we only consider inundation levels for rice area. We also use the rice maps to mask EVI data from MODIS so that we only consider EVI values for rice area.

Given the resolution of the data it is not possible to measure EVI on a rice field for use in predicting yields as in <cit.>. The rice maps exist at a spatial resolution of 250m while the mean size of a rice plot in the RMS is 0.68 ha. Nor is it possible to measure the exact, agronomically relevant inundation of a rice field, given that the flood maps exist at 500m resolution. To overcome this data limitation, we calculate a variety of EVI and flood metrics to ensure our proxies for relevant yield and flooding are robust. For EVI, we calculate the maximum value of EVI for the season, the minimum, the mean and median values. All of these metrics turn out to by highly correlated with each other, with correlation coefficients ranging from 0.75 up to 0.99, depending on the specific comparison. Because of this, all of the analysis in the paper is based on our preferred metric, the maximum value of EVI. Results are strongly robust to use of any of the other metrics.

For flood, we calculate four metrics. First, 1) cumulative flooding, which is measured as the sum of the weekly fractional pixel values for the growing season (June to December) for a given year. We also calculate 2) the maximum fractional values for the season as well as the 3) mean and 4) median values. Unlike with the EVI metrics, there is more variation in the flood data meaning the flood metrics are not as highly correlated (though mean and median are highly correlated). As one will see, our results are generally robust to the use of flood measure, though this is not always the case. Because of this, throughout the paper we present results using all four flood metrics.

Having derived our measures of EVI and flooding from the remote sensing time series, we when take the average value of that metric for each pixel within a district. This provides us with district-level values for each of our EO based metrics. We then combine this with the administrative data on seed dissemination. From IRRI and the Bangladesh Rice Research Institute (BRRI), we developed an exhaustive list of all seed production organizations, both private and public. While both IRRI and BRRI have been instrumental in developing submergence tolerant seeds varieties, neither is engaged in seed multiplication, which is conducted by a number of public and private entities. Investment in the seed system helped promote and disseminate STRVs from 2010 through 2019. We visited each seed producer/distributor and obtained disaggregate (district) level data on 1) the STRV variety names being multiplied or sold/distributed in the district, 2) the amount (tons) of each variety being multiplied in the district, and 3) the amount (tons) of each variety being sold/distributed in the district.[Private organizations sell seed while public organizations may sell the seed or distribute for free.] The administrative data provides a proxy for the availability of STRV seeds in each district in each year starting in 2010. A limitation of this data is that farmers may recycle seed or obtain seed from outside their district. We use the cumulative amount of seed available in each district in each year to try and account for the growing availability of STRVs.





 §.§ Household Survey Data
 

To investigate the impact of adoption of STRVs on household welfare, we use data from the Rice Monitoring Survey (RMS). The RMS is a BMGF project designed to capture varietal turnovers over time in Bangladesh, India, and Nepal. The data was originally collected as two waves of a panel in 2014 and 2017 <cit.>. Households were selected following a clustered random sampling procedure to ensure the overall survey was representative of rice growing regions of each country. The plan was to collect a third wave in 2020, but data collection efforts were delayed due to COVID until 2022 and only Bangladesh was revisited. In total, 1,500 households were part of the initial sampling frame in Bangladesh. The RMS was able to follow-up with 1,490 households in 2017 and 1,484 households in 2022. This gives us an attrition rate of 1.7% over a span of eight years. 

As the RMS was designed to monitor varietal turnover, the survey instrument is heavily focused on collecting information on rice production at the crop and plot level. While the data is a panel of households, limited information was collected about household well-being. These include demographics, asset ownership, membership with different farming organizations, farmer opinion on the prevalence of different stresses, such as flood, drought, and salinity, and the availability of different rice varieties. No information was collected on consumption, income, or other measures of welfare, such as food security or women's empowerment.
                
In terms of production data, the survey collected information by both plot and crop so as to be able to distinguish between different varieties of rice grown on the same plot. Plot data includes information on plot size, ownership, and land type. Crop data, within a plot, includes which rice variety was planted in which season, methods of planting, damage of crops by different abiotic shocks, input use, total production, and disposition of the harvest. 
            
We construct an unbalanced panel of households that cultivated rice in Aman season. This gives us a total of 3,865 observations from 1,488 households. Our primary outcome of interest is rice yield at the household level, which we construct by summing up rice harvest (kg) on all plots and dividing by the sum of area (ha) for all rice plot. We consider a household as having adopted STRVs if they report planting any flood tolerant rice variety on at least one plot. To measure the incidence of flooding faced by the household, we combine household GPS location with the EO flood maps we generate.[While we would ideally use plot locations to match with flood data, only household GPS locations were collected in 2014 and 2017. Thus, we only know the precise location of plots in 2022.] We then calculate five measures of flooding for each survey year: cumulative, maximum, mean, median, and a measure of the flood experienced by the village. Our village measure includes cumulative flooding for all households in the village, minus the household in question. We also construct flood time series for each household going back to 2002. In some specifications we use the household's historic experience with flooding to calculate the probability of a household experiencing a flood in a given year as an instrument for STRV adoption.






 §.§ Data Limitations
 

Before moving on to our empirical method, it is important to note and discuss the data limitations in the study. In an ideal situation, researchers would have collected baseline data from rice growing households prior to the release of STRV seeds and would have followed-up with these households periodically over the years. Unfortunately, no such panel data with pre-release baseline exists. This motivates our use of EO data to reconstruct how the EVI signal responds to flooding pre-STRV release and then to examine how that signal changes post release.

In terms of ideal EO data, one would like data from a sensor that is at both a high spatial and temporal resolution and that can penetrate clouds. Additionally, one would like to have ground truth data in terms of both flooding and rice area/yields across both space and time. As highlighted in Section <ref>, such data does not exist, mainly because high resolution, cloud penetrating EO sensors did not exist in the public prior to the introduction of STRVs. Thus, there is no way to construct a baseline at both a high spatial and temporal resolution. Additionally, baseline ground truth data do not exist because no baseline survey was conducted. This poverty of ideal data motivates our approach to constructing rice area maps using Google Earth images and flood maps using a fusion model.

In developing our approach, we prioritized products with a high temporal resolution (MODIS) at the expense of products with a high spatial resolution (Landsat) because of the importance of capturing transitory flood events and EVI values immediately prior to harvest. We are aware of this trade-off and the limitation it creates: our EO data set is constructed at 500m, which is an area equal to 25 ha. In a cropping system like rice in Bangladesh, where the average rice area for a household is less than one ha, this spatial resolution is not ideal. However, considering the types of measurement error introduced by the trade-off leads us to believe that prioritizing temporal over spatial resolution is the preferred approach. We have explored conducting the analysis using Landsat 30m data. However, Landsat's temporal resolution is 16 days, meaning it frequently misses known flood events and EVI readings at harvest are frequently missed. Given this temporal resolution, it is possible to under count by missing a flood or harvest but it is not possible to over count. This means that the measurement error introduced is non-classical and while the direction of the bias is known, the size of the bias is unknown. By contrast, the relatively course spatial resolution of MODIS, combined with the ground truth data, and our machine learning classification approach means that we may misallocate a pixel to rice or no-rice and to flood or no-flood, but that misallocation is likely random. This means the measurement error introduced should be classical - increasing the variance or noise in the data but not introducing any systematic bias. Essentially, the spatial fuzziness of MODIS allows us to make guided inference on whether or not a pixel is rice/non-rice or flooded/not-flooded while the missing temporal images of Landsat do not yield to making inference about what happened in those missing pixels. This spatial fuzziness can be measured in terms of accuracy metrics for our classification algorithms (RF and CNN-LSTM), which are in the range of 80-90% (see Online Appendix <ref> for more details).

In terms of data limitation to the RMS, as adoption of STRVs happen on a plot-by-plot and crop-by-crop basis, the ideal data would be plot-level panel data. Additionally, one would want historic and contemporaneous flood data at a high enough spatial and temporal resolution to measure flooding on that specific plot and not a neighboring plot. Unfortunately, in the RMS, it is not possible to track plots over time. Plot dimensions shift from year-to-year and farmers buy, sell, and rent plots from season-to-season. Additionally, GPS coordinates for plots were not collected as part of the RMS in 2014 and 2017, so we do not know the exact locations of plots in those years. Even if we did, like we do in 2022, the EO flood data is not at a high enough spatial resolution to measure flooding on one specific plot. We do, as a robustness check, conducted our analysis on the pooled plot data and compare these results to those from the household-panel.

Finally, we also lack objectively measured harvest information in the EO data and both harvest and rice area in the RMS data. In the EO data, we use EVI as a proxy for harvest, though this approach is not without its critics. Much of the research demonstrates strong correlation between EVI, and particularly peak EVI, and yields <cit.>. However, the accuracy of using EVI as a proxy for yields varies substantially by location <cit.> and crop, with EVI performing particularly poorly for rice in the U.S. <cit.>. For rice in Bangladesh, the correlation between EO derived metrics and yield can vary from 55-91$ <cit.>. However, <cit.> relied on government reported area yield statistics, a notoriously inaccurate source for accurate and objective measures of yield in low-income countries <cit.>, where statistical departments are often under-resourced and politicized. We are unaware of tests of accuracy of EVI with rice yields in Bangladesh that rely on objective measures of yield.

In the household survey data, we are forced to rely on self-reported measures of rice area and harvest. Since <cit.>, it has become standard to use handheld GPS devices to calculate land area for at least some plots. And since <cit.>, it has become common to conduct crop-cuts on at least some plots. While we did conduct GPS measures of plots in the 2022 round, these methods were not common when the first two rounds of RMS data were collected. As <cit.> shows that correcting for biased self-reporting in one (land area or harvest weight) but not the other can actually aggravate the bias, and as we have objectively measured land area for one year and lack objectively measured harvest in all years, we have decided to rely solely on the self-reported values in all years.

While these data limitations are not unique to our study, and in fact would be present in any attempt to measure impact of an agricultural technology introduced over a decade ago, they should be kept in mind when interpreting our results. 







§ RESEARCH DESIGN
 






 §.§ Pre-Analysis Plan
 

In developing the following empirical approach, we pre-specified our analysis and registered it with Open Science Framework (OSF) <cit.>.[Note that the PAP was registered on 9 February 2023. While in this paper we refer to the third wave of panel data as 2022, this is because we refer to all cropping season data by the year of planting not the year of harvest. Thus data from 2014, 2017, and 2022 all come from the Aman season in which planting occurred in 2014, 2017, or 2022 and harvest at the end of that calendar year, extending into the subsequent calendar year. Data collection for 2022 was not completed until 12 March 2023. Timestamps on the data are available for verification.] While pre-analysis plans (PAPs) are common in lab and field experiments, they remain relatively uncommon for observational studies. However, <cit.> argue that any studying collecting new data, regardless of the type of data, or trying to establish causal effects should develop and register a PAP. They point out that the very first PAP in economics was observational in nature and was created by <cit.>, who was studying the impacts of changes to minimum wage on employment - a particularly contentious issue - and one in which a PAP could be developed ahead of an announced wage increase.

Our study, though observational, is well suited to a PAP given that the research relies on a newly collected round of data and that it uses DID and IV methods to establish causal effects of STRV adoption. In addition to preventing the p-hacking that <cit.> shows can be common to DID and IV methods, the plan also helps limit research degrees of freedom - the ability of researchers to make many reasonable or justifiable decision during the data cleaning and analysis process. In our context, it is unclear what the best EO proxy is for agronomically relevant flooding or for yield. It is also unclear what the best IV for adoption might be. There are also a large number of reasonable criteria to use for deciding how to deal with outliers, how to convert, scale, and aggregate data, and how to deal with missing values. In the absence of a PAP, a researcher might be tempted to try many alternative options in these domains and then only present the choices that result it strongly significant results. Our PAP ties our hands so that we can avoid the temptation to selective report results favorable to our hypothesis.

Throughout the results section, where data definitions, approaches, or inference criteria differ from our plan, we highlight these differences. The most substantial deviation from the PAP is that we initially planned to measure flooding as the total number of distinct flood events. Since we filed our PAP, the research team developed the fusion model described in <cit.> that provides a fractional index of inundation in a pixel. This new approach provides much greater opportunity to develop more precise measures of flooding than the original binary indicator of flooding on a given day. Following advice in <cit.> and <cit.> to not stick to an inferior pre-specified method when a new, superior method becomes available, we adopted the fractional flood index data instead of the binary data. To help alleviate concerns that we selectively report a new flood measure that is particularly favorable to our hypotheses, we report results using four different flood measures based on the new data.






 §.§ Empirical Method
 

Our empirical method follows that laid out in <cit.> to estimate the impact of Green Revolution technologies on yields. We first estimate the effects of the availability of STRV seeds at the district-level on relative rice yields as proxied by EVI with and without flooding. Our empirical strategy relies on the release dates of various varieties and the staggered accumulation of seed in each district, which we argue is plausibly exogenous to rice yields. We then estimate the effects of adoption of STRV seed by a household on relative rice yields with and without flooding. We take a variety of approaches to address the endogeneity of the adoption decision. These include using a fixed effects with-in estimator as well as instrumental variables based on the probability that a households would experience a flood. We enrich these results by also estimating the impact of adoption using the RMS cross-sectional plot-level data.





  §.§.§ EO District-Level Framework
 

We estimate the effects of STRVs on EVI as a proxy for yields using annual data at the district-level. We start be estimating a simple event study model:


    EVI_dt = ∑_j ∈ Tα_j·1_t^t = τ + j + δrice + ϕflood + μ_d + μ_t + ϵ_dt


where d indexes districts and t indexes time. The two terms μ_d and μ_t denote district and time fixed effects, meaning that only within-district time variation in relative EVI remains. The district fixed effects control for all district-specific time-invariant variation and the time fixed effects control for all time-specific district-invariant variation. We also include controls for the percentage of a district that is under rice cultivation and the cumulative flooding in the district, both of which vary by district and time.

We expect the introduction of STRVs in a district to increase EVI relative to that district's EVI before the introduction. To capture this effect in the regression, we include an indicator function 1_t^t = τ + j that takes a value of one j years after the introduction of the first STRV, which we denote as τ. As ϵ_dt captures district-specific trends in relative yields, α_j measures by how much the relative yield in the average district has changed j years after the introduction of STRVs relative to the benchmark year. We set the benchmark year in each district as the year immediately prior to the introduction of STRVs in that district. Our hypothesis is that α_j > 0 after the introduction of STRVs and that α_j = 0 before the introduction of STRVs.

There are two shortcomings to the event study approach. One is that STRVs reach different districts in different years. It is plausible that STRV seeds reached more flood-prone districts, or districts with more advanced farming techniques, earlier than other districts. If these characteristics are time-invariant, our district fixed effects will pick them up. However, one can imagine situations where time-variant, district-specific events accelerated the arrival of STRV seed. Such a situation would be if public or private seed multipliers rushed STRV seed to a district in a year immediately following an unexpected flood event. While we explicitly control for contemporaneous flooding, flood events in the recent past might make the year a district got STRVs no longer exogenous. A second shortcoming is that STRVs are likely to increase EVI only very slightly over time, as a change in EVI would only occur when there was a flood lasting more than seven but no more than 17 days. If a district did not experience substantial flooding in a year after STRVs were released, we would not expect any change in EVI. Thus, the event study will under estimate the effects of STRV release date on EVI as STRVs will only effect EVI if there was a flood.

To address these two issue, we next estimate a simple DID model:


    EVI_dt = β( 1_dt^2010· 1_dt^flood)+ ρ 1_dt^2010 + γ 1_dt^flood + δrice + μ_d + μ_t + ϵ_dt


where 1_t^2010 is an indicator equal to one in the years after the first introduction of STRVs in 2010 and zero otherwise. Similarly, 1_t^flood equals one in districts that are prone to flooding and zero in districts that are not prone to flooding. To define a district as flood-prone or not, we first calculate the median amount of flooding across all districts and all years. We then calculate the median flood value in each district across all years. We categorize a district as flood-prone if its median value is above the median value in all the data. The remaining terms are as defined in Equation (<ref>).[Note that because we have converted both adoption and flood to binary indicators that do not vary over time, our DID estimator is not subject to the recently exposed potential biases of DID in complicated settings. For a summary of what settings complicate DID estimation and the numerous ways to correct for bias, see <cit.>.]

We expect EVI in flood-prone districts to increase after 2010, which was when STRVs first became available in Bangladesh. In this set-up, there are no pure control districts. Rather, we compare the difference in the differences that exist between flood-prone and not and pre/post 2010 to establish a counterfactual. Our hypothesis is that β > 0 while ρ = 0 and γ < 0. The main identifying assumption is that if STRVs had not been released, yields in flood-prone districts would have followed the same trend as yields in non-flood-prone districts. Obviously, flood-prone districts are likely to have lower yields than non-flood-prone districts. Thus we assume a mean shift in yields, as without STRVs mean yields are likely to be lower in flood-prone districts than in districts that are not prone to flooding, though the existence of parallel trends remains unobservable.

As with the event study set-up, the DID model is not without its shortcomings. In particular, in the DID model we assume that farmers in all districts have access to STRVs starting in 2010. This assumption avoids endogeneity concerns about targeting of certain districts for seed but, based on the data, we know that STRVs were not universally available in 2010. In fact, in our administrative data, only one district had seed locally produced and distributed that early. And in the RMS data, adoption in 2014 was at eight percent. This suggests that, like any agricultural technology, roll out, dissemination, and adoption was gradual and took time. Th end results is that our use of a dummy for seed release in 2010, while exogenous, likely overestimates the number of districts “treated” with STRV seed. A second shortcoming is that while a relative measure like flood-prone might drive adoption of STRVs and impact EVI, it is a coarse measure. It is also subjective and may not reflect agronomically important flood events.

Our third approach to modeling the district-level effects of STRV availability is a two way fixed effects (TWFE) approach that seeks to address the shortcoming of the event study and DID approaches. It explicitly accounts for seed availability and actual flooding in the district. The TWFE equation is essentially the same as the DID: 

    
    EVI_dt = β(seed_dt·flood_dt) + ρseed_dt + γflood_dt + δrice + μ_d + μ_t + ϵ_dt.


One main difference is that the TWFE approach uses administrative data on cumulative seed available in each district in each year instead of a binary indicator for if the observation comes before or after 2010. The other main difference is that we use EO data on actual flooding in a given year instead of a categorical variable for if the district is flood-prone or not. All other variables are the same as in the DID model.

The benefit of the TWFE approach is that it uses available data on seed availability and flooding. In the event study, we controlled for flooding but our variable of interest (α_j) measured changes in EVI in each year after seeds were introduce not changes to EVI when seeds were introduced and flooding occurred. This likely underestimates the effect of STRVs. In the DID model, we interact indicators for flooding and the introduction of STRVs but our variable of interest (β) considers all districts as having equal access to seeds in every year since 2010. This also likely underestimates the effect of STRVs. Our TWFE model addresses these issues but it comes with the caveat that if seed availability is a function of time-varying, district-level events it is endogenous. Recall that we control for time invariant district characteristics, like soil quality and climate, as well as district invariant time events, like changes to national policy. So, the administrative seed data will only be endogenous if dissemination changed based on some time varying, district-level event, such as targeting a district for increased dissemination in the year after an unexpectedly severe flood.

Our overall approach to estimating the long-term, large-scale effects of STRVs in Bangladesh is to balance the richness of the EO and administrative data with concerns about endogeneity. The event study and DID approach minimizes concerns about endogeneity by using coarser indicators for the year STRVs were approved in Bangladesh, a clearly exogenous event. The TWFE approach uses data on actual flooding and seed availability, the later of which may be endogenous. By presenting all three approaches we attempt to provide a robust picture of the causal effects of STRVs in Bangladesh.[Only the DID and TWFE approach was pre-specified. The event study approach was suggested by participants at a seminar presentation.]





  §.§.§ RMS Household-Level Framework
  

Similar to our district-level framework, we estimate the effects of STRV adoption by a household on rice yields using a TWFE model:

    
    lnyield_it = β(STRV_it·flood_it) + ρSTRV_it + γflood_it + δland + μ_i + μ_t + ϵ_it


where i indexes households and t indexes time. The two terms μ_i and μ_t denote household and time fixed effects, meaning that only within-household time variation in relative yields remains. The household fixed effects control for all household-specific time-invariant variation and the time fixed effects control for all time-specific household-invariant variation. We also include a control for the amount of land farmed by a household in a given year (land), which, analogous to our district control for rice area, varies by district and time.

In terms of identification, model (<ref>), like model (<ref>) relies on household and time fixed effects to identify changes in a household's yields resulting from a change in the household's adoption status. The model is identified only if the adoption decision is not drive by household-specific, time-varying events. We believe that adoption is plausibly exogenous and primarily driven either by time-invariant household characteristics like risk preferences and farming skill or by availability of seed, which changes over time but is not uniquely for a household. That said, there may be household-specific, time-varying events, like recent flood experience, that makes adoption endogenous in the current model.

To address this endogeneity concern, we estimate a TWFE model with instrumental variables (TWFE-IV). Note that the potentially endogenous variable, STRV, shows up twice in Equation (<ref>): once as a linear term and once interacted with the exogenous flood variable. For instruments, we use lagged values of flooding, and their interactions, for each household in order to calculate the probability that a household experiences flooding in each year of the panel. To instrument for the interaction between STRV and flooding, we use the interaction of lagged flood experience with the contemporaneous flood variable, which being exogenous, serves as an instrument for itself.

The first-stage equation, for either STRV or its interaction, is:

    
    STRV_it   = ∑_j ∈ Tψ_j flood_jt^t= τ - j + ∑_j ∈ T∑_k ∈ Tξ_jk( flood_jt^t= τ - j×flood_kt^t= τ - k)  
       + ∑_j ∈ Tζ_j ( flood_it^t= τ - j·flood_it) + γflood_it + δland + μ_i + μ_t + ν_it.


The first summation captures the lagged values of household flooding, with j representing the number of years of the lag from the panel year, which we represent with τ. We use 12 lagged years, so that the lagged flood data for adoption in 2014 is flooding experience from 2002 to 2013 while the lagged food data for adoption in 2022 is flooding experience from 2010 to 2021. We choose 12 years since 2002 is the earliest year we have EO flood data for. The next summation captures all interaction effects between the different lags. As we do not know exactly what type of flooding is most relevant for a household's decision to adopt, nor do we know how long a past flood may remain relevant for the adoption decision, our goal with the interaction term is to saturate the model by accounting for as much of the mean and covariance of past flood events. Note that ξ_jk captures both the interaction effects from different years (j ≠ k) but also the squared value of a flood (j = k) so as to provide greater weight to extreme flood events in influencing the adoption decision. The final summation term instruments for the interaction between adoption and contemporaneous flooding by using the lagged values of flood as the instrument for adoption and interacts it with contemporaneous flooding, which serves as an instrument for itself. All other terms in the equation are exogenous and serve as their own instrument.

The first-stage equation attempts to account for all relevant past data on flooding based on the meteorological assumption that these past events predict the probability of flooding in the current year. The behavioral assumption underlying our instruments is that households that are more likely to experience flooding in a given year are more likely to adopt STRVs. Beyond the assumption about the relevance of the IVs, these instruments also need to satisfy the exclusion restriction. We argue that past flooding will not affect current yields, conditional on the other controls, except through the choice of a household to adopt STRVs.[Instruments must also the stable unit treatment value assumption (SUTVA). While flooding is clearly covariate across space, SUTVA only requires that the effects of floods be stable across unit. So, if one household experiences a certain level of flooding, its neighbor, who experienced the same level of flooding, will make the same decision and experience the same outcome as the first household.]

After estimating Equation (<ref>), we then estimate the second stage of the TWFE-IV equation:

    
    lnyield_it = β(STRV_it·flood_it) + ρSTRV_it + γflood_it + δland + μ_i + μ_t + ϵ_it,


where STRV_it is the predicted value from the first-stage regression. All other terms are as previously defined. The TWFE-IV model measures the causal effects of a change in yield due to the adoption of STRVs when a household experiences flooding. It controls for time- and household-invariant factors and the instruments control for potential endogeneity in the adoption decision by controlling for household-specific, time-variant unobservables.

In all models we cluster standard errors at the level of the unit of observation. In our results, we present point estimates on the coefficients of interest as well as 95% confidence intervals calculated from the clustered standard errors.







§ RESULTS
 

In the following sections we discuss the results of our two levels of analysis. We begin by summarizing descriptive evidence of time trends in the EO data before moving on to discussing results from the event study, DID, and TWFE models. We then shift focus to the household survey data to see if it confirms results from the long-term, large-scale EO analysis. Again we begin by summarizing descriptive evidence and then move to discussing the econometric results. We then briefly summarize a set of robustness checks, with detailed discussion relegated to Online Appendix <ref> and <ref>. We conclude with a discussion of why our results fail to replicate existing evidence on the impact of STRVs and try to reconcile the differences with a set of exploratory analysis.






 §.§ District-Level Outcomes
 





  §.§.§ Descriptive Evidence


We begin by examining the increase in seed availability over time. Figure <ref> aggregates data from the 64 districts up to the division level. We see that until 2013 there was almost no STRV seed available in any district. Starting in 2014 availability in all divisions began to grow but especially in Rangpur. Availability continued to grow across the country until 2019, when growth began to slow or plateau, except in Khulna. This window of rapid expansion in the availability of STRV coincides with international investment in the seed system to expand multiplication and dissemination of new seed varieties.

Overall, the availability of seeds is clearly heterogeneous across space, with flood-prone regions of the country having greater need and thus greater access to the seeds. That dissemination is not random could create concern regarding the exogeneity of seed availability. However, if seed availability differs across districts based on how flood-prone a district is, this will be differenced out by our within estimators. Only if dissemination responded to year-to-year changes in districts, should we be concerned about endogeneity. Luckily, the growth trajectories in each division are fairly stable, though we do see some shifts in slopes that could be evidence of reallocation of seed in response to a district-specific event. Our inability to conclusively rule out endogeneity in the seed data is why we present multiple estimators that do not rely on the data.

Next we turn to trends in rice area per district and EVI values (see Figure <ref>). In Panel A we plot the share of each district that was planted to rice based on our EO generated rice maps. Along the horizontal axis, we plot these values relative to the year that STRVs were first available in the district, with event year -1 being the year immediately prior to introduction. To this data we fit non-parametric regressions both before and after the date of introduction and include 95% confidence intervals. For rice area, our concern is that households would start to cultivate more flood-prone land once they had STRVs. If more flood-prone land was brought into cultivation after STRVs, that could create a downward bias in estimates of the effect of STRVs as this land would have a greater probability of experiencing prolonged floods that even STRVs could not withstand. Examining the data, there is a slight upward trend in the share of rice area in Bangladesh, though there is no substantial change in that trend before and after STRVs became available in that district. This suggests STRVs have not increased the area planted to rice nor induced expansion of rice area into previously unused flood-prone land.

In Panel B, we also see a steady increase in EVI values over time. There does appear to be a change in the growth trend after STRVs became available, with growth rate slowing down or flattening. This is surprising because our hypothesis is that STRVs should result in a reduction in flood damage to rice which would manifest in an increase in EVI after introduction. One possible explanation is that the slope of the trend line is flattened by early “adopter” districts (those that have had STRV seed for more than five year) and these districts are the most flood-prone. These districts may have below average EVI values regardless of STRV status because of their propensity to experience flooding. So, while the seeds impact EVI for a given district, the average effect on EVI is muted because the most flood-prone districts get “treated” first. Regardless, the descriptive evidence regarding EVI does not appear to reflect an increase in the value as the introduction of STRVs as we would have expected.

Figure <ref> provides evidence on the change in our flood metrics relative to before and after STRV seed became available in the districts. Again, we plot district values and than fit a non-parametric regression to both the before and after data. While each metric has different level values or intercepts, they all have a similar downward slope. There does not appear to be a substantial change in the slope from before introduction to after. It might be surprising that the extent of flooding is decreasing over time, given the increased prevalence of catastrophic weather events due to climate change. However, event years do not correspond to calendar years, so in a year with substantial flooding, like 2017, the food values of districts from this year might end up as event year -1, or event year 5. A second reason is that, since the early 2000s, Bangladesh has made substantial improvements to an embankment system that has helped reduce the area of flooding even when streamflows are extremely high <cit.>.

Summarizing the descriptive evidence, STRVs only became widely available in most districts in 2015, five years after their introduction. Since then, availability has increased substantially in each division, though spatial heterogeneity is present in which regions have greater availability. Beyond that, rice area, EVI, and flood trends in the EO district data are mild. There has been little change in the area planted to rice across the 20 year study period. There has been a slight increase in the value of EVI and a slight decrease in flooding, but neither show dramatic changes in trends after the introduction of STRVs. We conclude that there is little descriptive evidence for the impact of STRVs at a national scale but also that there does not appear to be confounding factors that might bias our estimates of STRV impacts upward or downward.





  §.§.§ Econometric Evidence


We implement three econometric approaches to establishing the long-term, large-scale impacts of the introduction of STRVs on rice production. These are an event study regression, a DID specification, and a two way fixed effect estimator. Each approach has its strengths and weaknesses, so we rely on the overall preponderance of evidence from the three methods in drawing conclusions.

Figure <ref> reports the results of the event study regression.[The event study regression was not pre-specified and was suggested to us by seminar participants.] Using administrative data on the date that a district first had access to STRV seed, we estimate the change in that district's EVI value before and after the introduction date. As we are looking at changes in district-wide EVI values, instead of changes in EVI values where flooding occurred, our event study is likely an underestimate of the effectiveness of STRVs in preventing flood damage. The event study supports our stated hypothesis. Prior to the introduction, EVI values were flat, fluctuating around zero change relative to the year before STRV availability (indicated as -1). In the years after STRVs were available in the district, EVI has increased relative to the value immediately before introduction. The longer the time period since introduction, the greater the increase in EVI. This is consistent with the slow but gradual increase in seed availability presented in Figure <ref>. As the availability and amount of STRVs in the seed system has grown, so too has EVI values. This suggests that STRVs are effective in reducing yield loss due to flooding, as before their introduction flooded rice would have died, resulting in a lower EVI value than after STRV introduction. 

Panel A of Table <ref> presents results from the DID specification. We include an indicator for if the EVI observation is from after 2009, meaning all districts are including in the “treated” group starting in 2010, even if STRV seeds were not available until several years later. As can be seen in Figure <ref>, this is the case for most districts, meaning our DID estimate will also likely underestimate the true effect. Each column in the table represents a regression with one of our four measures of flooding (cumulative, maximum, mean, or median). Our variable of interest is the interaction between the indicator for post-STRV introduction and the indicator for if the district is considered flood-prone using that flood measure of interest. The DID specification provides week support for our stated hypotheses. Point estimates across the four flood measures are consistently positive, though confidence intervals extend below zero. The point estimates on flooding are consistently negative and significantly different from zero. The only hypothesis that is not supported is that the post-STRV indicator would be zero. Absent flooding, growing STRVs should have no impact on yields, though in our results we see EVI is positive and significantly different from zero.

Panel B of Table <ref> presents results from the TWFE estimator. This is our preferred specification, as it incorporates both flood data and seed data from each district in each year. As with the DID results, each column in the table represents a regression with one of our four measures of flooding (cumulative, maximum, mean, or median). Results are similar in terms of sign and significance to the DID results. Flooding significantly decreases EVI while the availability of STRV seed has a very small but non-zero effect on EVI. The interaction between flooding and STRVs is also very close to zero, with 95% confidence intervals centered around zero.

Summarizing the results, we find little evidence that the distribution of STRVs has had a significant impact on EVI, our proxy for yields, at a national-level. Only in the event study regression do we find statistically significant evidence for the positive impact of STRVs on EVI values in the years since STRVs became available in that district. We see marginally positive evidence in the DID specification, where flood-prone districts have higher EVI values after 2010 than before, though these estimates are not significant at standard levels of confidence. Finally, in the TWFE estimation, we see no significant evidence that EVI values are larger when it floods in a district with STRVs than EVI values when it floods in the same district before it had access to STRVs. Considering the preponderance of evidence, we conclude that the increased availability of STRVs has not reduced the loss of rice production due to flooding. This result stands in contrast to existing evidence from experimental data. 






 §.§ Household-Level Outcomes







  §.§.§ Descriptive Evidence


We now turn to the households panel data to see if this more traditional data source supports or contradicts our EO-only approach for measuring long-term, large-scale impacts. First, we provide summary statistics for key variables across the three years of the panel (see Table <ref>). 
The first three columns of the table report means and standard deviations for each variable in each year. The final three columns of the table report p-values on year-to-year comparisons of means using a t-test that allows for unequal variance. Statistically significant differences exist for almost every variable in almost every year-to-year comparison. Yields in 2014 and 2017 were around 3.7 tons per hectare and increased to 4.1 tons per hectare in 2022. Adoption was flat at eight percent of households for the first two years in the panel (in 2014, 111 households adopted STRVs, in 2017, 112 households adopted). Adoption grew substantially in the intervening five years, with 21% (234 households) adopting STRVs in 2022. Rice area has steadily declined over time and while statistically significant, the magnitude of the decline is not very large. In 2014, households had 0.72 hectares in rice, which fell by a tenth of a hectare to 0.62 in 2022. In terms of flooding, by most metrics households experienced the most flooding in 2014, followed by 2017, with the least flooding in 2022. However, in terms of the maximum intensity of flooding, 2022 was the worst. These differences highlight the importance of testing different flood metrics, as the duration (cumulative, mean, median), the extent (village), and the intensity (maximum) all are important considerations for the effectiveness of STRVs.

Table <ref> presents summary statistics by adoption status and for the data set overall. Households that adopt STRVs report higher yields in the years that they adopt compared to households not growing STRVs in a given year.[Households can and do dis-adopt STRVs, though given how low adoption rates were in 2014 and 2017, dis-adoption is much less common that adoption. Recall also that we classify a household as adopting if they use STRVs on at least one of their rice plot. Thus, yields for adopters includes yields from both STRV and non-STRV seeds. See Online Appendix <ref> for plot-level analysis.] The difference in yields between adopters and non-adopters suggests that the adoption decision is not random and may be correlated with household or farm unobservables, such as farmer ability or quality and location of plots. That said, we do not see significant differences in terms of rice area between adopters and not adopters. Surprisingly, we find that non-adopters experience great duration, extent, and intensity of flooding compared to adopters. One would assume households adopt because their plots are more prone to flooding. However, as adoption is not static or permanent, the difference in flood experience by adoption status may be the result of a non-adopting household experiencing an above average flood and then adopting STRVs the next year, when flood values are likely to regress to the mean. This potential time-varying adoption decision in response to flooding means that adoption is likely endogenous and also justifies our use of past flood experience as an IV for adoption.





  §.§.§ Econometric Evidence
 

To understand the household-level impact of adoption, and to verify if they support our long-term, large-scale EO results, we implement two econometric approaches. First is a TWFE model that is the household equivalent to the district-level TWFE model. Second, given endogeneity concerns regarding the adoption decision, we estimate a TWFE-IV model in which the adoption decision is instrumented using lagged values of flooding, and their interactions, to calculate the probability that a household experiences flooding in each year of the panel data.

Panel A of Table <ref> presents results from the TWFE estimator which incorporates both contemporaneous household flood data and an indicator if the household was using STRVs on at least one plot. As with the district-level analysis, each column in the table represents a regression with a different flood metric (cumulative, maximum, mean, or median). To these, we also add a measure of flooding in the village, by calculating the village average of each household's maximum flood value, excluding the household in question. The TWFE results provide evidence that confirms the district-level EO analysis. Flooding significantly reduces yields and the use of STRVs during flooding has no significant impact on yields. The one results from the household data that does not match the EO data is the use of STRVs when it does not flood. We found a small positive impact in the EO data while we find no impact in the household data. This household result conforms to our hypothesis that STRVs should have no effect on yields absent floods.

Our final set of regressions (Panel B of Table <ref>) uses historic flood data as an instrument for the adoption decision. The use of the IV addresses concerns that there may be unobserved transitory and idiosyncratic events that induce households to adopt but are not controlled for by the household and year fixed effects. Comparing the IV results to the TWFE results that do not employ the instruments, we observe more uncertainty in all of our estimates (wider confidence intervals), save for the estimates of the exogenous flood variable. This is consistent with the loss of efficiency in employing instrumental variables. With this loss of efficiency, it is unsurprisingly that we continue to find STRVs, absent flooding, have no significant impact on yields. For STRV's impact on yields during flooding, coefficients are sometimes positive and significant, sometimes negative and significant, and sometimes not significant. Given this mixed evidence, we do not place a great degree of confidence in any one regression result, preferring to assume the mixed results are due to a loss of efficiency in estimation, which can increase both Type I and Type II errors.

The preponderance of evidence from the household-level analysis supports our conclusions from the district-level EO analysis. STRVs have a small or zero effect on yields when there is no flooding. Flooding, absent adoption, significantly decreases yields. Adopting STRVs and experiencing a flood has little to no effect on yields, relative to when households do not adopt and do not experience a flood. These results are all consistent with our EO analysis but are at odds with the existing body of evidence.






§ FAILURE TO REPLICATE
 

The previous results lead to one obvious question: why do we fail to replicate the evidence in RCTs <cit.> and field trials <cit.> on the efficacy of STRVs? Two potential answers exist. First: there is no signal of STRV efficacy in the data because STRVs are not effective outside of a controlled experimental setting. We find this answer unlikely given the large number of observational and correlational studies that also find a positive association <cit.> and the anecdotal reports in the press on the effectiveness of STRVs. That said, we cannot completely discount this answer, as there are no other long-term, large-scale causal studies in other countries to compare against. A second, and to our minds more plausible, answer is that measurement error in our data generates sufficient noise to obscure the true signal. We find this answer particularly compelling given that STRVs are a stochastic technology, dependent on floods occurring within a narrow window. As we are attempting to capture a higher order treatment effect, our analysis may be highly sensitive to mismeasurement and/or mis-classification.

We follow two avenues to explore the possibility that the true signal of STRV effectiveness has been lost in the noise inherent to conducting an impact evaluation in a data poor setting. First, we return to the flood data and work to make a number of improvements to capturing the length and depth of floods. Second, we use RCT data from <cit.> and <cit.> to run Monte Carlo simulations on the sensitivity of their results to mismeasurement or mis-classification. In addition to the following exploratory analysis, we also conduct a number of pre-specified robustness checks. Summaries of those results, along with accompanying tables and figures, are in Online Appendix <ref> and <ref>. These robustness checks confirm that lack of statistically significant impact of STRVs on EVI or yields under a variety of variations to the data and econometric specifications.






 §.§ Improved Flood Metrics
 

Two key challenges exist in measuring the impact of STRVs. The first is a paucity of data, which we address through our approach to reconstructing the past using EO data. The second is the “Goldilocks Problem” or attenuation problem inherent in stress-tolerant seed varieties. For Sub1, any flooding outside the seven to 17 day window drives any impact on yields to zero. Additionally, inundation needs to be sufficiently deep to submerge the rice but not too deep (no greater than 25 cm). So, if our EO approach is to be successful, we need to ensure our flood maps capture the relevant flood event. Our main results are based on our pre-analysis plan, which in turn is based on what we knew was possible for capturing floods in the EO data. Subsequent to our initial analysis, we returned to the data and our fusion model in an attempt to refine the flood metrics. We generate new measures of flooding to more precisely estimate the depth and duration of the flood. 

First, we create new pixel-level data where the value of the fractional index varies based on how we define the baseflow of water. To do this, we draw the distribution of flooding across the entire country for the entire year. We then define a pixel as being flooded if its fractional value is in the top five percent of values and zero otherwise. We repeat this process for every five percentile increment up to the 95^th percentile. The upper panel of Figure <ref> shows the results for all of Bangladesh. When the quantile is set at the top five percent of values, almost nowhere in Bangladesh would be considered flooded. When the quantile is set at the top 95 percent of values, almost everywhere in Bangladesh would be considered flooded. As we have no direct measure of the depth of water, varying the level of water that we consider baseflow allows us to indirectly capture flood depth. Lower quantiles require a higher threshold above baseflow (deeper water) to be considered a flood. Higher quantiles requires a lower threshold above baseflow (shallower water) to be considered a flood.

Second, we create three new flood metrics based on the new data that attempt to more directly capture floods that occur inside or outside the necessary “sweet spot“ for STRVs to have a measurable impact. These metrics are 1) the maximum number of consecutive flood days, 2) the number of days within the seven to 17 day window, and 3) a binary equal to one if the number of consecutive flood days fell within the window. We calculate each of these metrics at each quantile threshold. The lower panel of Figure <ref> graphs the distribution for the number of consecutive days of flooding for each quantile. At lower quantiles, where inundation needs to be extreme to be considered a flood, the duration of a flood lasts very few days. At higher quantiles, where inundation needs to be just slightly above baseflow to be considered a flood, the duration of flooding lasts almost the entire season.

Using the three new flood metrics measured at each of the 18 quantiles, we estimate TWFE models to determine the effect of STRVs on our four EVI measures during periods of flooding. This produces 216 different combinations of the data, which we represent using specification charts in Figure <ref>. Each panel in the figure displays coefficient estimates and 95% confidence intervals on the interaction of STRVs and a new flood metric for a specific EVI measure (cumulative, max, mean, median). Specifications are sorted based on coefficient size from smallest (left) to largest (right). The gray diamonds below the coefficients indicate which of the three measures of flood duration was used in the regression along with which quantile was used to determine depth or intensity of flooding. As one can see, the vast majority of results are statistically insignificant.

Yet there are patterns which are clearly visible in the results. A first cluster of outcomes in the upper left of the specification charts that represent negative and (sometimes) significant effects on EVI. These occur when we have very high thresholds for inundation to be considered a flood and when we focus on the seven to 17 day flood window. A high threshold means that inundation must be extreme to be considered a flood and only the most extreme events would fall within the window in which STRVs are effective. The coefficients are negative when using this data signals that setting such extreme thresholds are likely too high, and what we are capturing in reality are flood events beyond the capacity of STRVs. A second cluster is indicated by the downward sloping diamonds marking maximum number of days at decreasing quantile thresholds. Coefficients from regressions using these data are all statistically insignificant, with coefficient sizes centered around zero. This indicates that just counting the maximum number of consecutive days of flooding, without regarding to the window of STRV effectiveness, is not a useful measure of flooding at most quantile thresholds. Finally, there is a cluster of data in the center right of the charts that represent positive and (sometimes) significant effects on EVI. These results mostly occur when we use a flood indicator that accounts for STRVs' window of effectiveness and when we set a threshold for flooding in the 25^th to 50^th percentile range.

There is a pessimistic and optimistic way to interpret the results in Figure <ref>. A pessimistic interpretation is that, even when we increase the precision of our flood maps, STRVs have no significant impact on EVI. This could be because the technology is ineffective or because or EO methods remain unable to reconstruct the past with sufficient precision to pick up the STRV signal among the noise. This interpretation is pessimistic because it assumes either the method or the technology does not work. The pessimistic interpretation recognizes that there are some significant results for very specific flood measures, but discounts these as outcome occurring at random because we are running so many regressions. A correction for multiple hypothesis testing results in a loss of significance for all coefficients. The optimistic interpretation is that, given sufficient care in defining floods in the relevant way, STRVs have a significant impact on EVI. The problem is not the method or the technology but refining the method to a sufficient degree of precision so as to capture a treatment effect of a technology that only occurs under very specific circumstances. The optimistic interpretation discounts the plethora of null results as the outcome of mis-classification in what constitutes a flood and mismeasurement of the relevant flood window. The small number of positive and significant coefficients is not due to running many regressions but of defining flooding in the appropriate way. In the next section we turn to Monte Carlo simulations to try and assess the believability of the optimistic interpretation by determining just how sensitive results are to mis-classification and mismeasurement.






 §.§ Monte Carlo Simulations
 

Evidence from experimental trials, both agronomic and economic, have reliably show positive effects of STRVs on yields during floods. One benefit of experimental data is that the experimenter typically has substantial control over how the experiment is implemented as well as how data is measured and collected. In trying to understand why we fail to replicate the existing experimental data, and to try and determine if the pessimistic or optimistic interpretation of our specification charts is more appropriate, we conduct a number of Monte Carlo simulations using the experimental data. Specifically, we use data that form the basis of <cit.> and <cit.>. We then, in separate Monte Carlo simulations, add noise to the flood data, the yield data, and the adoption indicator in order to simulate mismeasurement or mis-classification.

We begin by conducting a pure replication of the results in <cit.>, which reports on the findings from the first year of an RCT in Odisha, India. They find consistent, positive and significant impacts on yields for adoption of Swarna-Sub1 during flooding, relative to both classic Swarna and other traditional varieties. This evidence is consistent with the field trial data presented in Figure <ref>. We are also able to replicate these results using the complete two years of RCT data from <cit.>.[Though the two papers rely on the same source of data, the regressions run in <cit.>, which closely match tests for efficacy in the agronomic field trial literature, and not the specifications used in <cit.>. We are not replicating any specific result in <cit.>. Rather we are using the full two years of data to reproduce results that match <cit.> in magnitude, sign, and significance.]

The simulations for mismeasurement in the yield and flood data are conducted in the same way but are completed separately. Here, we describe our method for injecting noise into the yield data but the method applies exactly to the flood data, as well. We start by calculating the mean (μ = 2,214) and standard deviation (σ = 1,587) of yield in the RCT data. We then draw, with replacement, from a normal distribution with mean zero and standard deviation equal to one percent of the standard deviation in the data (0.01σ = 15.87). For each observation of yield, we add these randomly drawn values. We run the regression in <cit.> to estimate the impact of STRVs during flooding on the new yield data, which has the same mean but a larger variance. We repeat this 10,000 times. We then increase the noise by one percent, so that the new distribution that we draw from has mean zero and a standard deviation equal to two percent of the original standard deviation (0.01σ). We again rerun the regression for 10,000 draws from this new distribution. We conduct this process starting at one percent intervals from 0.00σ up to 0.50σ.

Results for both yield and flood are presented in the top two panels of Figure <ref>. In this graph, we plot the distribution of p-values on the interaction of STRV adoption and flood for each level of added noise (from 0.00σ to 0.20σ). The vertical dashed red line marks p < 0.05. In the top column for both panels, all p-values from the 10,000 simulations are statistically significant. This is because we are just replicating <cit.> and <cit.> 10,000 times. When we add 0.01σ to the true yield data, 99% of p-values are still significant. But just adding two percent of the true standard deviation as noise to the yield data results in only 21% of regressions producing significant results. Adding any more than nine percent of the true standard deviation to the true yield data, so that the yield data is distributed with mean μ and standard deviation σ + 0.09σ results in only five to 10 percent of significant p-values: the same amount of p-values one would expect to be significant due to random chance with a 90% or 95% critical value. Results are similar for flooding, but the loss of significance happens even quicker, with only six percent of p-values significant after adding just 0.02σ to the flood data.

We follow a similar process for simulating mis-classification in the adoption data. However, as adoption is a binary indicator, it is not useful to use the value of standard deviation to add noise to the data. Rather, we calculate the percentage of adopters (42%) in the data and then re-assign adopters and non-adopters at random. So, for one percent of the data we draw at random four households, determine if they were an adopter or non-adopter, and then switch their adoption status. We repeat this 10,000 times. As before, we start the process at zero, so that we replicate the original results, and then move upwards at one percent intervals until we are re-assigning half of all observations.

Results from the mis-classification simulations are graphed in the bottom panel of Figure <ref>. Clearly results are much more robust to mis-classification of adoption status. Mis-classifying up to five percent of observations has no meaningful impact on the outcomes. Mis-classifying 14% of outcomes results in about half of all regressions producing significant results. Only if one mis-classified 37% of observations does the share of significant p-values fall into the five to ten percent threshold, where their significance could be wholly ascribed to random chance.

The take-away from this exercise is that even results on the efficacy of STRVs using experimental data can be very sensitive to mis-measurement (though less so to mis-classification). Perturbing the true yield data by adding to each value a random number drawn from a distribution 𝒩(0,31.75), or two percent of the standard deviation, produces non-significant results in nearly 80% of regressions. Significance goes away adding even less noise to the flood data. In terms of what these means for our failure to replicate, the Monte Carlo simulations add support for the optimistic interpretation. STRVs are effective during floods and we can measure that effect using a geospatial impact evaluation that relies on EO data. But the method requires extreme care and precision in accurately capturing the variables of interest. The acute need for precision is likely the result of trying to measure higher order treatment effects from a stochastic technology.







§ CONCLUSION


There are numerous research questions that economists would like to answer but are unable to because of a lack of data. Recent years have seen a proliferation in the availability of data, in particular remotely sensed earth observation (EO) data. This has allowed economists and other quantitative researchers to be able to answer questions that were previously viewed as unanswerable. However, there is a recency bias in these new data, meaning the past is still a place that remains data poor.

Our study details a new method for overcoming this recency bias in EO data and with this method we conduct an impact evaluation in a data poor environment. To demonstrate this methodology, we use the case of STRV dissemination and adoption in Bangladesh. We combine data from a variety of sources and use innovative approaches to generating ground truth data. Using recent high resolution EO data and deep learning algorithms, we are able to infer where flooding occurred in the past and generate EO data sets at a higher accuracy than previously existed. We combine these new flood and rice area maps, which cover the entire country of Bangladesh for 20 years, with administrative data on seed dissemination efforts in order to conduct a large-scale, long-term evaluation of the effectiveness of STRVs.

Surprisingly, we do not find evidence that STRVs significantly increase EVI values. We confirm this failure to replicate long-standing results from experimental research by using a three year panel of household survey data and more traditional methods for economic impact evaluations. We hypothesize that our failure to replicate is due to the stochastic nature of STRVs, which require flooding of a very specific amount if the seeds are to produce treatment effects greater than zero. We examine this hypothesis by exploring if improvements to our flood metric in terms of duration and intensity. We find that we can produce positive and significant results but only for a narrow and specific set of measures. We buttress this evidence by conducting Monte Carlo simulations using existing experimental data to show just how sensitive results are to mismeasurement.

This paper demonstrates the possibility and challenges in conducting impact evaluations in data poor settings. It also reveals the added dimensionality of the challenge when trying to capture higher order treatment effects associated with a stochastic technology. As the recent economic and geospatial literature shows, and this paper reaffirms, the mismeasurement problem in data can be acute, often biasing or obscure a true signal. However, the methods developed and deployed in this paper open the possibility to answer previously unanswerable economic questions, allowing us to reduce the number of places that remain data poor, including the past.





chicago 




























 

















  





 











	

§ ONLINE-ONLY APPENDIX TO “IMPACT EVALUATIONS IN DATA POOR SETTINGS: THE CASE OF STRESS TOLERANT RICE VARIETIES IN BANGLADESH”
 
















§ SUB1 VARIETAL DEVELOPMENT AND DISSEMINATION
 

In the paper we briefly summarize the development, agronomic characteristics, and dissemination of STRVs. In this is appendix we expand on those details and provide additional evidence regarding the when and how STRVs will generate non-zero treatment effects relative to non-STRVs.

The process of developing high yielding STRVs began when researchers identified a rice variety in India named Dhalputtia, which, despite its poor grain quality and yield, possesses an unusual ability to survive complete submergence for over 14 days. In 1980, scientists identified the Sub1 locus and its associated gene, which subsequently led to the isolation of the Sub1 gene <cit.>. This process of genetic mapping began with two parent varieties: a japonica rice from California and a submergence-tolerant line derived from the donor variety Dhalputtia. 

All STRVs exhibit a similar cellular and molecular mechanism. The Sub1 sequence closely resembles a protein that functions as a transcription factor[Transcription factors are proteins that regulate genetic expression by binding to specific DNA sequences <cit.>]. This transcription factor facilitates the accumulation of the ethylene hormone in response to submergence stress. The accumulation of ethylene also triggers the production of gibberellic acid. The ethylene hormone is critical for the plant's vegetative growth, while gibberellic acid promotes the elongation of plant shoots. During complete submergence, however, shoot elongation is reduced. Additionally, other plant metabolic processes, such as carbohydrate consumption and chlorophyll breakdown, are slowed down, activating alternative energy pathways. As a result, the plant enters a “hold your breath” state, conserving energy until the flooding subsides <cit.>. In contrast, normal rice varieties attempt to elongate their stems and leaves to escape deep submergence. This strategy often results in excessive energy expenditure, leaving the plant unable to recover once the submergence is over. So, the flood-escaping strategy for conventional high-yielding rice varieties differs from that of STRVs, making the latter more adaptable and economically viable <cit.>.

Having isolated Sub1 and understood how it conveys submergence tolerance, researchers at IRRI were then able to use a combination of traditional cross-breeding methods and marker assisted backcross breeding to create Swarna-Sub1, a hybrid with the popular high yielding Swarna rice variety <cit.>. <cit.> have estimated that there is no yield penalty for STRVs under normal condition but a yield advantage of around 1 to 3 t/ha during flooding situation. Additionally, <cit.> highlight that the Sub1 cultivar out performs after complete submergence at reproductive stage compared to traditional varieties in terms of carbohydrate quantities and total dehydrogenase.

Numerous studies have been conducted to assess the yield of STRVs at various stages of their growth cycle. Researchers have experimented with different levels of submergence, including variations in duration and water depth, yielding distinctive results. For instance, <cit.> report that after five days of complete submergence, Swarna-Sub1 rice achieved a yield of 3.98 t/ha, compared to 2.68 t/ha for the traditional Swarna. When submergence extended to 7 to 9 days with water depths of 6 to 9 feet, traditional rice varieties failed to produce any yield. Conversely, the Sub1 variety experienced only a 10-30% reduction in yield. 

In a separate study, <cit.> found that brief, late flooding during the panicle initiation stage could diminish the yield of all traditional rice varieties to 1.7 t/ha, whereas the Sub1 variety yielded 4.75 t/ha. However, if flooding persisted for more than 10 days at this stage, the average yield of the Sub1 variety dropped to 3 t/ha. This observation suggests that while mortality during the reproductive stage is not a significant concern, maintaining yield remains a critical issue <cit.>.

Given the success of the Sub1 varieties, the Bill and Melinda Gates Foundation (BMGF) supported production and dissemination of the seeds under the IRRI-Africa Rice collaboration Stress-Tolerant Rice for Africa and South Asia (STRASA) program <cit.>. As part of the dissemination effort, BMGF funded, through a variety of initiatives, both an RCT on STRV impact in Odisha (2011-2012) <cit.> and the Rice Monitoring Survey (RMS) in Bangladesh, India and Nepal (2014-2017) <cit.>. The RCT produced evidence of the impact of STRVs in the 64 treated villages for farmers who were given seed. STRVs reduced downside risk and resulted in farmers investing more in production through increased cultivation area, fertilizer use, and credit utilization <cit.>. The RMS produced evidence of the extent of adoption in South Asia. Farmers who experienced flooding in the previous year were more likely to adopt and the adoption impact was larger for neighbors of early adopters <cit.>. There are also numerous studies coming out of the STRASA program in Asia and Africa that examine the correlation between adoption of STRVs and various outcomes, such as yield, profit, and rice consumption. A non-exhaustive list includes <cit.>.














§ WHY NOT OTHER DATA
 

In the paper we make the claim that no socioeconomic data exists to allow for a traditional impact evaluation of the long-term, larve scale effects of STRV adoption in Bangladesh. In this appendix we provide evidence to support that claim by discussing the existing nationally or sub-nationally representative surveys, along with their strengths and weakness for answering our research question.

We searched through government, non-government, and university databases to collect a set of possible socioeconomic survey data that would allow us to conduct a strongly identified impact evaluation of STRVs. Recall, our criteria encompass the following prerequisites:

    
        
  * Needs to be panel data
        
  * Needs to have observations before and after 2010
        
  * Needs to be representative at the national level or at least of rice cultivation
        
  * Needs to have data on rice varieties grown and yields
    

Many candidate data sets satisfied none or only one of the criteria. Frequently a data set would have multiple observations over time but not contain information on rice variety, the critical data need. Only a few satisfied three of the criteria and none satisfied all four. Te best three data sources, other than the RMS data we used in this study, are:


    
  * Village Dynamics in South Asia (VDSA): This dataset is an outcome of a research initiative by the International Crop Research Institute for the Semi-Arid Tropics (ICRISAT).[https://vdsa.icrisat.org/vdsa-database.aspxhttps://vdsa.icrisat.org/vdsa-database.aspx]. Spanning 42 villages in Bangladesh, the dataset comprises survey data from 1,831 households. The survey is designed to be representative of agricultural households in Bangladesh, is a panel, and has data on varieties and yields. However, there is one critical limitation:
        
            
  * The temporal scope of the data panel is limited, spanning from 2009 to 2014, in contrast to the RMS dataset, which extends from 2014 to 2022. Given that seed availability remained very low as late as 2015, the VDSA does not extend far enough past 2010 to capture adoption. In fact, adoption of STRVs in the 2014 round of the VDSA is 0%, making the VDSA unusable for studying STRV adoption.
        

    
  * Bangladesh Integrated Household Survey (BIHS): This survey was implemented by the International Food Policy Research Institute (IFPRI).[https://dataverse.harvard.edu/dataverse/IFPRI/?q=BIHShttps://dataverse.harvard.edu/dataverse/IFPRI/?q=BIHS] It is nationally representative, a panel, and contains a host of information on plot-level agricultural production and practices, dietary intake of individual household members, anthropometric measurements (height and weight) of all household members, and data to measure women’s empowerment in agriculture index (WEAI). However, it faces the same limitation as the RMS data set:
        
            
  * The panel is constrained to three rounds of data (2011/12, 2015, and 2018/19), all post-STRV introduction. The absence of pre-introduction data limits our capacity to evaluate the impact of STRV adoption in the same ways as the RMS.
        

    
  * The Bangladesh Rice Research Institute-Rice Database (BRRI-RD): As a government-owned rice database it provides comprehensive data across all years and districts regarding total production[https://bit.ly/47McOjIhttps://bit.ly/47McOjI]. It is nationally representative and contains production information. Nevertheless, this dataset is encumbered by certain limitations:
        
            
  * While a panel, it is not a micro-level panel (plot, household, farm) but constructed as an aggregate district or division-level panel. Thus, it only has data on total production and acreage data by divisions and districts of Bangladesh. Thus, there is no variety-specific production, which restricts the use of this data set.
        
















§ REMOTE SENSING DETAILS
 

In the paper we provide a non-technical, intuitive summary of the methods used to generate the EO data. In this appendix we provide the technical details sufficient to implement our procedures in alternative contexts or point to the existing publications, data, and code to replicate the deep learning and fushion model work.






 §.§ Rice Area Map


In order to train the rice area mapping algorithm, ground truth data first needed to be generated. Three districts were selected to be the base for the study to extrapolate to the country: Barisal, Kurigram, and Rajshahi). The three districts come from the north, central, and south of the country and experience substantial flooding.

Google Earth imagery were inspected as the basis in determining the land cover of each cell. Pixels were categorized as rice if more than 70% of the cell was rice and categorized as non-rice otherwise. In total 150 points were in each district (75 rice, 75 non-rice). This generated 450 rice-no-rice (RNR) points for each year (2002, 2004, 2006, 2009, 2015, 2016, 2018 -  2020) or 4,050 RNR points as a training data set (see Figure <ref>).

We developed a RF model using the training data generated above and used a leave-one-out cross-validation scheme. The input data of our model is: a PCA is applied on the median band values, calculated from the time series covering the growth period of Aman rice. The two principal components are used as input to the model. Additionally, we use the median, 5^th and 95^th quantile EVI values over the growth period for each pixel, as well as elevation and slope as static features.

In summary, we apply the following process in Google Earth Engine to prepare the data before applying the RF algorithm:


    
  * Extract the MOD09Q1.061 Terra Surface Reflectance 8-Day Global 250m data (band 1 and 2) images over the Aman rice growth period.
    
  * Calculate the median value for each pixel.
    
  * Compute the PCA over the bands.
    
  * Extract the time series of EVI from MOD13Q1.061 Terra Vegetation Indices 16-Day Global 250m, and compute the median, 5^th and 95^th quantiles over the period for each pixel.
    
  * Extract the elevation from FABDEM <cit.>, and compute the slope for each pixel.
    
  * Stack all layers together to create the input to the RF.


The RF model consists of a 1000 trees, with each tree having a minimum of 5 leaves. We used the Statistical Machine Intelligence and Learning Engine (smile) RF implementation of Google Earth Engine. The random forest is trained with a leave-one-out cross validation scheme, with an additional separated test set. The leave-one-out scheme consists of rotating the districts used for training and leaving one of the districts out each time. The data used here consists of all years except 2020, which is removed and kept as test set. The leave-one-out validation scheme ensures the model is generalizable in space, meaning that it performs similarly outside of the area used for training. The hold out test set of 2020 allows us to validate the model in time, meaning that it will perform similarly outside of the observed years. The accuracies are reported in Table <ref>.

This way of validating and training the model naturally creates as many models as leave-one-out sets. To run the inference on the whole country, we implemented a majority voting scheme, i.e. each model infers the presence/absence of rice for the whole country, and the category with the most votes receives the final classification. This means that here, with three regions, we have three models, and each pixels needs at least two of the models to classify the pixel as rice, for the pixel to be set as rice in the final map.

After reviewing the result, and comparing them with a study performed by IRRI in 2012 <cit.> on Aman rice presence for 2010, we realized that submergence-rice, i.e. rice irrigated by submergence, was not being detected by our RF. According to <cit.>, submergence-rice is only present in the north east of Bangladesh, and our ground sampling did not covering any of the north east, the algorithm could not pick up on it. We thus decided to add the submergence rice area from <cit.> as additional training data and extended the time frame used in the training data at point 1 and 4 to capture post-flooding periods. This allowed us to successfully capture the submergence rice in the northeast.
After generating the rice maps for all of Bangladesh with MODIS, we had to investigate the necessity of generating them with Landsat. The same process as described at the district level cannot be reproduced: the satellite images were hand-picked for each of the three districts, as cloud free as possible, and representative of the growth period. This process cannot be automated at the country scale. Additionally, because of the technical challenges of working with Landsat 7 (gap filling due to SLC failure), it made sense to investigate the necessity to generate such data before producing it.
In this study, the rice field data is primarily used to mask the inundation and EVI data, but is then aggregated at the district level. Additionally, the data being masked is given at 500 meters (inundation), and 250 meters (EVI). In order to understand the necessity of the higher resolution map, we downloaded a high resolution map based on Sentinel-1 (10 meters) generated by <cit.>, of all of Bangladesh for 2017. We then masked the inundation data with this high resolution mask, and with the 250 meters mask generated through the RF algorithm, and compared the two at the district level.

The figure shows that the two datasets do not significantly differ from each other. Given the complexity of generating high resolution masks with Landsat, the source of errors would be relatively high, and would not significantly improve the accuracy of the data at the district level.



 §.§ Flood Map


As with the rice maps, building reliable flood maps that look so far back into the past raises data availability challenges. While Satellite data allows for analysis geographical areas, particularly in remote or inaccessible regions it suffers from a recency bias, at least in terms of high spatial and temporal resolution products that can see beyond the visible spectrum. For flood mapping, MODIS is often used because of its long history, its availability on a daily basis, and its relatively cloud free image derived from an 8-days composite image. Products like Landsat, which come at a higher spatial resolution and have a longer time series, do not provide such a consistent image series. In particular, Landsat has a low revisit frequency, harmonization issues between Landsat versions, and instrument failure in Landsat 7.

However, MODIS is not without its own limitations. It is an optical sensor that acquires two images daily. But, being an optical sensor, the content of these images is often obscured by clouds. This means that during the tiime of the year when floods are most frequent (the rainy season), MODIS is at its least useful because cloud cover often makes it difficult to see flooding. Furthermore, MODIS's coarse spatial resolution (250m, 500m, and 1km) compared to LANDSAT, makes it unsuitable for accurately extracting flooded information in urban regions or other complex and heterogeneous landscapes. For us, studying the flat rice growing plains in rural Bangladesh, this last issue is not a major concern.

In recent years, radar sensors can overcome the limitations of optical sensors by penetrating cloud cover to consistently detect water signals. Sentinel-1 is an ideal satellite for flood detection as it captures imagines at a high spatial resolution (10m) and is equipped with a radar sensor so that even in cloudy conditions it can capture accurate and consistent flood events. However, Sentinel-1's weakness, relative to MODIS and Landsat, is that it only became available since 2017 and is thus unsuitable for historical flood mapping.

Given the strengths and weaknesses of each product, we first ran comparisons of food detection between MODIS and Landsat. While Landsat has greater spatial resolution, which is extremely beneficial in pinpointing floods on small rice plots, its coarse temporal resolution was a critical detriment. Landsat often completely missed flood events detected by MODIS because of Landsat's infrequency of images and the extent of cloud over. Because of this, we decided to use MODIS as are base product for both flood detection and rice mapping.

Having settled on MODIS, we then developed a method to fuse together the shorter time series of Sentinel-1 with the longer MODIS time series as a potential solution to the disadvantages of either sensor. This fusion model allows us to leverage the advantages of high resolution modern flood detection and historical data availability. The data used for building our flood maps are the product of a deep learning model trained on Sentinel-1 derived fractional inundated area to predict historical MODIS satellite data. We utilize a Convolutional Neural - Long Short-Term Memory Network (CNN-LSTM) to take advantage of both the spatial and temporal feature representation of inundation and compare it to a traditional Convolutional Neural Network (CNN). We apply the CNN-LSTM model from 2002 to 2020 over all of Bangladesh at 500m resolution. From this inundation dataset, we extract one map per year with various statistics over the monsoon season: mean, median, max, min, quantiles, day of max and cumulative flooding.

Figure <ref> provides tests for accuracy of the fusion model by comparing results with the high resolution Sentinel-1 data. Our model produces results that match the high resolution data with a high degree of accuracy, ranging from .79 to .92. Additional details and validation tests, along with reproducible data and code, has been published by the research team in <cit.>.




























§ ALTERNATIVE EO RESULTS
 

In the analysis relying on EO data, we are concerned about two things. First, that our choices regarding key variables (EVI, flooding) are driving the results and that, had we made different choices, our results would not be significant. Second, that our choice regarding the unit of analysis (districts) is driving the results and that, had we conducted the analysis at a more dis-aggregated level, our results would not be significant. To address the first concern, we re-run all of the district-level analysis using three measures of EVI (cumulative, maximum, mean) that differ from the one presented in our main results (median EVI).

As one can see in this appendix, results are not substantially different when we use one of these alternative EVI measures. We also re-run the DID analysis using various cut-offs for flood-prone. The variable of interest in the DID specification was not significant in our main results, and we see this is true for most other definitions of flood-prone. There is, in fact, one definition of flood-prone that produces significant result, but this is one of 42 possible definitions that we test.

To address the second concern, we re-run all of the analysis using upazila (county) as the unit of analysis.[There are 64 districts in Bangladesh and these are divided into 553 upazilas. We combine many extremely small upazilas in the city and metro area of Dhaka, where rice is not grown, to get a final data set of 503 upazilas.] The caveat with using upazilas is that the administrative seed data is at the district-level, so all upazilas in a district share the same year of introduction and same availability of STRV. The event study results are not robust to the choice of the unit of analysis. This may be because while EVI varies from upazila to upazila, the event year is the same for every upazila in the district and we do not consider the degree of flooding within an upazila. When we estimate the DID and TWFE models, upazila-level results reinforce to the district-level results.















































§ PLOT-LEVEL RESULTS
  

In the analysis relying on household data, we are concerned that our choice of the household, and not the plot, as the unit of analysis may be driving our results. If anything, conducting the analysis at the household-level is likely to create a downward bias, as we consider a household to have adopted if they grow STRVs on at least one plot, meaning many non-STRV plots cultivated by a household are considered “treated” even though they are not. Because of this, our preference would be to conduct a plot-level analysis. However, we cannot track plots over time in the household panel, meaning we must treat the plot data as a repeated cross section. 

We start by presenting summary statistics at the plot level which mirror the household level summary tables in the paper. Using plots as the unit of analysis, and controlling for household and year fixed effects, results are generally robust. In five of the eight regressions across both TWFE and TWFE-IV estimators, we reject that yields are equal for plots that experienced flooding and were planted with STRVs and plots that experienced flooding but were not planted with STRVs. These results should be interpreted with caution, as we cannot compare a plot with itself, meaning plot-level unobservables are left uncontrolled for.




















