Climate Analytics: GOES-16 and Aviation Contrails Detection

Objective: Develop a model using satellite and aircraft data to predict contrail formation, number, and duration in specific areas.

---

GOES-16 Satellite: Earth Monitoring

GOES-16 provides continuous monitoring from its geostationary orbit 22,300 mi above Earth.

Contrail clouds, which originate from aircraft exhaust, play a pivotal role in global warming by modulating Earth's radiation balance European Geosciences Union, 2019. This modulation is quantified by effective radiative forcing (ERF), which represents the net energy flux alteration at the top of the atmosphere (TOA).

The relationship between the TOA energy imbalance $\Delta N$, effective radiative forcing $F$, climate feedback $\lambda$, and global temperature shift $\Delta T$ can be expressed as:

$$\Delta N = F - \lambda \Delta T$$

The magnitude and sign of $\lambda$ determine the direction of thermal response.

Atmospheric column water vapor is a critical greenhouse determinant, quantifying the vertical water vapor content from Earth's surface to TOA. This metric is represented as the condensed liquid's height/depth uniformly distributed across the column, with units in $kg/m^2$.

The amount of precipitable water vapor, denoted as $W$, can be calculated using the formula:

$$W = \frac{1}{\rho g} \int_{\rho_1}^{\rho_2} x dp$$

where $\rho$ is the density of water, $g$ is the gravitational acceleration, and $x(p)$ denotes the mixing ratio at a specific pressure $p$. The concentration of $x(p)$ varies with temperature, humidity, and geographical factors. source: precipitable water vapor

While the ERF is related to energy balance and dynamics at the top of the atmosphere, precipitable water vapor, is related to the distribution of water vapor in the atmosphere.

---

Table of Contents:

1. Introduction: Climate Analytics and Contrails
2. Objective
3. GOES-16 Satellite: Earth Monitoring
4. Contrail Detection: Climate Change Studies
5. Dataset OpenContrails: Benchmarking on GOES-16 ABI
6. Kaggle Competition: Identify Contrails
7. Documentation and Resources
8. Setup
9. Pipeline: Connect to Kaggle Datasets
10. Run and Usage
11. Output Example
12. Credits
13. Contributing
14. License
15. Acknowledgments and Support

---

Contrail Detection: Climate Change Studies

OpenContrails: Benchmarking Contrails Detection paper underlines contrail's importance, attributing them to $\sim \frac{2}{3}$ of aviation's climate impact and $\sim 2%$ of all anthropogenic climate changes.

Right side shows detected contrails (⇥); left side shows absence (⇤).

Dataset OpenContrails: Benchmarking Contrail Detection on GOES-16 ABI

• OpenContrails dataset, collected between April 2019-2020, encompasses:

• High-resolution contrail masks.
• Contrail detection model outputs from multiple GOES-16 image years.
• Emphasis on young, linear-shaped contrails.
• Utilization of ResNet and DeeplabV3+ architectures for contrail detection.
• Dataset is publicly available on Google Cloud Storage.

---

Kaggle Competition: Identify Contrails to Reduce Global Warming

The competition aims to develop a model predicting contrail formation and duration.

The dataset contains 244,400 images, each with 16 spectral bands, from the GOES-16 satellite. The images are labeled with contrail masks, and the goal is to predict the contrail masks for the test set. The competition is sponsored by Google Research and the Laboratory for Aviation and the Environment at MIT.

Our work will quantifiably improve the confidence in the prediction of contrail-forming regions and the techniques to avoid creating them.

---

Docs

• Flowchart 📈 | Decision tree for contrail identification • Context | Research insights for this study • Pre-print ArXiv | OpenContrails and GOES-16 ABI | Visual Overview • Roadmap 📍| Contrail Analysis

---

The overlayed histograms highlight varying pixel distributions across spectral bands, predominantly showcasing lower reflectance values in satellite imagery data.

---

Setup

• using conda

conda env create -f requirements.yml
conda activate contrail_env

• using pip and venv

python -m venv contrails_env
source contrails_env/bin/activate
pip install -r requirements.txt

• or, using conda and pip

conda create -n contrail_env
conda activate contrail_env
pip install -r requirements.txt

Both conda and pip can be used in the same environment, but issues may arise. Using them back-to-back can create an unreproducible state and overwrite packages. To avoid problems, create an isolated conda environment, install most packages with conda, and use pip with --upgrade-strategy only-if-needed.

---

Pipeline: Connect to Kaggle Datasets

⦿ Kaggle api key

Visit Kaggle Settings. Under the API section, click on “Create New API Token” to download the kaggle.json file.

Run the following commands in your terminal:

pip install kaggle
mkdir ~/.kaggle
mv /path/to/kaggle.json ~/.kaggle/kaggle.json # move .json to kaggle dir (i.e. mv ~ops/Downloads/kaggle.json ~/.kaggle/kaggle.json)
chmod 600 ~/.kaggle/kaggle.json
kaggle competitions list

⦿ Download a Kaggle dataset

∙ sample-dataset ▸ ash-color 22.4k files - 11.74 GB

kaggle datasets download shashwatraman/contrails-images-ash-color -p /path/to/desired/directory
unzip contrails-images-ash-color.zip -d /path/to/desired/directory
rm contrails-images-ash-color.zip

∙ full-dataset ▸ OpenContrails 244.4k files - 450.91 GB

kaggle competitions download -c google-research-identify-contrails-reduce-global-warming

---

Run

conda activate contrail_env 
pytest -sv

◼︎ Stop

ctrl + c

conda deactivate

Usage

python src/dataset_to_histogram_reports.py ./samples/kaggle_competition_mini_sample/
#--- 
python src/interactive_globe.py
#---
python -m src.utils.coordinate_converter samples/kaggle_competition_mini_sample/test/1000834164244036115 output
#---
python src/utils/rand_record_viz_with_masks_false_color.py --base_dir samples/kaggle_competition_mini_sample/test/1000834164244036115  --n_records 2 --n_times_before 4
#---
python src/utils/get_shape.py samples/kaggle_competition_mini_sample/test/1000834164244036115/band_08.npy
#---
python src/utils/rle_encoding_submission.py samples/kaggle_competition_mini_sample 2

Output Example

/output/tmp.png

python src/main.py

---

Credits go to the authors and contributors ⤵︎

Contrails Research

• OpenContrails: Benchmarking Contrail Detection on GOES-16 ABI (April 2023) - Led by MIT Professor Steven Barrett from the Laboratory for Aviation and the Environment. • Satellite images are from NOAA GOES-16. • goes_contrails_dataset • Contrail Recognition with Convolutional Neural Network and Contrail Parameterizations Evaluation (August 2018) • The Application of a Convolutional Neural Network for the Detection of Contrails in Satellite Imagery (May 2023) • Light Contrail Segmentation via Augmented Transfer Learning with Novel SR Loss Function in Hough Space (July 2023)

Visualization Tools

• RAMMB CIRA • GOES-16/17 • NASA's Eyes On The Earth Software, Demo • Ash RGB Guide • RGB Recipes • deck.gl • windy.com/aerosol • @blaylockbk | goes2go

Educational Resources

• Discover contrails at school • Science of contrails • Contrails-labeling-guide • Infrared Satellite Imagery • Interpreting Satellite Imagery • Using Python with GOES-16 Data • Q&A with SATAVIA • Atmospheric Optics Catalogues • STAC • WGS84 Coordinate System • Moderate Resolution Imaging Spectroradiometer (MODIS) • Could air someday power your flight? Airlines are betting on it. • Efficacy of climate forcings | Hansen et. al • ⽕⛆ Pyrocumulonimbus (storm clouds from extreme wildfires)| Identifying the Causes PyroCb

GOES-16 Resources

• gcp-public-data-goes-16 • Beginner's Guide to GOES-R • GOES-R Series Product Definition • GOES-16 • GOES-16 Band Reference Guide • github: @awslab | noaa-goes16 • aws/noaa-goes

From Kagglers

• Inversion - isualize (input dataset 450.91 GB) • Shashwatraman - contrails dataset sample (11.74 GB) train_df.csv, valid_df.csv • egortrushin - high score example • keegil - Using U-Net to Predict Segmentation Masks in Python & Keras • anshuls235 - Time Series Forecasting-EDA, FE & Modelling • jamesmcguigan - RAM/CPU Optimization | downcasting unit8 → float64 • Opencontrails Competition | One month to go! Summary of everything that happened

---

Contributing

👋 Welcome to the contributing section! We're excited to have you join us in enhancing the GOES-16 Satellite Contrail Detection project. Contribute by forking the repository, making changes in a descriptive branch, and submitting a pull request. Join our Slack channel for real-time communication with other contributors. Follow and contribute to this impactful project to combat climate change through advanced technology 🌍✨.

License

This project is licensed under the terms of the MIT license.

Work under construction. If there are inaccurate or missing quotes or credits, please email 👷 dev@patimejia.com. Thanks!

---

If you found this repository helpful...

Please consider starring this repository to help others find it. Feedback and contributions are appreciated. Thank you for reading this study. Wishing you all the best in your journey.

- pat [¬º-°]¬