Satellite SAM Dashboard

Mohanad Albughdadi. (2023). AlbughdadiM/satellite-sam-dashboard: v0.1.0 (v0.1.0). Zenodo. https://doi.org/10.5281/zenodo.8142916

About this app

This app integrates the Segment Anything Model ¹ (SAM) with Sentinel-2 data. The app is built using Dash Plotly and dash leaflet ². It allows segmenting satellite images using the two ways provided by SAM: automatic mask generator and prompt segmentation (using points and bounding boxes).

Environment

Features

• Integration of Sentinel-2 using WMS protocol.
• OSM as base map.
• Downloading Sentinel-2 data directly from the app using an ROI.
• Integration of the three available models from Meta of SAM.
• The ability to directly annotate Sentinel-2 images using bounding boxes and points.
• Different support of bounding boxes:
  • ROI
  • Object BBox
• Different support of points:
  • Foreground points
  • Background points
• Modify location of bounding boxes and points.
• Provide prompts for more than one object at a time.
• Refine segmentation inside a bounding box of an object by adding foreground and background points.
• Visualize the obtained results directly on the map.
• Download the results (RGB GeoTiff of the ROI, segmentation mask in PNG and GeoTiff).

How to deploy the app

Dev

1. Clone the repository, checkout to the dev branch, create a virtual environment and install the requirements.

git clone https://github.com/AlbughdadiM/satellite-sam-dashboard.git
cd satellite-sam-dashboard
git checkout dev
python3.10 -m venv myven
source myvenv/bin/activate
python3.10 -m pip install -r requirements.txt
mkdir src/weights
wget -P src/weights/ https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth

Note that the installation of rasterio requires GDAL to be installed ³. 2. Once all the dependencies are installed, do the following

cd src
python3.10 app.py

3. Navigate to 127.0.0.1:8050 to start working with the app.

Prod

1. Clone the repository, checkout to the main branch, build the docker image and create a container.

git clone https://github.com/AlbughdadiM/satellite-sam-dashboard.git
cd satellite-sam-dashboard
docker build . -t satellite-sam-dashboard:v0.1
docker run --name=dashboard -p 8080:8080 satellite-sam-dashboard:v0.1

Alternatively, you can pull the already-built docker image

docker pull ghcr.io/albughdadim/satellite-sam-dashboard:latest
docker run --name=dashboard -p 8080:8080 satellite-sam-dashboard:latest

Screenshot of the app

HowTo

The app provides two options to use SAM with Sentinel-2 images:

1. Automatic Mask Generation

   1. The user draws a bounding box on the ROI to generate a segmentation mask for.
   
   2. The bounding box annotation will be added to the table "annotated data" marking xmin, ymin, xmax, ymax, id and the type of the geometry.
   
   3. To generate an automatic mask for the whole region covered by the bounding box, the user must select type as "ROI BBox".
   
   4. In the Automatic Mask Configuration part of the dashboard, the two parameters prediction IoU threshold and stability score threshold contol the results of the generated mask.
   5. Once the ROI is selected and the parameters are adjusted, click on Segment ROI.
   6. The user can visualize the obtained results on the map.
   
   7. The user can download the results as a zip file by clicking on Download Results
   
   8. To start a new segmentation task, click on Refresh.

2. SAM Prompts

   1. The user draws a bounding box around the object to be segmented using the bounding box drawing tool.
   
   2. The user can obtain a binary mask showing the object inside this bounding box.
   
   3. The user can refine the segmentation results by using foreground and background points.
   
   4. The user can draw multiple bounding boxes associated with multiple objects in the satellite image. Additionally, each bounding box can be associated with multiple foreground and background points.
   5. Once the object is selected and the supporting points, click on Segment ROI.
   6. The user can visualize the obtained results on the map.
   7. The user can download the results as a zip file by clicking on Download Results
   
   8. To start a new segmentation task, click on Refresh.

Notes

• On the prod branch, sam_vit_b is the only model that is downloaded to the docker image to keep it lighter. However, feel free to add the other model weights as needed.
• For prompt segmentation, the current code supports two cases: only bounding boxes, and bounding boxes with supporting points. The only points case is not covered as the results were not always satisfactory with Sentinel-2. Note that in the case of bounding boxes and points, if the points are outside the bounding boxes, they will be ignored.

References

1 https://github.com/facebookresearch/segment-anything/tree/main

2 https://plotly.com/dash/

3 https://gdal.org/download.html