Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update context model code #296

Open
wants to merge 27 commits into
base: main
Choose a base branch
from
Open

Update context model code #296

wants to merge 27 commits into from

Conversation

rwood-97
Copy link
Collaborator

@rwood-97 rwood-97 commented Nov 27, 2023

Summary

The current version of MapReader contains unused/untested code for using the patch context when training the model. From the 2021 paper:

"Context-aware patchwork method.
A limitation of the patchwork method is that it ignores neighboring patches when training or performing model inference. To capture the strong, spatial dependencies between neighboring patches (which will be common in maps for many types of labels [16]), MapReader supports building model ensembles as shown in Fig. 8. Model-1 and Model-2 are two CV models with neural network architectures defined by the user. These models can have different or similar architectures. For exam- ple, in one of our experiments, Model-2 was a pre-trained model on our patch dataset while Model-1 was a pre-trained model from PyTorch Image Models [40] with a different architecture. As shown in Fig. 8, a patch and its context image are fed into Model-2 and Model-1, respectively. In practice, the user only specifies the size of the context image, and MapReader extracts and preprocesses the context image from the dataset. Model-1 and Model-2 generate vector representations for the input images, V1 and V2. The size of these vectors are defined by the user and a combination of the two (e.g., by concatenation) is then fed into Model-3 for prediction. Such model ensembles can be an efficient approach to achieving high- performing CV models [37]."
image

This PR updates this code to be more simple/user-friendly.

Fixes #287
Addresses #17

Describe your changes

  • Both PatchDataset and PatchContextDataset return image(s) as tuples. These datasets now return (img,) and (img, context_img) respectively.
  • Classifier handles "inputs" as tuples (so now can handle both context and normal inputs)
  • Add context_dataset arg when creating datasets to enable creating of 'PatchContextDataset' type datasets from the annotations loader.
  • Update twoParrellelModels class to clarify patch/context branches of the model
  • Update generate_layerwise_lrs method for context datasets

Checklist before assigning a reviewer (update as needed)

  • Self-review code
  • Ensure submission passes current tests
  • Add tests
  • Update relevant docs

Reviewer checklist

Please add anything you want reviewers to specifically focus/comment on.

  • Everything looks ok?

@rwood-97

This comment was marked as resolved.

This comment was marked as resolved.

@rwood-97
Copy link
Collaborator Author

rwood-97 commented Jan 11, 2024

Things still to do:

  • Check annotator works for edge patches
  • Think about how to annotate context patches
  • Geotiff saving!!!

@codecov-commenter
Copy link

codecov-commenter commented Jan 12, 2024

Codecov Report

Attention: 31 lines in your changes are missing coverage. Please review.

Comparison is base (566e602) 59.58% compared to head (11afa54) 61.60%.
Report is 2 commits behind head on main.

❗ Current head 11afa54 differs from pull request most recent head a5e36ca. Consider uploading reports for the commit a5e36ca to get more accurate results

Files Patch % Lines
mapreader/classify/classifier.py 72.13% 17 Missing ⚠️
mapreader/classify/load_annotations.py 60.00% 8 Missing ⚠️
mapreader/classify/custom_models.py 0.00% 4 Missing ⚠️
mapreader/classify/datasets.py 60.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #296      +/-   ##
==========================================
+ Coverage   59.58%   61.60%   +2.01%     
==========================================
  Files          35       34       -1     
  Lines        6164     5949     -215     
==========================================
- Hits         3673     3665       -8     
+ Misses       2491     2284     -207     
Flag Coverage Δ
unittests 61.60% <65.55%> (+2.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rwood-97
Copy link
Collaborator Author

rwood-97 commented Jan 23, 2024

Current workflow is:

  1. patchify as normal - there is no longer the square_cuts arg so now patches are always square but will be padded if at an edge
  2. annotate at patch level by calling annotate() or context level by calling annotate_context(). Annotations are saved as either filename.csv or filename_context.csv for patch or patch+context annotations respectively. You can only assign context labels to patches which you've annotated.
  3. load context annotations (contains patch and context labels) and create patch context dataset. __get_item__() returns tuples of patch/context images and patch/context labels.
  4. train context classifier (TODO)

@rwood-97 rwood-97 mentioned this pull request Jul 3, 2024
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Think about using context model for training/fine-tuning model
2 participants