Learning disentangled representations in the imaging domain

Visual summary of disentangled representation learning applications in medical imaging. Red connections indicate vector-based disentanglement, while blue connections indicate tensor/vector-based one (CSD). The visual examples are taken from the papers and repositories of the applications that are reported in the Tables below.

Applications of Disentanglement

Exemplar methods that exploit disentanglement to improve challenging tasks in computer vision and medical image analysis.

NOTE: The scope of this tutorial is not to survey every existing application for the to domains. If you feel that we should include your research as part of this tutorial we would be happy to do so. Please send an email to any of the contributor and help us create a place where everyone can understand and leverage disentanglement!

Computer Vision

For each application we denote the type of disentanglement as either "Vector" or content-style "C-S".

Task	Model Abbrev.	Paper (link to PDF)	Repository	Framework	Type	Original Implementation
I2I translation	MUNIT	pdf	repo	PyTorch	C-S
Face Attribute Transfer	ELEGANT	pdf	repo	PyTorch	Vector
Pose Estimation	n/a	pdf	repo	TensorFlow	C-S
Point cloud Generation	SetVAE	pdf	repo	PyTorch	Vector

Medical Image Analysis

For each application we denote the type of disentanglement as either "Vector" or content-style "C-S".

Task	Model Abbrev.	Paper (link to PDF)	Repository	Framework	Type	Original Implementation
Single-modal Segmentation	SDNet	pdf	repo	Keras	C-S
Brain Synthesis	n/a	pdf	repo	Keras	C-S
Causal Image Synthesis	n/a	pdf	repo	PyTorch	C-S
Classification	MIDNET	pdf	repo	Tensorflow	Vector

Note that the SDNet model has also been implemented in PyTorch and has been integrated into the GaNDLF framework.

Metrics

Measuring disentanglement in single vector latent variables:

Property	Paper (link to PDF)	Repository
Modularity	pdf pdf pdf	repo repo repo
Compactness / Completeness	pdf pdf pdf pdf	repo repo repo repo
Linear separability	pdf	repo
Explicitness	pdf pdf	repo repo
Consistency & Restrictiveness	pdf	repo
Robustness	pdf	repo

Measuring disentanglement between two latent variables of the same or different dimensionality:

Metric	Paper (link to PDF	Repository
Distance Correlation & Information over Bias	pdf	repo
Kernel-target alignment	pdf	repo
Hilbert-Schmidt independence criterion	pdf	repo

Citation

@article{liu2022learning,
 title={Learning disentangled representations in the imaging domain},
 author={Liu, Xiao and Sanchez, Pedro and Thermos, Spyridon and O’Neil, Alison Q and Tsaftaris, Sotirios A},
 journal={Medical Image Analysis},
 pages={102516},
 year={2022},
 publisher={Elsevier}
}