reference_based_sketch_image_colorization

PyTorch implementation of the paper "Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence" (CVPR 2020)

Dependencies

• Pytorch
• torchvision
• numpy
• PIL
• OpenCV
• tqdm

Usage

1. Clone the repository

• git clone https://github.com/Snailpong/style_transfer_implementation.git

2. Dataset download

• Tag2Pix (filtered Danbooru2020): Link
• You need to change the script 'danbooru2018' to 'danbooru2020' (can be changed)
• In my experiment, I used about 6000 images filtered by python preprocessor/tagset_extractor.py
  • I stopped the process when 0080 folder was finished downloading.

3. Sketch image generation

• XDoG: Link
• For automatic genration, I edited main function as follows:

if __name__ == '__main__':
  for file_name in os.listdir('../data/danbooru/color'):
      print(file_name, end='\r')
      image = cv2.imread(f'../data/danbooru/color/{file_name}', cv2.IMREAD_GRAYSCALE)
      result = xdog(image)
      cv2.imwrite(f'../data/danbooru/sketch/{file_name}', result)

• folder structure example

.
└── data
    ├── danbooru
    |   ├── color
    |   |   ├── 7.jpg
    |   |   └── ...
    |   └── sketch
    |       ├── 7.jpg
    |       └── ...
    └── val
        ├── color
        |   ├── 1.jpg
        |   └── ...
        └── sketch
            ├── 1.jpg
            └── ...

4. TPS transformation module

• TPS: Link
• Place thinplate folder to main folder

5. Train

• python train.py
• arguments
  • load_model: True/False
  • cuda_visible: CUDA_VISIBLE_DEVICES (e.g. 1)

6. Test

• python test.py

• arguments

  • image_path: folder path to convert the images
  • cuda_visible

Results

Sketch	Reference	Result

Observation & Discussion

• In Eq. (1), I could not scale the number of activation map, instead I scaled activation map into .
• In Eq. (5), I implemented the negative region as same region in different batches since the negative region is ambiguous.
• In Eq. (9), since is unclear in contrast to Eq. (8), I computed style (gram) loss with relu5_1 activation map.
• In this experiment, there was little difference in quality with or without the similarity-based triplet loss. After convergence from 20 to 0 from 1 epoch, there was little change.
• When the test image was predicted every 1 epoch after the content loss was converged, the color quality difference was remarkable.
• The converged adversarial losses of the generator and discriminator were 0.7 ~ 0.8 and 0.15 ~ 0.2, respectively.

Code Reference

• https://github.com/blandocs/Tag2Pix
• https://github.com/garygrossi/XDoG-Python
• https://github.com/cheind/py-thin-plate-spline
• https://gist.github.com/alper111/8233cdb0414b4cb5853f2f730ab95a49
• https://github.com/Jungjaewon/Reference_based_Skectch_Image_Colorization