RealSmileNet pytorch Implementation

Dependency

• python 3.7
• numpy 1.18.1
• Pillow 7.0.0
• dlib 19.19.0
• opencv-python 4.2.0.32
• torch 1.4.0
• torchvision 0.5.0

Dataset

• obtain the whole UVA-NEMO database from https://www.uva-nemo.org
• obtain the whole MMI database from https://mmifacedb.eu
• obtain the whole BBC database from https://www.bbc.co.uk/science/humanbody/mind/surveys/smiles/
• obtain the whole SPOS database from https://www.oulu.fi/cmvs/node/41317

The sample data is contained in processed_data folder

Train RealSmileNet

• test.py contains the test program for deep smile net. We do not provide the test program for model performance as it is time-consuming. Only the basic test are provided. run python test.py

• run python demo.py for the sample training of UVA-NEMO databases, the verbose.txt will contains the log of training. Note only one public data from UVA-NEMO are provided.

• The model weight will be saved in database_training folder, for example if the model is trained on UVA-NEMO database, the folder names uvanemo_training

               [--label_path LABEL_PATH] [--frame_path FRAME_PATH]
               [--frequency FREQUENCY] [--sub SUB] [--epochs EPOCHS] [--lr LR]

DeepSmileNet training, optional arguments for demo.py :

  -h, --help            show this help message and exit
  --database DATABASE   select the database to run, please selected from
                        UVANEMO,SPOS,MMI and BBC
  --batch_size BATCH_SIZE
                        the mini batch size used for training
  --label_path LABEL_PATH
                        the path contains training labels
  --frame_path FRAME_PATH
                        the path contains processed data
  --frequency FREQUENCY
                        the frequency used to sample the data
  --sub SUB             the subsitution for the model, please selected from
                        org,LSTM,GRU,resnet,miniAlexnet,minidensenet
  --epochs EPOCHS       number of total epochs to run
  --lr LR, --learning-rate LR
                        learning rate

• data_preprocessin.py contains the function used to preprocess UVA-NEMO,BBC,SPOS and MMI database. The pre trained shape_predictor_68_face_landmarks model can be downloaded from http://dlib.net/files/

• model.py contains the implementation of different model structures.

Notice

• The current version of code only tested on MacOS.
• We refactor the current implementation from our original training code. If you find any implementation error, please do not hesitate to contact us.
• Code for domain adaptation will be added quickly.

License

• The implementation of NonLocalBlock is from https://github.com/AlexHex7/Non-local_pytorch/tree/master/lib

• The implementation of ConvLSTM is based on https://github.com/ndrplz/ConvLSTM_pytorch/blob/master/convlstm.py

• The implmentation of resnet is based on https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py and https://github.com/zhunzhong07/Random-Erasing/blob/master/models/cifar/resnet.py

• The implementation of AlexNet is based on https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py

• The implementation of DenseNet is based on https://github.com/kuangliu/pytorch-cifar/blob/master/models/densenet.py