EAGLE-TextureMapping

This is an attempt for implementation of the paper "Patch-Based Optimization for Image-Based Texture Mapping".

Before use

1. Make sure you have installed Boost, Eigen3, VTK 6.3, PCL 1.9 (with OpenNI and CUDA 10.1 if using PCL::KinectFusion to get a PLY file from RGBDs in the getmeshcamera.cpp, otherwise you can delete all things related to gpu in eagle_textureMapping.pro), OpenCV, and ImageMagick. (I'm using Ubuntu 16.04, but the project works if you successfully install these things on Windows.)

2. Open the project with Qt creator (be sure to set the Make path as same as the project path), then you need to edit the INCLUDE paths in eagle_textureMapping.pro file to fit in your envirenment. (If necessary, recompile ./lib/Eagle_Utils.cpp and ./patchmatch/eagle_pm_minimal.cpp, the surfix _id of the patchmatch bin filename is the patch width.)

3. [UPDATE in 2020.1.9] I upgrade a mini-version of TextureMapping codes, which only include things relating to Patch-Based Optimization. The old version can still be found on branch Full-Ver.

How to use

All parameters are in the settings.h file, change them to your own data's info as the comment says. (The data in the project has been configed on init_zhou() function.)

• If you don't have a PLY file: （You need to run codes under Full-Ver branch to get a PLY file.）

  1. You need to rename all original Kinect pictures with consistent indexs, and put them under a folder.

  2. Change settings.h with all infos, including the folder path, RGBD name patterns, camera intrinsics...

  3. Run the project. If out-of-memory error occurs (This happens when the GPU has limited free memory. To restart the computer or close Qt creator and run sudo sync may help.), find the world.pcd under the keyFramesPath path, and do pcl_kinfu_largeScale_mesh_output world.pcd -vs 4.0 in the console yourself (4.0 is the volume_size in getmeshcamera.cpp, change it if you editted). If memory is enough, the PCL will generate some PLYs under the folder named mesh_1.ply, mesh_2.ply... Generally, mesh_1.ply is what we want.

  4. Edit the mesh_1.ply with Meshlab, do some stuffs like deleting useless faces, deleting unreferenced vertex and faces, doing simplification...

  5. Now you have the PLY file, then do as in "If you have a PLY file".

• If you have a PLY file:

  1. It's recommended to choose some keyframes manually for better results, also to delete the keyframe which shifts too much among others. (what the kfIndexs variable does.) (That is to say, these frames' camera matrixs are inaccurate or with large errors. It's really difficult for my code to make these frames fit others. So ghosting areas remain at such situation.)

  2. Change settings.h, especially checking for kfCameraTxtFile and plyFile under the keyFramesPath.

  3. Run the project.

About settings

1. The patchStep variable controls the patch numbers when voting. If it's set to 1, the lamda variable needs to be large enough to make Targets not as same as Sources. (I explore patchStep = patchWidth / 2 makes lamda = 0.2 ~ 0.4 effectively work, but larger lamda makes the result fall into local minima too early, while smaller lamda makes no sense as Textures will do nothing to Targets.)

2. The scaleInitH variable sets the first scale resolution's height. I find if it's set to 64, it's quite difficult to store minor textures at finer scales in my datas. So I set it to originImgH / 4 which is better than too small.