Timestamps synchronization error when running on Campus-Outdoor dataset provided by Kimera-Multi

[lhbuaa2009]

Description:
Thanks for sharing this work with the community.

I'm encountering a segmentation issue when testing Kimera-Multi on the provided mit-campus-outdoor datasets. After checking the console output, the problem comes from the VIO-fronent (i.e, kimera_vio_ros) which always crashed on this dataset.

Therefore, I separately test the kimera_vio_ros on the campus_outdoor dataset (only on one sequence) using the provided launch file and params. And the timestamp synchronization issues (e..g, time out of order, QueueSynchronizer check failed etc.) always come out followed by the termination of program. Following the reported issue #77, I already lower the play rate to 0.5 and 0.2, but cannot solve this problem.

Besides, I already test kimera_vio_ros on EuROC (V1_01_easy) and uHumans2 datasets, and it runs smoothly without this issue mentioned above. I would appreciate it much if you could give some hints on this issue.

Command:

roslaunch kimera_multi kimera_vio_jackal.launch robot_name:=acl_jackal robot_id:=0 use_d455:=true multirobot:=true lcd_no_optimize:=false use_external_odom:=false replay:=true should_use_sime_time:=true

rosbag play **/campus_outdoor_1014/10_14_acl_jackal.bag --clock -r 0.5 -s 0.2

Console output:

# play with -r 0.5
F0727 15:57:40.656031 13391 QueueSynchronizer.h:138] Check failed: timestamp == payload_timestamp (1665772999982196808 vs. 1665772999982204914) Syncing queue data_provider_right_frame_queue in module Stereo Data Provider failed;
 Could not retrieve exact timestamp requested:
 - Requested timestamp: 1665772999982196808
 - Actual timestamp:    1665772999982204914
*** Check failure stack trace: ***
    @     0x7fe2810550cd  google::LogMessage::Fail()
    @     0x7fe281056f33  google::LogMessage::SendToLog()
    @     0x7fe281054c28  google::LogMessage::Flush()
    @     0x7fe281057999  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fe27a856d66  VIO::StereoDataProviderModule::getInputPacket()
    @     0x7fe27a845df7  VIO::PipelineModule<>::spin()
    @     0x7fe27a9fa656  VIO::Pipeline::spin()
    @     0x7fe281737ad3  _ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_DeleterEEvENS1_12_Task_setterIS0_INS1_7_ResultIbEES3_ENSt6thread8_InvokerISt5tupleIJMN3VIO8PipelineEFbvEPSE_EEEEbEEE9_M_invokeERKSt9_Any_data
    @     0x7fe281737059  std::__future_base::_State_baseV2::_M_do_set()
    @     0x7fe27facb907  __pthread_once_slow
    @     0x7fe28173add8  _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZNSt13__future_base17_Async_state_implINS1_IS2_IJMN3VIO8PipelineEFbvEPS6_EEEEbEC4EOSB_EUlvE_EEEEE6_M_runEv
    @     0x7fe2805a56df  (unknown)
    @     0x7fe27fac36db  start_thread
    @     0x7fe28000061f  clone

# play with -r 0.2
F0727 17:09:27.603626 28907 DataProviderModule.cpp:32] Check failed: timestamp_last_frame_ < timestamp (1665773000653616428 vs. 1665773000653608322) Timestamps out of order:
 - Last Frame Timestamp = 1665773000653616428
 - Current Timestamp = 1665773000653608322
*** Check failure stack trace: ***
    @     0x7fa51a52b0cd  google::LogMessage::Fail()
    @     0x7fa51a52cf33  google::LogMessage::SendToLog()
    @     0x7fa51a52ac28  google::LogMessage::Flush()
    @     0x7fa51a52d999  google::LogMessageFatal::~LogMessageFatal()
    @     0x7fa513d1aff5  VIO::DataProviderModule::getTimeSyncedImuMeasurements()
    @     0x7fa513d20e3b  VIO::MonoDataProviderModule::getMonoImuSyncPacket()
    @     0x7fa513d2bc5b  VIO::StereoDataProviderModule::getInputPacket()
    @     0x7fa513d1bdf7  VIO::PipelineModule<>::spin()
    @     0x7fa513ed0656  VIO::Pipeline::spin()
    @     0x7fa51ac0dad3  _ZNSt17_Function_handlerIFSt10unique_ptrINSt13__future_base12_Result_baseENS2_8_DeleterEEvENS1_12_Task_setterIS0_INS1_7_ResultIbEES3_ENSt6thread8_InvokerISt5tupleIJMN3VIO8PipelineEFbvEPSE_EEEEbEEE9_M_invokeERKSt9_Any_data
    @     0x7fa51ac0d059  std::__future_base::_State_baseV2::_M_do_set()
    @     0x7fa518fa1907  __pthread_once_slow
    @     0x7fa51ac10dd8  _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZNSt13__future_base17_Async_state_implINS1_IS2_IJMN3VIO8PipelineEFbvEPS6_EEEEbEC4EOSB_EUlvE_EEEEE6_M_runEv
    @     0x7fa519a7b6df  (unknown)
    @     0x7fa518f996db  start_thread
    @     0x7fa5194d661f  clone

Additional files:
Please attach all the files needed to reproduce the error.

Please give also the following information:

• KimeraVIO branch, tag or commit used: master
• GTSAM version used: recommended
• OpenGV version used: recommended
• OpenCV version used: 3.4.8
• Operating system and version: Ubuntu 18.04
• Did you change the source code?: no

[iftahnaf]

Hi @lhbuaa2009, did you manage to solve this issue?

[lhbuaa2009]

@iftahnaf Hallo, sorry for my late reply. Unfortunately i have not solved this issue yet. I tried with differnt playing back rates and launching parameters, but it didn't work. Since in my case the program can run on EuRoc and uHumans2 dataset, i guess this problem could be related to the sequence of mit-campus-outdoor dataset and the parameters within its launch file. And if you have other insights about this issue, I would aprreciate it if you could share it with me.

[iftahnaf]

Hi @lhbuaa2009,

Thanks for the reply.

We encounter this problem when trying to run the VIO on a live stream from an OAK-d-lite camera.
We created our ROS-based driver for the camera using depthai library.
When we published the images on a topic, we assigned different timestamps for left and right frames, which was causing the issue.
To fix it, we assigned the same timestamp for both cameras, and it seems to solve the issue.