I was trying to compare ORB-Slam2 performace with ORB-Slam3 performance. Consider following comparison from ORB-Slam3 paper (columns with numbers correspond to different sequences from EuRoC dataset):
In 5 sequences (green), ORB-SLAM3 has better performance, in 4 (red), ORB-SLAM2 has better performance. One (blue) is tie. For one (pink), ORB-SLAM2 did not run completely, but its error is lesser in that case. Can we interpret this as ORB-SLAM-3 performs better in 50% of scenario and ORB-SLAM-2 in at least 40%.
I thought ORB-Slam3 should at least work considerably better than ORB-Slam2 at least when we use IMU. That is, running ORB-Slam3 in stereo inertial mode should perform better than ORB-Slam2 in stereo mode for same sequence / path trajectory. But as can be seen in below comparison, Stereo-ORB-Slam2 perform better (red underlined) than Stereo-inertial ORB-Slam3 in 5 sequences, while ORB-Slam3 performs better (green underlined) than Stereo-inertial ORB-Slam2 in 5 sequences. (In last sequence (pink underline), ORB-Slam2 did not run completely, ORB-Slam3 did run completely.)
Pointing out to discussions on different github issues:
- Also, this comment says: ORB-SLAM3 was better but its trajectory is very jittery which is a big downside
- This thread tries to discuss why ORB-SLAM2 perform better than ORB-SLAM3. Three people found ORB-SLAM2 performing better than ORB-SLAM3 (one is on KITTI dataset) with no contradiction.
So was guessing if ORB-Slam3 does really perform better than ORB-Slam2?
