Regression between OpenVINS v2.2 and OpenVINS v2.3 on KAIST Urban Track 39

[adas-dev]

Dear OpenVINS developers,

I've been following this project for quite a while now. I mostly work with KAIST Urban dataset and benchmarking the OpenVINS performance on the Track 39.

I noticed an accuracy degradation on track 39 after the upgrade from v2.2 to v2.3. The degradation can be noticed visually when the car completes the first loop. Metrics also suggest that the problem exists.

Left: OpenVINS v2.2 | Right: OpenVINS v2.3

The only adjustment here for OpenVINS v2.2 was the upgrade to OpenCV 4. The OpenVINS v2.3 here is the same as in your master branch. In both cases, the KAIST Urban file player produced measurements thrice real time.

@goldbattle , could you please look into this issue and/or suggest where the problem might be?

I looked at you PR #117 briefly but it is very huge and I cannot find the root cause.

[goldbattle]

Hi, did you try the latest launch file (pushed 3 days ago in commit 6fcd132)? I did not evaluate accuracy, but if you have a configuration which perform better please feel free to post here or a PR with what you found had the best accuracy.

[adas-dev]

@goldbattle , with OpenVINS v2.2 I tried the original config. file only. With OpenVINS v2.3 I tried both the original (published in commit 6fcd132) and updated config. file with values from v2.2 taken from the diff. from PR #117.

OpenVINS v2.3 performance with updated config. file (old values) is worse that the original config. file from commit 6fcd132.

I will try to find better configuration values.

Do you think the regression is not caused by the changes in the code logic in v2.3? There are a lot of changes between these two releases...

[goldbattle]

You might want to checkout issue #127 for other details.

[adas-dev]

@goldbattle , setting cam0_is_fisheye and cam1_is_fisheye to false and false correspondingly solved the problem. I got better result for v2.3 stereo by leaving max_clones unchanged (15). For v2.2 stereo I got a better result for max_clones=20.

ov_eval from v2.2:

ov_eval from v2.3:

I think this issue can be closed.

[adas-dev]

@goldbattle, comparison of the accuracy on the track Urban39 before (with fisheye flags set to false) and after the latest commit 16f5c37 in OpenVINS 2.3, showed that "before" configuration is noticeably better than the "after" configuration.

So, for the stereo camera setup the configuration before 16f5c37 (with fixed fisheye flags) is better than the configuration after 16f5c37.

Does it make sense to split configuration for KAIST Urban to two configurations for mono and stereo cameras correspondingly?