combinedimufactor + nonlinearequality<pose3> + genericprojectionfactor sometimes throw 'gtsam::InconsistentEliminationRequested'

[JzHuai0108]

Context: The gtsam library is compiled from the gtsam develop branch on ubuntu 14.04. In linking against either the release or debug gtsam library, the issue may occur with the information, "terminate called after throwing an instance of 'gtsam::InconsistentEliminationRequested' what(): An inference algorithm was called with inconsistent arguments. The factor graph, ordering, or variable index were inconsistent with each other, or a full elimination routine was called with an ordering that does not include all of the variables."

Problem statement: The problem I am trying to solve is the local bundle adjustment with visual and inertial data. The graph nodes consist of A. previous camera poses that are supposed to be fixed during the optimization, B. recent camera poses that are to be optimized, C. points in the world whose positions are to be optimized, D additionally the velocity and IMU biases associated with recent camera poses that are to be optimized. The factors are (1) nonlinearEquality<Pose3> to fix nodes of A, (2) genericprojectionfactor to constrain nodes from B and C, (3) combinedimufactor to constrain nodes in B and D.

Issue description The program that implements the above method can run pretty good for many calls of the local bundle adjustment function. As the camera moves, the number of fixed camera poses may increase if there are many overlaps between scenes at different camera poses. As the number of fixed camera poses increases, the exception 'gtsam::InconsistentEliminationRequested' may be thrown. The number of fixed camera poses upon throwing the exception are in the range of 15- 30.

Failed debug attempts (1) I tried both the debug and release gtsam library. Both throws the exception after running the program for a while and the number of fixed poses are around 25. (2) using custom key numbering, i.e., my own key generator, or symbol key numbering, i.e., Symbol('X', i), does not matter. (3) changing the insertion order of nodes in A and B to the graph and to the initialEstimate does not matter. (4) all the points in C have more than 2 observations used as GenericProjectionFactor, so all points are well constrained. (5) Bluntly reducing the number of fixed camera poses does not improve the situation much. Leaving only one fixed camera poses gives very bad results, but postpones the exception a lot.

Do you have any suggestions on solving the issue or debugging the code? For now, even when the program is linked against libgtsamDebug.so, only assembly code is shown in the Qt debugger. This prevents me from effectively pinpoint the cause.

P.S. For your information, the attached file contains the local bundle adjustment function.

[Frank Dellaert]

Hi

We're very happy you are using GTSAM. However, but I'm not sure I would qualify this as a GTSAM issue, but rather as a technical support question. The front-ends for bundle adjustment can be complex in their own right. Why do you think the issue is with GTSAM and not your code? Especially since we are doing the same things with thousands of cameras and tens of thousands of points.

If you find one of our examples throws this exception in error, so be it, but I don't know what to say or do when there is foreign code involved. Until you can give a more precise pinpoint of an actual issue with GTSAM, I will classify this issue as "invalid".

FD

[Frank Dellaert]

See comment: not a reproducible issue (yet)

[JzHuai0108]

Thanks for the comment which hints on the solution to the issue. Indeed, the exception is within the code I wrote. It is caused by velocity and bias nodes inserted into the initialEstimate, but not constrained by any factor. Now that these nodes are carefully examined and inserted, the program runs through successfully.

[Frank Dellaert]

Great to hear !