Wrong output of incumbent performance

[Matthias Feurer]

Currently, instensify outputs the incumbent performance based on the runs it ran the challenger on. This is very confusing as the performance wiggles, although it actually does not.

[Marius Lindauer]

Could you give a concrete example. If you don't need the debug mode, you should only see lines such as

INFO:intensifier:Updated estimated performance of incumbent on 6 runs: 0.4763

This is always for all incumbent runs

INFO:intensifier:Challenger (0.0794) is better than incumbent (1.3925) on 6 runs.

shows the performance of the challenger and incumbent on the full set of instance-seed pair we ran so far.

Only in the debug mode, you see something like

Incumbent 0.1 is better than challenger 0.3 on 2 runs.

which could be a subset of the runs.

[Matthias Feurer]

Okay, I didn't see that this is only for the debug mode.