Incumbent in Acquisition Function

[Marius Lindauer]

Hi Aaron,

If I understand the code of the acq. module correctly, I would say that the determination of the incumbent (BestObservation) does not consider the multi-instance case. Since we keep track of the incumbent its performance anyway, we could provide this information directly without any code in the acq module.

What do you think?

Best, Marius

[Aaron Klein]

I removed the incumbent estimation in the acquisition function, however, now we need to retrieve the value of the incumbent inside the SMBO class. I am not sure how we do this in a multi-instance scenario?

[Marius Lindauer]

Thanks!

According to the SMAC Paper (LION 2011), I would say that incumbent performance is simply the "empirical mean performance". The Intensifier.intensify() method knows the empirical mean performance. The function could simply return it and we could put it in choose_next().

[Matthias Feurer]

I'll update my PR on choose_next() in a few minutes. Can you please wait until that?

[Marius Lindauer]

maybe a stupid idea, but what would happen if we simply use the predictive mean instead of the empirical mean?

[Matthias Feurer]

In my opinion the RunHistory2EPM module should return the empirical mean performance of the incumbent or provide a method to do so. This would allow us to put all code for dealing with raw run data in one place. Later, one would only have to change code in two places (RH2EPM, SMBO) instead of currently three (RH2EPM, Intensify, SMBO) to adapt SMAC to a new use case. Of course, we would have to rename RunHistory2EPM.