One run multiple times in runhistory

[Marius Lindauer]

Hi Tobi,

you wrote in the runhistory code:

"TODO JTS: We might have to execute one config multiple times since the results can be noisy and then we can't simply overwrite the old config result here!"

Unfortunately, I don't understand the problem. If you have noisy runs, I would expect that you set a random seed (which is also important for SMAC to know to reduce a degree of variance) and the random seed is part of the runhistory keys. So, I would say that we already support this feature.

Best, Marius

[Tobias Springenberg]

Right, if the seed is set everywhere then that would work, still in our RL experiments for example we have to interact with real robots and runs of the same parameters are not necessarily deterministic, there is noise in the real world and noise in the returned function values ... Also AFAIK a lot of the "synthetic" benchmarks used in bayesian optimization are noisy functions and I guess you want to be able to optimize these. Cheers, Tob

[Marius Lindauer]

Hi Tobi,

I see your point.

However, I would prefer to not change this. If you use SMAC with the option algo-deterministic=false, SMAC will automatically generate random seed for your target algorithm runs. You don't need to use these seeds if you don't want, but still the runhistory can store the results properly.

Cheers, Marius

[Marius Lindauer]

untouched since some month and nobody really complained ... so won't fix for the time being