HTTPS SSH

Algorithm Configuration Library 2.0

Install

AClib requires Python 3.5 (we implemented AClib under Anaconda 3.4) and Java >8

pip install -r requirements.txt

Some target algorithms may have further dependencies.

Installation of Instances

Since the instance sets are by far too large to upload on bitbucket (compressed >20GB), please download the instance sets manually.

Please extract the instances in the root directory of AClib:

tar xvfz XXX.tar.gz

For some instance sets, we don't know the actual license they were published under --- not listed above. Please write us an e-mail to get details how to access these instance sets.

Requirements for AC procedures

SMAC

  • Java 8

ParamILS

GGA

  • precompiled on Ubuntu 14.04 LTS with gcc -- probably needs to be recompiled

IRACE

  • R (see irace_2.0 README)
  • jsonlite package in R (install.packages('jsonlite'))

Call the following command to install irace:

cd configurators/ && make install

Example

Configuration

To run a scenario call:

python aclib2/aclib/run.py -s circuit_fuzz_cryptominisat -c SMAC2 -n 2 --env local

It runs the scenario circuit_fuzz_cryptominisat with 2 independent SMAC (v2) runs.

To use the same resources for GGA, call:

python aclib2/aclib/run.py -s circuit_fuzz_cryptominisat -c GGA -n 1 --ac_cores 2 --cores_per_job 2 --env local

Validation without Workers

To validate the runs (here training and test performance of final incumbent):

python aclib2/aclib/validate.py -s circuit_fuzz_cryptominisat -c SMAC2 -n 2 --set TRAIN+TEST --mode INC --env local

Validation with Workers

To validate the runs (here performance over time with using workers on a MYSQL data base -- this will work only on our meta cluster in Freiburg. Send us an e-mail if you also want to use it on your system.):

python aclib2/aclib/validate.py -s circuit_fuzz_cryptominisat -c SMAC2 -n 2 --set TEST --mode TIME --pool test_aclib2 --env local

and to submit workers:

python aclib2/aclib/submit_workers.py --pool test_aclib2 -n 10000 --time_limit 910

To get read results from the database, run again:

python aclib2/aclib/validate.py -s circuit_fuzz_cryptominisat -c SMAC2 -n 8 --set TEST --mode TIME --pool test_aclib2 --env local

Statistics and Plots

If you have validated your runs, you can run the following command to get some basic statistics and scatter plots:

python aclib2/aclib/get_evaluation_stats.py

This script will look into "." for runs generated with the previous scripts.

If you have validated your runs over time (--mode TIME), you can plot the performance of the configurators over time:

python aclib2/aclib/plot_perf_over_time.py -s circuit_fuzz_cryptominisat

Issue Tracker

https://bitbucket.org/mlindauer/aclib2/issues

Scenarios

Here is a small selection of well-studied AC scenarios to show the diversity of AClib:

Scenario Domain Configurators #Params #Instances Budget
clasp-weighted-sequence ASP SMAC, ROAR, PILS 90 240/240 4d
cplex_regions200 MIP SMAC, ROAR, PILS, GGA 73 1000/1000 2d
lpg-zenotravel Planning SMAC, ROAR, PILS 67 2000/2000 2d
cryptominisat_circuit_fuzz SAT SMAC, ROAR, PILS, GGA 36 299/302 2d
probsat_7sat90 SAT SMAC, ROAR, PILS, GGA? 9 250/250 3h
spear_swgcp SAT SMAC, ROAR, PILS 26 1000/2000 5h
branin (multi-instance) BBOB SMAC, ROAR, PILS 1 76/75 1000 runs
xgboost_covertype ML SMAC, ROAR, PILS, IRACE 11 10/1 500 runs
svm_mnist ML SMAC, ROAR, PILS, IRACE 7 10/1 500 runs

Adding new scenarios

Please see ADD_SCENARIOS.md for all format requirements.

Contact

Marius Lindauer lindauer@cs.uni-freiburg.de