HTTPS SSH

AutoFolio entering the Algorithm Selection Challenge 2015

Authors

Homepage

www.ml4aad.org/autofolio

System Description

Overall Workflow (Training)

  • train a cheap model of FlexFolio (cost-sensitive pairwise random forest models without any solver schedules)
  • evaluate the performance of different FlexFolio configurations on training data (using cross-validation)
  • if training time remains, use SMAC to configure FlexFolio (AutoFolio) on training data (using only a small and efficient subspace of AutoFolio)

Autofolio

AutoFolio [1] automatically configures an algorithm selector by determining a well-performing algorithm selection strategy and its (hyper-)parameters. Here, we use the algorithm configuration system SMAC [2] to configure the algorithm selection framework FlexFolio (successor of claspfolio 2 [3]). In detail, the AutoFolio version used in the Algorithm Selection Challenge consists of two phases: (i) AutoFolio assesses the performance of a small set of complementary FlexFolio configurations (hand picked based on human experience) which performed well in the past and (ii) AutoFolio configures FlexFolio, using SMAC, on a subspace of the configuration space of FlexFolio, called AutoFolio_voting (see [1]), which includes different pre-preprocessing approaches and an algorithm selection approach with pairwise cost-sensitive models [4] with different classifiers and their hyper-parameters.

Literature

[1] M Lindauer and H. Hoos and F. Hutter and T. Schaub AutoFolio: An Automatically Configured Algorithm Selector. In: Journal of Artifical Intelligence Research (JAIR). 2015. To appear

[2] F. Hutter and H. Hoos and K. Leyton-Brown Sequential Model-Based Optimization for General Algorithm Configuration In: Proceedings of the conference on Learning and Intelligent OptimizatioN (LION 5). 2010. pages 507-523

[3] H. Hoos and M. Lindauer and T. Schaub claspfolio 2: Advances in Algorithm Selection for Answer Set Programming In: Theory and Practice of Logic Programming 14. 2014. pages 569-585

[4] L. Xu and F. Hutter and J. Shen and H. Hoos and K. Leyton-Brown SATzilla2012: Improved Algorithm Selection Based on Cost-sensitive Classification Models In: Proceedings of SAT Challenge 2012: Solver and Benchmark Descriptions. 2012. pages 57-58

[5] H. Hoos and R. Kaminski and M. Lindauer and T. Schaub aspeed: Solver Scheduling via Answer Set Programming In: Theory and Practice of Logic Programming 15 (2015): 117-142

Pre-Solvers

We used Aspeed [5] to compute the pre-solving schedules based on a 10-fold CV using FlexFolio.

ASP-POTASSCO

'clasp/2.1.3/h8-n1': 4

CSP

NONE

MAXSAT

NONE

PREMARSHALLING

astar-symmullt-transmul': 307

PROTEUS-2014

choco': 2060

QBF

QuBE': 299

SAT11-HAND

sattime+_2011-03-02': 254

SAT11-INDU

NONE

SAT11-RAND

adaptg2wsat2011_2011-03-02': 33

SAT12-ALL

tnm': 5

SAT12-HAND

sparrow': 16

SAT12-INDU

lrglshr': 5

SAT12-RAND

NONE

Feature Groups

We used the results of AutoFolio [1] to determine well-performing feature groups.

We assume that all given data (training and test) only includes instance features we specified here.

ASP-POTASSCO

Static, Dynamic-1, Dynamic-2, Dynamic-3

CSP

all_feats

MAXSAT

group_basics

PREMARSHALLING

all

PROTEUS-2014

csp

QBF

all_feats

SAT11-HAND

Pre, Basic, KLB, CG

SAT11-INDU

Pre, Basic, sp, ls_saps, cl, DIAMETER

SAT11-RAND

Pre, ls_gsat, sp, ls_saps

SAT12-ALL

Pre, KLB, Basic, lobjois, sp, ls_saps

SAT12-HAND

Pre, Basic, CG, DIAMETER, lobjois, ls_saps, sp

SAT12-INDU

Pre, ls_gsat, Basic, lobjois, sp, cl, CG

SAT12-RAND

Pre, KLB, Basic

Installation

AutoFolio

  • Python 2.7 is required
  • run selectors/flexfolio/python_env.sh to install a virtualenv with the required packages of FlexFolio
  • set an alias of Python to selectors/flexfolio/virtualenv/bin/python (alias python=selectors/flexfolio/virtualenv/bin/python)
  • test the binaries in selectors/flexfolio/binaries; recompile them if they are broken
  • test runsolver binary in runsolver/; recompile it if it is broken

Call

Training

cd <root-folder>
mkdir <training-output-directory>
python ./train.py -s <ASlib-scenario> -o <training-output-directory -p <presolver> -t <presolver_runtime>

Testing

cd <root-folder>
mkdir <testing-output-directory>
python ./test.py -s <ASlib-scenario> -m <training-output-directory> -o <testing-output-directory> -p <presolver> -t <presolver-runtime>