Merged in muellerphil/aclib2/Surrogates (pull request #11)
ADD solver from the CSSC'14
ADD instance meta files for all CSSC instance sets
ADD CSSC scenarios
FIX permissions of cssc algorithms' binaries
ADD features for UNSAT-UNIF-K5
FIX instance feature prefix of UNSAT-UNIF-K5
ADD rcw2, cls and corlat as cplex scenarios
FIX wallclock_limit in new cplex scenarios
ADD options to plot only a subset of configurators
MAINT restructure reading traj files in extra package
ADD start to implement script to get more statistics wrt traj-validation
MAINT working on get_stats_over_time
unifiying sat features
ADD add multiple scenario support in get_stats_over_time.py
MAINT increase font size in perf_over_time plots and fix name sorting
ADD option to remove common timeouts from stats (get_stats_over_time)
ADD option also in plot_perf_over_time
ADD option to disable legend in perf-time plots
FIX don't get worse again for speeedup calc
MAINT get_stats_over_time script
rename cssc scenarios to new name scheme
RM duplicate
ADD more clasp scenarios from CSSC'14
UPDATE README
MAINT plot_perf_over_time
MAINT get_stats_over_time
FIX change pcs to clasp-cssc
ADD calls_per_job to run script
ADD satenstein scenarios
reduce clasp-ws configuration budget to 3.5d
ADD SWGCP_SAT for satenstein
ADD satenstein instance features
MAINT cplex conf budget 2d
RM "--log True" from scenario files
ADD lingeling CSSC'14 industrial scenarios
RM prints
FIX plotting with Agg on ssh-sessions
FIX lingeling wrapper
FIX wallclock-limit -> wallclock_limit
FIX validate train over time
ADD option to plot estimated train and validated train
FIX cmd parsing of plotting/stat scripts
fix again
MAINT remove empty lines in scenario files (because of gga)
ADD option to remove conditionals from pcs
FIX pcs file format of lingeling
FIX submission on meta cluster
ADD validate.py can use scenario_no_conds.txt for validation
ADD two new scenarios and one instance set
scenarios: spear_ibm and spear_swv
instance set: SWV-Calysto
minor
ADD script to plot bootstrapped runs
ENH minor
ADD option to define optimizers to not bootstrap
FIX set seed to 1
FIX change bootstrap opt option
--bootstrap_opt is no required is a list of strings. Use bootstrap samples for each optimizer that contains one of the strings.
FIX satenstein scenario to also consider instance features
FIX QCP features (nan -> -512)
MAINT restructure scenario modifications
ADD two new scenario modifiers
ADD no_features
ADD enum_features
FIX validate.py to use local scenario file over ACLib file
ADD support reading validation over time on training instances
autoweka iris scenario
other autoweka scenarios
FIX replace "/" with "_" in configurator folder name
ADD implement a more flexible version of the bootstrapping script
rename "replace" with "legendlabel" in bootstrapping plot
MAINT clean up autoweka scenarios
FIX translate autoweka's pcs to new AClib2 format since it needed some features of the new format
new autoweka version with cv
README.md edited online with Bitbucket
MAINT get_evaluation_stats
add "--name_map" option to map configurator names (in final tables)
add "--scatter-plot" option to enable plotting of all scatter plots (default vs. configured)
MAINT output + warnings are debug messages now
ADD two further lpg (planning) scenarios from the ablation paper
ADD tsp scenarios from irace
FIX acotsp-var wrapper
FIX tsp scenarios
MAINT disable TSP scenarios until Manuel verified them
ADD new scenarios: cryptominisat on gi and labs
ADD new scenarios: clasp_ricochet, clasp_riposte
MAINT rename clasp-weighted-sequence to clasp_weighted-sequence
MAINT validate.py: local scenario file is only used for "--no_conds" option
FIX instance list files for depots and satellite
FIX wallclock-limit -> wallclock_limit in new asp scenarios
MAINT new version of validate libraries
now in a separated folder to not influence SMAC libraries
FIX satenstein wrapper such it is compatible with --no_conds
ADD name_map.json can already be used on run and validation level
FIX use quotes for paths in local env to support spaces in paths
FIX same fix for validate.py
FIX bootstrapped_perf_over_time for less observ than samples
FIX job names on nemo with white spaces
ADD started to reimplement stat scripts
ADD table generation in new get_evaluation_stats2.py
ADD (median) scatter plots for get_evaluation_stats2.py
ADD rm common tos for get_evaluation_stats2.py
FIX quartiles in tables
ADD more warn messages for missing files in get_evaluation_stats2
ADD overtime plots for get_evaluation_stats2
FIX tolerate missing validation data in get_evaluation_stats2
FIX no display variable for plotting
FIX avoid MWU on two identical vectors
FIX handle not validated configurators in get_evaluation_stats2
ADD all sparrowtoriss scenarios from cssc14
ADD use regex instead of str for name_map in run.py
add little script to extract best incumbent on train
FIX rm_tos with missing runs
add option to validate all incumbents
ADD plot over time of all incumbents with step functions
min and max run ids for get_evaluation_stats2
MAINT add some plotting arguments in get_evaluation_stats2
FIX get_evaluation_stats2 for missing runs (plot over time)
FIX merging of trajectories with different time stamps
adding scenario-specific ymax
FIX scatter plots for simulated parallel ACs
MAINT allow missing training data for sequential ACs in get_evaluation_stats2.py
MAINT allowing validation with all incumbents in get_evaluation_stats2.py
FIX config.json example in get_evaluation_stats2.py
update read_val.py
reduce marker size in overtime plots
ADD argument to evaluate train perf with get_evaluation_stats2
FIX get_evaluation_stats2\n\n * format of default perf\n * underline all best results
MAINT get_evaluation_stats2: add argument to multiply perf values with
MAINT get_evaluation_stats2: move code to own functions
MAINT get_evaluation_stats2: add aggregated overtime plots
MAINT get_evaluation_stats2: use mean instead of median for aggregation plot
MAINT get_evaluation_stats2: read config.json in ordereddict
FIX get_evaluation_stats2 mwu with different x,y lengths
FIX ylabel of over_time plot in get_evaluation_stats2
MAINT tight_layout for over_time plots
add speedup tables in get_evaluation_stats2
FIX remove ranksum test for speedup tests
FIX stattest for speedups: test also against def perf
FIX speedup table for validation of all incumbents
FIX missing train def for aggregation plot
FIX proper formating of speedup table
FIX use permutation test instead of MWU
ADD argument to change ylabel in get_evaluation_stats2 plots
FIX aggregate_over_time plot
RM print
ADD debug output for number of common timeouts
ADD cplex_regions100
ADD complete cplex_regions100 scenario
ADD smaller pcs for lingeling-cssc14
ADD argument to disable legend in get_evaluation_plots
MAINT get_evaluation_stats2
ADD unpaired permutation test and FIX: use it everywhere
ADD combined (large) table
ADD speedup TA table
ADD option "--compare_to" to give a reference AC used in the performance table for stat. sig. tests
ADD "^*" notation in performance table to highlight ACs that are stat. better than reference AC
Fix tight_layout in get_evaluation_stats2
ADD default config in traj file for GGA and GGA++
ADD description of "how to a new configurator"
ADD_CONFIGURATOR.md edited online with Bitbucket
FIX aclib2 scenario translation (mean10 par10)
ADD default config of GGA(++) (--ie 100)
FIX get_evaluation_stats2 for missing results
MAINT validate irace runs
MAINT change speedup computation
ADD GGA_def and GGA++_def to validate.py
ADD glucose-syrup-41_ibm scenario
ADD lingeling-small_ibm
ADD scenarios: glucose-syrup-41_swv and lingeling-small_swv
ADD glucose-syrup-41_bmc
FIX speedup calcuation for over-tuning
FIX header of Speedup Table (TA)
ADD fix example calls in readme
FIX maximal version of configspace
UPDATE to new configspace version
FIX glucose wrapper
FIX traj_file_regex for SMAC3 and in validate.py
FIX of last fix
FIX escape smac3 traj file path in call for smac-validate
RENAME --mode option of validate.py to --confs
ADD leading-ones and counting-ones scenarios (bbob)
update SMAC3 to 0.8.0
FIX import in satenstein wrapper
FIX number of digits considered by irace to 10
FIX imports and requirements
FIX stattest_per_timestep
ADD instance files for ones (bbob) scenarios
MAINT increase slack for nemo jobs; add startup for workers on nemo
ADD scripts to copy virt env to TMPDIR on nemo
ADD clasp_potassco-suite scenario
Â
ADD solver from the CSSC'14
ADD instance meta files for all CSSC instance sets
ADD CSSC scenarios
FIX permissions of cssc algorithms' binaries
ADD features for UNSAT-UNIF-K5
FIX instance feature prefix of UNSAT-UNIF-K5
ADD rcw2, cls and corlat as cplex scenarios
FIX wallclock_limit in new cplex scenarios
ADD options to plot only a subset of configurators
MAINT restructure reading traj files in extra package
ADD start to implement script to get more statistics wrt traj-validation
MAINT working on get_stats_over_time
unifiying sat features
ADD add multiple scenario support in get_stats_over_time.py
MAINT increase font size in perf_over_time plots and fix name sorting
ADD option to remove common timeouts from stats (get_stats_over_time)
ADD option also in plot_perf_over_time
ADD option to disable legend in perf-time plots
FIX don't get worse again for speeedup calc
MAINT get_stats_over_time script
rename cssc scenarios to new name scheme
RM duplicate
ADD more clasp scenarios from CSSC'14
UPDATE README
MAINT plot_perf_over_time
MAINT get_stats_over_time
FIX change pcs to clasp-cssc
ADD calls_per_job to run script
ADD satenstein scenarios
reduce clasp-ws configuration budget to 3.5d
ADD SWGCP_SAT for satenstein
ADD satenstein instance features
MAINT cplex conf budget 2d
RM "--log True" from scenario files
ADD lingeling CSSC'14 industrial scenarios
RM prints
FIX plotting with Agg on ssh-sessions
FIX lingeling wrapper
FIX wallclock-limit -> wallclock_limit
FIX validate train over time
ADD option to plot estimated train and validated train
FIX cmd parsing of plotting/stat scripts
fix again
MAINT remove empty lines in scenario files (because of gga)
ADD option to remove conditionals from pcs
FIX pcs file format of lingeling
FIX submission on meta cluster
ADD validate.py can use scenario_no_conds.txt for validation
ADD two new scenarios and one instance set
scenarios: spear_ibm and spear_swv
instance set: SWV-Calysto
minor
ADD script to plot bootstrapped runs
ENH minor
ADD option to define optimizers to not bootstrap
FIX set seed to 1
FIX change bootstrap opt option
--bootstrap_opt is no required is a list of strings. Use bootstrap samples for each optimizer that contains one of the strings.
FIX satenstein scenario to also consider instance features
FIX QCP features (nan -> -512)
MAINT restructure scenario modifications
ADD two new scenario modifiers
ADD no_features
ADD enum_features
FIX validate.py to use local scenario file over ACLib file
ADD support reading validation over time on training instances
autoweka iris scenario
other autoweka scenarios
FIX replace "/" with "_" in configurator folder name
ADD implement a more flexible version of the bootstrapping script
rename "replace" with "legendlabel" in bootstrapping plot
MAINT clean up autoweka scenarios
FIX translate autoweka's pcs to new AClib2 format since it needed some features of the new format
new autoweka version with cv
README.md edited online with Bitbucket
MAINT get_evaluation_stats
add "--name_map" option to map configurator names (in final tables)
add "--scatter-plot" option to enable plotting of all scatter plots (default vs. configured)
MAINT output + warnings are debug messages now
ADD two further lpg (planning) scenarios from the ablation paper
ADD tsp scenarios from irace
FIX acotsp-var wrapper
FIX tsp scenarios
MAINT disable TSP scenarios until Manuel verified them
ADD new scenarios: cryptominisat on gi and labs
ADD new scenarios: clasp_ricochet, clasp_riposte
MAINT rename clasp-weighted-sequence to clasp_weighted-sequence
MAINT validate.py: local scenario file is only used for "--no_conds" option
FIX instance list files for depots and satellite
FIX wallclock-limit -> wallclock_limit in new asp scenarios
MAINT new version of validate libraries
now in a separated folder to not influence SMAC libraries
FIX satenstein wrapper such it is compatible with --no_conds
ADD name_map.json can already be used on run and validation level
FIX use quotes for paths in local env to support spaces in paths
FIX same fix for validate.py
FIX bootstrapped_perf_over_time for less observ than samples
FIX job names on nemo with white spaces
ADD started to reimplement stat scripts
ADD table generation in new get_evaluation_stats2.py
ADD (median) scatter plots for get_evaluation_stats2.py
ADD rm common tos for get_evaluation_stats2.py
FIX quartiles in tables
ADD more warn messages for missing files in get_evaluation_stats2
ADD overtime plots for get_evaluation_stats2
FIX tolerate missing validation data in get_evaluation_stats2
FIX no display variable for plotting
FIX avoid MWU on two identical vectors
FIX handle not validated configurators in get_evaluation_stats2
ADD all sparrowtoriss scenarios from cssc14
ADD use regex instead of str for name_map in run.py
add little script to extract best incumbent on train
FIX rm_tos with missing runs
add option to validate all incumbents
ADD plot over time of all incumbents with step functions
min and max run ids for get_evaluation_stats2
MAINT add some plotting arguments in get_evaluation_stats2
FIX get_evaluation_stats2 for missing runs (plot over time)
FIX merging of trajectories with different time stamps
adding scenario-specific ymax
adding scenario-specific ymax
adding scenario-specific ymax
adding scenario-specific ymax
FIX scatter plots for simulated parallel ACs
MAINT allow missing training data for sequential ACs in get_evaluation_stats2.py
MAINT allowing validation with all incumbents in get_evaluation_stats2.py
FIX config.json example in get_evaluation_stats2.py
update read_val.py
reduce marker size in overtime plots
ADD argument to evaluate train perf with get_evaluation_stats2
FIX get_evaluation_stats2\n\n * format of default perf\n * underline all best results
MAINT get_evaluation_stats2: add argument to multiply perf values with
MAINT get_evaluation_stats2: move code to own functions
MAINT get_evaluation_stats2: add aggregated overtime plots
MAINT get_evaluation_stats2: use mean instead of median for aggregation plot
MAINT get_evaluation_stats2: read config.json in ordereddict
FIX get_evaluation_stats2 mwu with different x,y lengths
FIX ylabel of over_time plot in get_evaluation_stats2
MAINT tight_layout for over_time plots
add speedup tables in get_evaluation_stats2
FIX remove ranksum test for speedup tests
FIX stattest for speedups: test also against def perf
FIX speedup table for validation of all incumbents
FIX missing train def for aggregation plot
FIX proper formating of speedup table
FIX use permutation test instead of MWU
ADD argument to change ylabel in get_evaluation_stats2 plots
FIX aggregate_over_time plot
RM print
ADD debug output for number of common timeouts
ADD cplex_regions100
ADD complete cplex_regions100 scenario
ADD smaller pcs for lingeling-cssc14
ADD argument to disable legend in get_evaluation_plots
MAINT get_evaluation_stats2
ADD unpaired permutation test and FIX: use it everywhere
ADD combined (large) table
ADD speedup TA table
ADD option "--compare_to" to give a reference AC used in the performance table for stat. sig. tests
ADD "^*" notation in performance table to highlight ACs that are stat. better than reference AC
Fix tight_layout in get_evaluation_stats2
ADD default config in traj file for GGA and GGA++
ADD description of "how to a new configurator"
ADD_CONFIGURATOR.md edited online with Bitbucket
ADD_CONFIGURATOR.md edited online with Bitbucket
FIX aclib2 scenario translation (mean10 par10)
ADD default config of GGA(++) (--ie 100)
FIX get_evaluation_stats2 for missing results
MAINT validate irace runs
MAINT change speedup computation
ADD GGA_def and GGA++_def to validate.py
ADD glucose-syrup-41_ibm scenario
ADD lingeling-small_ibm
ADD scenarios: glucose-syrup-41_swv and lingeling-small_swv
ADD glucose-syrup-41_bmc
FIX speedup calcuation for over-tuning
FIX header of Speedup Table (TA)
ADD fix example calls in readme
FIX maximal version of configspace
UPDATE to new configspace version
FIX glucose wrapper
FIX traj_file_regex for SMAC3 and in validate.py
FIX of last fix
FIX escape smac3 traj file path in call for smac-validate
RENAME --mode option of validate.py to --confs
ADD leading-ones and counting-ones scenarios (bbob)
update SMAC3 to 0.8.0
FIX import in satenstein wrapper
FIX number of digits considered by irace to 10
FIX imports and requirements
FIX stattest_per_timestep
ADD instance files for ones (bbob) scenarios
MAINT increase slack for nemo jobs; add startup for workers on nemo
ADD scripts to copy virt env to TMPDIR on nemo
ADD clasp_potassco-suite scenario
Â