Abstract

Texel's Tuning Method is a method developed by Peter Österlund to tune chess evaluation parameters. TTM requires one to play many hyper-bullet games and extract a (random) set of positions from the games. Next, the evaluation parameters are tuned such that the selected positions evaluate closer to the actual game result (i.e. high-positive score indicates white won, 0 means tie, low-negative score indicates black won). While TTM has been used successfully by several engines including Zurichess, Texel, Gaviota, Pedone selecting which positions generate best tuned values is still unknown. In this article we explore what are the best way to select tuning positions.

Setup

Using the following cutechess command line and one of the latest versions of Zurichess (as of 08.Jan.2016) we generated 350.000 hyper-bullet games.

#!bash

$HOME/cutechess/projects/cli/cutechess-cli \
        -srand $RANDOM \
        -pgnout games.pgn \
        -repeat \
        -recover \
        -tournament gauntlet \
        -rounds 500000 \
        -concurrency 16 \
        -ratinginterval 50 \
        -draw movenumber=50 movecount=5 score=20 \
        -openings file=$HOME/2moves_v1.pgn format=pgn order=random \
        -engine cmd=$HOME/src/games/zurichess name=zurichess1 tc=40/2+0.05 \
        -engine cmd=$HOME/src/games/zurichess name=zurichess2 tc=40/2+0.05 \
        -each timemargin=60000 option.Hash=512 proto=uci

We used hyper-bullet time control (40 moves in 2 seconds plus 0.05s increment) as recommended by the inventor of TTM. A very high time margin was used in order to eliminate timeouts. We used automatic draw adjudication only to eliminate long boring games that could skew the set of positions. We didn't use any end game tables because Zurichess lacks such support. The opening book is 2moves_v1.pgn which is used by Stockfish and Zurichess testing framework and contains all balanced positions that are 2 moves away from the starting position.

From the generated pgn we extracted O(10.000.000) positions out of which we randomly selected 2.000.000 positions for tuning.

For each selected position we searched with Zurichess at depth 1 (plus quiescence search). We picked the position at the end of returned principal variation to evaluate and tune the weights. Zurichess' evaluation function is a simple neural network with no hidden layer and one output as described here. We tuned the weights with the ADAM optimizer provided by the Tensorflow open-source library. After the weight tuning is completed we tested the parameters using self-play between the new version and the commit c46a425 of Zurichess with stopping criteria: more than 30.000 games, or SPRT with goal posts (-3 ELO, 1 ELO).

In each test we varied which positions were extracted and repeated the process three times.

Results

We ran several test changing the position selection method.

capt: duplicates, captures, no checks, no mates (default)

In this test we included duplicates, captures, but excluded 5 positions before and including checks, and mates.

quiet: duplicates, no captures, no checks, no mates

In this test we included duplicates, but excluded 5 positions before and including captures, checks, and mates. It's clear that not including captures results in a huge elo drop.

There is a huge variation between mid game and end game piece scores.

unique: no duplicates, captures, no checks, no mates

In this test we included captures, but excluded duplicates and 5 positions before and including captures, checks, and mates. Dropping duplicate positions gives more consistent results.

nodraw: no draws, no duplicates, captures, no checks, no mates

This test is similar to unique, but we dropped positions from drawn games. From the results including the draws is beneficial.

Dropping draws, probably dropped a lot of positions from drawn long late end games.

drop4: more than 4 non-pawns, no duplicates, captures, checks, no mates

This test is similar to unique, but we dropped all positions that had at most 4 non-pawns on the board. In this test we dropped all classical endgames rook vs rook, or rook vs minor. Surprisingly, the results were positive.

Again, dropping positions from end games tends to increase piece scores in the endgames.

drop30: at most 30 pieces, no duplicates, captures, checks, no mates

This test is similar to unique, but we dropped all positions that had at more than 30 pieces. These positions occur mostly in the opening. Early results suggest a slight improvement.