Proposal for regession tests

[dsmic]

I saw the gnugo regression tests in script.

They are nice, but have disadvantages: 1. Only the move answers can be checked. 2. It is a lot of work to set them up

To overcome 1. I would like to introduce owner of xy (to check for correct live and death)

for 2.: I would love to have the regression commands in the sgf files.

The workflow: - find a usefull test - add in the sgf comments (with usefull prefix) the test commands - copy the sgf to the test directory

P.S.: is there a way I can assign bugs to me?

[Francois van Niekerk]

I don't think Point 1 is true, any commands can be inserted into .tst files, not just reg_genmove. The appropriate "#? []" comments will just need to be added. I haven't tested this, but I assume it will work.

Point 2 is debatable in my opinion. I think that the current method is very powerful, and I don't think it is too difficult, but I understand that you might disagree with the second part.

For interest's sake, I believe Pachi makes use of a system like you describe. I would be interested to find out more details.

At the moment the workflow is:

1. Find a useful test.
2. Add the sgf to the test directory
3. Add a few lines to the relevant .tst file

PS: I have added you as a member of the oakfoam repo with read access, which seems to show that you can have bugs assigned to you now.

[dsmic]

I see, it is much more powerfull as I initially thought. I will start using it and could add a script later to do my Point 2 to convert sgfs to the standart test if I still find it usefull!

I will start with a command reg_ownership (position) (ratio of success) (number of test)

number of test: The intention of this is that oakfoam could print a information, if a test is fullfilled extreamly good (ratio 0.99 with threshold 0.9 e.g.), to have a chance to reduce the threshold.

[dsmic]

alternatively I could add

reg_ownership (position) (ratio necessary) (ratio too good)

and let the test fail to increase the threshold and run again. What is the better solution?

[Francois van Niekerk]

I like the future idea of writing a script that converts sgfs to the current format. That would have the best of both worlds.

I am not quite sure what your reg_ownership command is designed to test. Is the idea to test what the ownership of a specific point is, according to the playouts? In that case I would recommend something like:

reg_ownership <position> <color> <ratio necessary>

and have the command simply output TRUE or FALSE. Would this be sufficient?

PS: I have assigned this to you, mostly because you seem to be working on it already.

[dsmic]

fuego regression tests and regression tests from the last KGS run added in my branch.

[Steve Kroon]

Attaching a (slightly edited) perl script written by Detlef for this issue. See discussion in http://sourceforge.net/p/fuego/mailman/message/32664071/ and https://groups.google.com/forum/#!topic/oakfoam/UK3LEg_RRDg for other discussion related to this issue. Edits from the version Detlef posted: (a) dos2unix conversion; (b) applied a fix for a bug Detlef pointed out in the thread; added some TODO comments.

[Steve Kroon]

Francois: The fuego test suite, which Detlef is incorporating, uses upper-case X-coordinates on the board. In his fork, he has changed the GTP output to upper case, since there are only a couple of trivial tests in the original Oakfoam codebase. Are there any other tools/applications/scripts you use that rely on lower-case output of board coordinates?

[Francois van Niekerk]

I can't recall if there are any other places where the case matters, but including a switch can always be done if an issue crops up at a later stage.