Proposal for regession tests
I saw the gnugo regression tests in script.
They are nice, but have disadvantages: 1. Only the move answers can be checked. 2. It is a lot of work to set them up
To overcome 1. I would like to introduce owner of xy (to check for correct live and death)
for 2.: I would love to have the regression commands in the sgf files.
The workflow: - find a usefull test - add in the sgf comments (with usefull prefix) the test commands - copy the sgf to the test directory
P.S.: is there a way I can assign bugs to me?
Comments (8)
-
repo owner -
reporter I see, it is much more powerfull as I initially thought. I will start using it and could add a script later to do my Point 2 to convert sgfs to the standart test if I still find it usefull!
I will start with a command reg_ownership (position) (ratio of success) (number of test)
number of test: The intention of this is that oakfoam could print a information, if a test is fullfilled extreamly good (ratio 0.99 with threshold 0.9 e.g.), to have a chance to reduce the threshold.
-
reporter alternatively I could add
reg_ownership (position) (ratio necessary) (ratio too good)
and let the test fail to increase the threshold and run again. What is the better solution?
-
repo owner -
assigned issue to
I like the future idea of writing a script that converts sgfs to the current format. That would have the best of both worlds.
I am not quite sure what your reg_ownership command is designed to test. Is the idea to test what the ownership of a specific point is, according to the playouts? In that case I would recommend something like:
reg_ownership <position> <color> <ratio necessary>
and have the command simply output TRUE or FALSE. Would this be sufficient?
PS: I have assigned this to you, mostly because you seem to be working on it already.
-
assigned issue to
-
reporter fuego regression tests and regression tests from the last KGS run added in my branch.
-
- attached create_expected.pl
Attaching a (slightly edited) perl script written by Detlef for this issue. See discussion in http://sourceforge.net/p/fuego/mailman/message/32664071/ and https://groups.google.com/forum/#!topic/oakfoam/UK3LEg_RRDg for other discussion related to this issue. Edits from the version Detlef posted: (a) dos2unix conversion; (b) applied a fix for a bug Detlef pointed out in the thread; added some TODO comments.
-
Francois: The fuego test suite, which Detlef is incorporating, uses upper-case X-coordinates on the board. In his fork, he has changed the GTP output to upper case, since there are only a couple of trivial tests in the original Oakfoam codebase. Are there any other tools/applications/scripts you use that rely on lower-case output of board coordinates?
-
repo owner I can't recall if there are any other places where the case matters, but including a switch can always be done if an issue crops up at a later stage.
- Log in to comment
I don't think Point 1 is true, any commands can be inserted into .tst files, not just reg_genmove. The appropriate "#? []" comments will just need to be added. I haven't tested this, but I assume it will work.
Point 2 is debatable in my opinion. I think that the current method is very powerful, and I don't think it is too difficult, but I understand that you might disagree with the second part.
For interest's sake, I believe Pachi makes use of a system like you describe. I would be interested to find out more details.
At the moment the workflow is:
PS: I have added you as a member of the oakfoam repo with read access, which seems to show that you can have bugs assigned to you now.