I just made the following comment in the pull request (I am posting it here to access these numbers in the future):
@geisserf I want to get some issues closed, and this is one of them. So I started to make some experiments with this, and it turns out that it is impossible to find a number of simulations that is such that a (approximately) constant runtime is achieved. The actual numbers are:
As you can see, sometimes it takes a lot less time, and sometimes a lot more. It should be possible to improve this, as the difference in runtime - it's not the number of evaluated states, they are the same - could probably be diminished by a more efficient implementation of the state transitions, which should be added with issue #34. My proposal for this is therefore:
I am currently experiments with the same code that lead to the numbers above.
If the results are comparable, we ignore the fact that parsing takes sometimes quite long
Once issue #34 is merged, we re-evaluate the used parameters to see if we find good numbers such that most instances take approximately 2 seconds.
I don't have any objections, but I think we should create a separate issue which considers aspects of a new implementation of state transitions and representation of logical formulae, since this is a very large topic at hand.
I'm currently investigating possible implementations for another project; if you want I can create this issue and write a comment with regard to the problems of the current implementation (the fact that we always create raw pointers and do not have any memory management often leads to problems when we transformed logical expressions for e.g. normal forms of CPFs), features the new implementation should support and consider some possible implementations/concepts which already exist in other work.
These are the results based on the deterministic parser. The only relevant change is the training set, which is used by IDS to learn a reasonable depth, and it seems as if it didn't affect the results much. I'll close the issue later and merge the changes into the master repository.
@geisserf, please go ahead and start the issue. If it is a large task, I'd suggest you create one "super-issue" with reference to multiple smaller ones. That way, we can track the progress of the larger problem in the super-issue, but can merge sub tasks once they have been adressed.