create proper testing for the regex token definition
Issue #42
new
Little changes may have big effects. Therefore, create proper test cases for the regular expression being used for identifying tokens.