create proper testing for the regex token definition

[Ali Hürriyetoglu]

Little changes may have big effects. Therefore, create proper test cases for the regular expression being used for identifying tokens.