UTF-8 Support

[Sean Kauffman]

nfer should support specifications and events with non-ASCII characters.

This may be fairly simple. I believe that adding support to the specifications should be as simple as being more flexible in what is accepted during parsing. Tests certainly need to be added.

[Sean Kauffman]

This post has a good suggestion for building Flex support for UTF-8 identifiers. https://stackoverflow.com/questions/9611682/flexlexer-support-for-unicode

The gist of it is that you need to add character classes like the following:

ASC     [\x00-\x7f]
ASCN    [\x00-\t\v-\x7f]
U       [\x80-\xbf]
U1      [\x80-\x8f]
U2      [\xc2-\xdf]
U3      [\xe0-\xef]
U4      [\xf0-\xf7]

UANY    {ASC}|{U2}{U}|{U3}{U}{U}|{U4}{U1}{U}{U}
UANYN   {ASCN}|{U2}{U}|{U3}{U}{U}|{U4}{U1}{U}{U} 
UONLY   {U2}{U}|{U3}{U}{U}|{U4}{U1}{U}{U}

Where UANY matches any ASCII or UTF-8 char, UANYN omits newlines, and UONLY omits ASCII.

[Sean Kauffman]

I have spent some time looking at this, and suggested at least one change. As a result, I am changing its status to open.

[Sean Kauffman]

This has been implemented (I'm confused why the related commit message did not resolve the issue or appear here). A functional test was added with Chinese characters for event names to demonstrate that it works.