Some invalid symbols get ignored

[Fabian Maurer]

The following code compiles without errors, and outputs 43. The invalid characters seems to be ignored.

testVar§$!=€@42;
say "test", "$testVar";

[Chimaine]

This can cause unintended behavior, e.g. when confusing != with ~=.

[Chimaine]

I can't seem to get ANTLR to actually recognize these characters to be there; they're just ignored. Has to be investigated further why this is a thing.

[mezzodrinker]

It appears that ANTLR does recognize the invalid characters correctly. For example, when running the following script in the Interactive Console…

testVar§$!=€@42; print(testVar)

…, the following errors are printed to the console…

line 1:7 token recognition error at: '§'
line 1:8 token recognition error at: '$'
line 1:9 token recognition error at: '!'
line 1:11 token recognition error at: '€'
line 1:12 token recognition error at: '@'

…and the script prints 42 to the output afterwards.

Hence, it seems that the error recovery setting of the ANTLR lexer/parser only prints error to stderr without raising any errors in the Java code.

I’m not sure if MTS sets the error recovery manually. Let’s dive into the code.

[mezzodrinker]

Issue is caused by the Mts3Lexer instance only having its default error listener (which prints any errors to stderr and continues as if nothing happened). By adding a custom error handler which functions similarly to the existing MtsAntlrErrorListener class, this issue can be prevented.

[mezzodrinker]

Add error listener to ANTLR lexer (fixes #29)

Renamed MtsAntlrErrorListener to MtsParserErrorListener for clarity

→ <<cset 9ebc48d65496>>