newline token always missing after cppcomment token

[Derek Bruening]

I have found that the vera++ parser consistently omits a newline token when there is a C++ comment at the end of a line.

Test case in /tmp/test.cpp:

  if (a)
    return 3;
  if (b) // Comment.
    return 4;

Printing all the tokens with this tcl rule:

foreach fname [getSourceFileNames] {
    foreach token [getTokens $fname 1 0 -1 -1 {}] {
        report $fname [lindex $token 1] "Token: |$token|"
    }
}

The results are below. You can see there is no newline after the cppcomment: it goes straight to the space on the next line.

I'm using a manually-built vera++ from unmodified sources at cf13c413f641b5816a4fc9cd4b2fb248c81bf2bd.

$ vera++ --version
1.3.0
$ vera++ --root xxx --rule print_tokens /tmp/test.cpp
/tmp/test.cpp:1: error: Token: |{  } 1 0 space|
/tmp/test.cpp:1: error: Token: |if 1 2 if|
/tmp/test.cpp:1: error: Token: |{ } 1 4 space|
/tmp/test.cpp:1: error: Token: |( 1 5 leftparen|
/tmp/test.cpp:1: error: Token: |a 1 6 identifier|
/tmp/test.cpp:1: error: Token: |) 1 7 rightparen|
/tmp/test.cpp:1: error: Token: |{
} 1 8 newline|
/tmp/test.cpp:2: error: Token: |{    } 2 0 space|
/tmp/test.cpp:2: error: Token: |return 2 4 return|
/tmp/test.cpp:2: error: Token: |{ } 2 10 space|
/tmp/test.cpp:2: error: Token: |3 2 11 intlit|
/tmp/test.cpp:2: error: Token: |{;} 2 12 semicolon|
/tmp/test.cpp:2: error: Token: |{
} 2 13 newline|
/tmp/test.cpp:3: error: Token: |{  } 3 0 space|
/tmp/test.cpp:3: error: Token: |if 3 2 if|
/tmp/test.cpp:3: error: Token: |{ } 3 4 space|
/tmp/test.cpp:3: error: Token: |( 3 5 leftparen|
/tmp/test.cpp:3: error: Token: |b 3 6 identifier|
/tmp/test.cpp:3: error: Token: |) 3 7 rightparen|
/tmp/test.cpp:3: error: Token: |{ } 3 8 space|
/tmp/test.cpp:3: error: Token: |{// Comment.
} 3 9 cppcomment|
/tmp/test.cpp:4: error: Token: |{    } 4 0 space|
/tmp/test.cpp:4: error: Token: |return 4 4 return|
/tmp/test.cpp:4: error: Token: |{ } 4 10 space|
/tmp/test.cpp:4: error: Token: |4 4 11 intlit|
/tmp/test.cpp:4: error: Token: |{;} 4 12 semicolon|
/tmp/test.cpp:4: error: Token: |{
} 4 13 newline|
/tmp/test.cpp:5: error: Token: |{} 5 0 eof|