Source

bllip-parser / sample-text / tokenization_tests.sgml

Full commit
<s> This file has some very strange tokenization tests... </s>
<s> A sentence ending with a parenthesized abbreviation (P.A.) </s>
<s> Fazadjin, V.A. </s>
<s> ``Let's eat cake...'' </s>
<s> He realized the true meaning of the letters "J.W." </s>
<s> She is heading to the U.S.. </s>
<s> There are a lot of people in the U.S. </s>
<s> Born in the U.S </s>
<s> Weird sentence, maybe partially in German, has parentheses and a comma (HELLO), wie z.B. </s>
<s> This sentence has double dashes -- just passed some of them -- but it doesn't end with a period </s>
<s> 1. This sentence is numbered for some reason. </s>
<s> 2) This sentence is also numbered (but for no reason). </s>
<s> The above sentence had a parentheses and a comma!!! </s>
<s> This one ends in an interrobang!? </s>
<s> Two question marks?? </s>
<s> THREE question marks??? </s>
<s> A space and two question marks ?? </s>
<s> One-two--three----four hyphens. </s>
<s> Lots of periods and a question mark......? </s>
<s> -6 Q.R.) </s>
<s> Ain't it a shame? </s>