Wiki

Clone wiki

inf225 / glossary / Token

[Alphabetical Index | Tag Index]

Token

A lexeme or group of characters that forms a basic unit of parsing, categorised according to type, e.g., identifier, number, addition operator, etc. Forms the alphabet of the parser in scannerful parsing.

[Wikipedia]

Updated