| Tanl Linguistic Pipeline |
Represents a token recognized by the scanner. More...
#include <Tokenizer.h>
Public Member Functions | |
| Token (char const *text=0, TokenType type=(TokenType) 0) | |
| Token const & | operator= (Token const &o) |
| void | serialize (std::ostream &stream) |
Public Attributes | |
| char const * | text |
| the text of the token | |
| TokenType | type |
| the type of the token | |
Represents a token recognized by the scanner.