Tanl::PtbScanner::Token Struct Reference

Represents a token recognized by the scanner. More...

#include <PtbTokenizer.h>

List of all members.

Public Member Functions
	Token (char const *text="", TokenType type=Eof)
	Token (std::string const &text, TokenType type=Eof)
Token const &	operator= (Token const &o)
Public Attributes
char const *	text
	the text of the token
TokenType	type
	the type of the token

---

Detailed Description

Represents a token recognized by the scanner.

---

The documentation for this struct was generated from the following file:

• split/Tokenizer/PtbTokenizer.h