Tanl::PtbScanner Class Reference

Inheritance diagram for Tanl::PtbScanner:

List of all members.

Classes
struct	Token
	Represents a token recognized by the scanner. More...
Public Types
enum	TokenType {   Eof = 0, Word = 200, Abbrev, Date,   Email, None, Number, Phone,   Special, Tag, Url, Punct,   Eos }
	Token types. More...
Public Member Functions
Token	scan ()
	We define our own method scan() instead of yylex(), since we return a Token instead of int.

---

Member Enumeration Documentation

enum Tanl::PtbScanner::TokenType

Token types.

---

The documentation for this class was generated from the following file:

• split/Tokenizer/PtbTokenizer.h