Tanl::TokenizerPipe Struct Reference

Tokenizer pipe connected to a stream. More...

#include <Tokenizer.h>

Inheritance diagram for Tanl::TokenizerPipe:

List of all members.

Public Member Functions
	TokenizerPipe (Tokenizer &tokenizer, std::istream &is)
	Create a pipe pulling from a stream is.
	TokenizerPipe (Tokenizer &tokenizer, Enumerator< std::string * > &se)
	Create a pipe pulling from a string Enumerator se.
bool	MoveNext ()
	Advances to the next element of the collection.
Token *	Current ()
Public Attributes
int	len
	sentence length (except tags)
Protected Attributes
quex::Token	token
quex::quex_tokenizer	qlex
Context *	currContext

---

Detailed Description

Tokenizer pipe connected to a stream.

---

Constructor & Destructor Documentation

Tanl::TokenizerPipe::TokenizerPipe	(	Tokenizer &	tokenizer,
		std::istream &	is
	)

Create a pipe pulling from a stream is.

Tanl::TokenizerPipe::TokenizerPipe	(	Tokenizer &	tokenizer,
		Enumerator< std::string * > &	se
	)

Create a pipe pulling from a string Enumerator se.

---

Member Function Documentation

Token * Tanl::TokenizerPipe::Current

(

)

[virtual]

Returns:

the current element.

Implements Tanl::Enumerator< Token * >.

References Tanl::Token::set().

bool Tanl::TokenizerPipe::MoveNext

(

)

[virtual]

Advances to the next element of the collection.

Returns:

true if there is another item available.

Implements Tanl::Enumerator< Token * >.

Reimplemented in Tanl::TokenizerPipeEnum, and Tanl::TokenizerPipePython.

References len.

---

The documentation for this struct was generated from the following files:

• split/Tokenizer/Tokenizer.h
• split/Tokenizer/Tokenizer.cpp