Uses of Interface
opennlp.tools.tokenize.Tokenizer

Packages that use Tokenizer

opennlp.tools.lang.english	Package related to the processing of English data.
opennlp.tools.namefind	Package related to finding proper names and numeric amounts.
opennlp.tools.tokenize	Package related to finding tokens or word segments.

Uses of Tokenizer in opennlp.tools.lang.english

Classes in opennlp.tools.lang.english that implement Tokenizer

class

Tokenizer A tokenizer which uses default English data for the maxent model.

Uses of Tokenizer in opennlp.tools.namefind

Constructors in opennlp.tools.namefind with parameters of type Tokenizer

TokenPatternFeatureGenerator(Tokenizer supportTokenizer)
Initializes a new instance.

Uses of Tokenizer in opennlp.tools.tokenize

Classes in opennlp.tools.tokenize that implement Tokenizer

class	SimpleTokenizer Performs tokenization using character classes.
class	TokenizerME A Tokenizer for converting raw text into separated tokens.
class	WhitespaceTokenizer This tokenizer uses white spaces to tokenize the input text.