Package opennlp.tools.tokenize

Package related to finding tokens or word segments.

See:
Description

Interface Summary

TokenContextGenerator	Interface for TokenizerME context generators.
Tokenizer	The interface for tokenizers, which turn messy text into nicely segmented text tokens.

Class Summary

DefaultTokenContextGenerator	Generate events for maxent decisions for tokenization.
SimpleTokenizer	Performs tokenization using character classes.
TokenizerME	A Tokenizer for converting raw text into separated tokens.
TokEventCollector	Generate event contexts for maxent decisions for tokenization detection.
TokSpanEventStream	An implementation of EventStream which allows events to be added by offset and returns events for these offset-based tokens.
WhitespaceTokenizer	This tokenizer uses white spaces to tokenize the input text.

Package opennlp.tools.tokenize Description

Package related to finding tokens or word segments.