opennlp.tools.tokenize
Class SimpleTokenizer

java.lang.Object
  opennlp.tools.tokenize.SimpleTokenizer

All Implemented Interfaces:

Tokenizer

public class SimpleTokenizer
extends java.lang.Object

Performs tokenization using character classes.

Author:

tsmorton

Constructor Summary

SimpleTokenizer()

Method Summary

static void	main(java.lang.String[] args)
java.lang.String[]	tokenize(java.lang.String s) Tokenize a string.
Span[]	tokenizePos(java.lang.String s) Tokenize a string.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SimpleTokenizer

public SimpleTokenizer()

Method Detail

tokenizePos

public Span[] tokenizePos(java.lang.String s)

Description copied from interface: Tokenizer

Tokenize a string.

Parameters:

s - The string to be tokenized.

Returns:

The Span[] with the spans (offsets into s) for each token as the individuals array elements.

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Parameters:

args -

Throws:

java.io.IOException

tokenize

public java.lang.String[] tokenize(java.lang.String s)

Description copied from interface: Tokenizer

Tokenize a string.

Specified by:

tokenize in interface Tokenizer

Parameters:

s - The string to be tokenized.

Returns:

The String[] with the individual tokens as the array elements.