opennlp.tools.lang.thai
Class Tokenizer

java.lang.Object
  opennlp.tools.tokenize.TokenizerME
      opennlp.tools.lang.thai.Tokenizer

All Implemented Interfaces:

Tokenizer

public class Tokenizer
extends TokenizerME

Tokenizer for Thai text.

Field Summary

Fields inherited from class opennlp.tools.tokenize.TokenizerME

alphaNumeric

Constructor Summary

Tokenizer(opennlp.maxent.MaxentModel model)

Tokenizer(java.lang.String name)

Method Summary

static void	main(java.lang.String[] args)
java.lang.String[]	tokenize(java.lang.String s) Tokenize a string.

Methods inherited from class opennlp.tools.tokenize.TokenizerME

getTokenProbabilities, setAlphaNumericOptimization, tokenizePos, train, train, useAlphaNumericOptimization

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(java.lang.String name)
          throws java.io.IOException

Throws:

java.io.IOException

Tokenizer

public Tokenizer(opennlp.maxent.MaxentModel model)
          throws java.io.IOException

Throws:

java.io.IOException

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

Throws:

java.io.IOException

tokenize

public java.lang.String[] tokenize(java.lang.String s)

Description copied from interface: Tokenizer

Tokenize a string.

Specified by:

tokenize in interface Tokenizer

Parameters:

s - The string to be tokenized.

Returns:

The String[] with the individual tokens as the array elements.