Package org.languagetool.tokenizers.ml
Class MalayalamWordTokenizer
java.lang.Object
org.languagetool.tokenizers.ml.MalayalamWordTokenizer
- All Implemented Interfaces:
org.languagetool.tokenizers.Tokenizer
Tokenizes a sentence into words. Punctuation and whitespace gets its own token.
-
Constructor Summary
Constructors -
Method Summary
-
Constructor Details
-
MalayalamWordTokenizer
public MalayalamWordTokenizer()
-
-
Method Details
-
tokenize
- Specified by:
tokenize
in interfaceorg.languagetool.tokenizers.Tokenizer
-