Class MalayalamWordTokenizer

java.lang.Object
org.languagetool.tokenizers.ml.MalayalamWordTokenizer
All Implemented Interfaces:
org.languagetool.tokenizers.Tokenizer

public class MalayalamWordTokenizer extends Object implements org.languagetool.tokenizers.Tokenizer
Tokenizes a sentence into words. Punctuation and whitespace gets its own token.
  • Constructor Details

    • MalayalamWordTokenizer

      public MalayalamWordTokenizer()
  • Method Details

    • tokenize

      public List<String> tokenize(String text)
      Specified by:
      tokenize in interface org.languagetool.tokenizers.Tokenizer