Class RomanianWordTokenizer

java.lang.Object
org.languagetool.tokenizers.WordTokenizer
org.languagetool.tokenizers.ro.RomanianWordTokenizer
All Implemented Interfaces:
org.languagetool.tokenizers.Tokenizer

public class RomanianWordTokenizer extends org.languagetool.tokenizers.WordTokenizer
Tokenizes a sentence into words. Punctuation and whitespace gets its own token. Like EnglishWordTokenizer except for some characters: eg: "-'
Since:
20.02.2009 19:53:50
  • Constructor Details

    • RomanianWordTokenizer

      public RomanianWordTokenizer()
  • Method Details

    • tokenize

      public List<String> tokenize(String text)
      Specified by:
      tokenize in interface org.languagetool.tokenizers.Tokenizer
      Overrides:
      tokenize in class org.languagetool.tokenizers.WordTokenizer