Package org.languagetool.tokenizers.ro
Class RomanianWordTokenizer
java.lang.Object
org.languagetool.tokenizers.WordTokenizer
org.languagetool.tokenizers.ro.RomanianWordTokenizer
- All Implemented Interfaces:
org.languagetool.tokenizers.Tokenizer
public class RomanianWordTokenizer
extends org.languagetool.tokenizers.WordTokenizer
Tokenizes a sentence into words. Punctuation and whitespace gets its own
token. Like EnglishWordTokenizer except for some characters: eg: "-'
- Since:
- 20.02.2009 19:53:50
-
Constructor Summary
Constructors -
Method Summary
Methods inherited from class org.languagetool.tokenizers.WordTokenizer
getProtocols, getTokenizingCharacters, isEMail, isUrl, joinEMails, joinEMailsAndUrls, joinUrls
-
Constructor Details
-
RomanianWordTokenizer
public RomanianWordTokenizer()
-
-
Method Details
-
tokenize
- Specified by:
tokenize
in interfaceorg.languagetool.tokenizers.Tokenizer
- Overrides:
tokenize
in classorg.languagetool.tokenizers.WordTokenizer
-