Class KoreanNumberFilter

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, Unwrappable<TokenStream>

    public class KoreanNumberFilter
    extends TokenFilter
    A TokenFilter that normalizes Korean numbers to regular Arabic decimal numbers in half-width characters.

    Korean numbers are often written using a combination of Hangul and Arabic numbers with various kinds punctuation. For example, 3.2천 means 3200. This filter does this kind of normalization and allows a search for 3200 to match 3.2천 in text, but can also be used to make range facets based on the normalized numbers and so on.

    Notice that this analyzer uses a token composition scheme and relies on punctuation tokens being found in the token stream. Please make sure your KoreanTokenizer has discardPunctuation set to false. In case punctuation characters, such as . (U+FF0E FULLWIDTH FULL STOP), is removed from the token stream, this filter would find input tokens tokens 3 and 2천 and give outputs 3 and 2000 instead of 3200, which is likely not the intended result. If you want to remove punctuation characters from your index that are not part of normalized numbers, add a StopFilter with the punctuation you wish to remove after KoreanNumberFilter in your analyzer chain.

    Below are some examples of normalizations this filter supports. The input is untokenized text and the result is the single term attribute emitted for the input.

    • 영영칠 becomes 7
    • 일영영영 becomes 1000
    • 삼천2백2십삼 becomes 3223
    • 조육백만오천일 becomes 1000006005001
    • 3.2천 becomes 3200
    • 1.2만345.67 becomes 12345.67
    • 4,647.100 becomes 4647.1
    • 15,7 becomes 157 (be aware of this weakness)

    Tokens preceded by a token with PositionIncrementAttribute of zero are left left untouched and emitted as-is.

    This filter does not use any part-of-speech information for its normalization and the motivation for this is to also support n-grammed token streams in the future.

    This filter may in some cases normalize tokens that are not numbers in their context. For example, is 전중경일 is a name and means Tanaka Kyōichi, but 경일 (Kyōichi) out of context can strictly speaking also represent the number 10000000000000001. This filter respects the KeywordAttribute, which can be used to prevent specific normalizations from happening.

    • Constructor Detail

      • KoreanNumberFilter

        public KoreanNumberFilter​(TokenStream input)
    • Method Detail

      • incrementToken

        public final boolean incrementToken()
                                     throws java.io.IOException
        Description copied from class: TokenStream
        Consumers (i.e., IndexWriter) use this method to advance the stream to the next token. Implementing classes must implement this method and update the appropriate AttributeImpls with the attributes of the next token.

        The producer must make no assumptions about the attributes after the method has been returned: the caller may arbitrarily change it. If the producer needs to preserve the state for subsequent calls, it can use AttributeSource.captureState() to create a copy of the current attribute state.

        This method is called for every token of a document, so an efficient implementation is crucial for good performance. To avoid calls to AttributeSource.addAttribute(Class) and AttributeSource.getAttribute(Class), references to all AttributeImpls that this stream uses should be retrieved during instantiation.

        To ensure that filters and consumers know which attributes are available, the attributes must be added during instantiation. Filters and consumers are not required to check for availability of attributes in TokenStream.incrementToken().

        Specified by:
        incrementToken in class TokenStream
        Returns:
        false for end of stream; true otherwise
        Throws:
        java.io.IOException
      • reset

        public void reset()
                   throws java.io.IOException
        Description copied from class: TokenFilter
        This method is called by a consumer before it begins consumption using TokenStream.incrementToken().

        Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

        If you override this method, always call super.reset(), otherwise some internal state will not be correctly reset (e.g., Tokenizer will throw IllegalStateException on further usage).

        NOTE: The default implementation chains the call to the input TokenStream, so be sure to call super.reset() when overriding this method.

        Overrides:
        reset in class TokenFilter
        Throws:
        java.io.IOException
      • normalizeNumber

        public java.lang.String normalizeNumber​(java.lang.String number)
        Normalizes a Korean number
        Parameters:
        number - number or normalize
        Returns:
        normalized number, or number to normalize on error (no op)
      • parseNumber

        private java.math.BigDecimal parseNumber​(KoreanNumberFilter.NumberBuffer buffer)
        Parses a Korean number
        Parameters:
        buffer - buffer to parse
        Returns:
        parsed number, or null on error or end of input
      • parseLargePair

        private java.math.BigDecimal parseLargePair​(KoreanNumberFilter.NumberBuffer buffer)
        Parses a pair of large numbers, i.e. large Hangul factor is 10,000(만)or larger
        Parameters:
        buffer - buffer to parse
        Returns:
        parsed pair, or null on error or end of input
      • parseMediumNumber

        private java.math.BigDecimal parseMediumNumber​(KoreanNumberFilter.NumberBuffer buffer)
        Parses a "medium sized" number, typically less than 10,000(만), but might be larger due to a larger factor from {link parseBasicNumber}.
        Parameters:
        buffer - buffer to parse
        Returns:
        parsed number, or null on error or end of input
      • parseMediumPair

        private java.math.BigDecimal parseMediumPair​(KoreanNumberFilter.NumberBuffer buffer)
        Parses a pair of "medium sized" numbers, i.e. large Hangul factor is at most 1,000(천)
        Parameters:
        buffer - buffer to parse
        Returns:
        parsed pair, or null on error or end of input
      • parseBasicNumber

        private java.math.BigDecimal parseBasicNumber​(KoreanNumberFilter.NumberBuffer buffer)
        Parse a basic number, which is a sequence of Arabic numbers or a sequence or 0-9 Hangul numerals (영 to 구).
        Parameters:
        buffer - buffer to parse
        Returns:
        parsed number, or null on error or end of input
      • parseLargeHangulNumeral

        public java.math.BigDecimal parseLargeHangulNumeral​(KoreanNumberFilter.NumberBuffer buffer)
        Parse large Hangul numerals (ten thousands or larger)
        Parameters:
        buffer - buffer to parse
        Returns:
        parsed number, or null on error or end of input
      • parseMediumHangulNumeral

        public java.math.BigDecimal parseMediumHangulNumeral​(KoreanNumberFilter.NumberBuffer buffer)
        Parse medium Hangul numerals (tens, hundreds or thousands)
        Parameters:
        buffer - buffer to parse
        Returns:
        parsed number or null on error
      • isNumeral

        public boolean isNumeral​(java.lang.String input)
        Numeral predicate
        Parameters:
        input - string to test
        Returns:
        true if and only if input is a numeral
      • isNumeral

        public boolean isNumeral​(char c)
        Numeral predicate
        Parameters:
        c - character to test
        Returns:
        true if and only if c is a numeral
      • isNumeralPunctuation

        public boolean isNumeralPunctuation​(java.lang.String input)
        Numeral punctuation predicate
        Parameters:
        input - string to test
        Returns:
        true if and only if c is a numeral punctuation string
      • isNumeralPunctuation

        public boolean isNumeralPunctuation​(char c)
        Numeral punctuation predicate
        Parameters:
        c - character to test
        Returns:
        true if and only if c is a numeral punctuation character
      • isArabicNumeral

        public boolean isArabicNumeral​(char c)
        Arabic numeral predicate. Both half-width and full-width characters are supported
        Parameters:
        c - character to test
        Returns:
        true if and only if c is an Arabic numeral
      • isHalfWidthArabicNumeral

        private boolean isHalfWidthArabicNumeral​(char c)
        Arabic half-width numeral predicate
        Parameters:
        c - character to test
        Returns:
        true if and only if c is a half-width Arabic numeral
      • isFullWidthArabicNumeral

        private boolean isFullWidthArabicNumeral​(char c)
        Arabic full-width numeral predicate
        Parameters:
        c - character to test
        Returns:
        true if and only if c is a full-width Arabic numeral
      • arabicNumeralValue

        private int arabicNumeralValue​(char c)
        Returns the numeric value for the specified character Arabic numeral. Behavior is undefined if a non-Arabic numeral is provided
        Parameters:
        c - arabic numeral character
        Returns:
        numeral value
      • isHangulNumeral

        private boolean isHangulNumeral​(char c)
        Hangul numeral predicate that tests if the provided character is one of 영, 일, 이, 삼, 사, 오, 육, 칠, 팔, or 구. Larger number Hangul gives a false value.
        Parameters:
        c - character to test
        Returns:
        true if and only is character is one of 영, 일, 이, 삼, 사, 오, 육, 칠, 팔, or 구 (0 to 9)
      • HangulNumeralValue

        private int HangulNumeralValue​(char c)
        Returns the value for the provided Hangul numeral. Only numeric values for the characters where {link isHangulNumeral} return true are supported - behavior is undefined for other characters.
        Parameters:
        c - Hangul numeral character
        Returns:
        numeral value
        See Also:
        isHangulNumeral(char)
      • isDecimalPoint

        private boolean isDecimalPoint​(char c)
        Decimal point predicate
        Parameters:
        c - character to test
        Returns:
        true if and only if c is a decimal point
      • isThousandSeparator

        private boolean isThousandSeparator​(char c)
        Thousand separator predicate
        Parameters:
        c - character to test
        Returns:
        true if and only if c is a thousand separator predicate