Class TokenBuffer.Segment

  • Enclosing class:
    TokenBuffer

    protected static final class TokenBuffer.Segment
    extends java.lang.Object
    Individual segment of TokenBuffer that can store up to 16 tokens (limited by 4 bits per token type marker requirement). Current implementation uses fixed length array; could alternatively use 16 distinct fields and switch statement (slightly more efficient storage, slightly slower access)
    • Field Detail

      • _tokenTypes

        protected long _tokenTypes
        Bit field used to store types of buffered tokens; 4 bits per token. Value 0 is reserved for "not in use"
      • _tokens

        protected final java.lang.Object[] _tokens
    • Constructor Detail

      • Segment

        public Segment()