Class Lucene95HnswVectorsFormat

  • All Implemented Interfaces:
    NamedSPILoader.NamedSPI

    public class Lucene95HnswVectorsFormat
    extends KnnVectorsFormat
    Lucene 9.5 vector format, which encodes numeric vector values and an optional associated graph connecting the documents having values. The graph is used to power HNSW search. The format consists of three files:

    .vec (vector data) file

    For each field:

    • Vector data ordered by field, document ordinal, and vector dimension. When the vectorEncoding is BYTE, each sample is stored as a single byte. When it is FLOAT32, each sample is stored as an IEEE float in little-endian byte order.
    • DocIds encoded by IndexedDISI.writeBitSet(DocIdSetIterator, IndexOutput, byte), note that only in sparse case
    • OrdToDoc was encoded by DirectMonotonicWriter, note that only in sparse case

    .vex (vector index)

    Stores graphs connecting the documents for each field organized as a list of nodes' neighbours as following:

    • For each level:
      • For each node:
        • [vint] the number of neighbor nodes
        • array[vint] the delta encoded neighbor ordinals
    • After all levels are encoded memory offsets for each node's neighbor nodes encoded by DirectMonotonicWriter are appened to the end of the file.

    .vem (vector metadata) file

    For each field:

    • [int32] field number
    • [int32] vector similarity function ordinal
    • [vlong] offset to this field's vectors in the .vec file
    • [vlong] length of this field's vectors, in bytes
    • [vlong] offset to this field's index in the .vex file
    • [vlong] length of this field's index data, in bytes
    • [vint] dimension of this field's vectors
    • [int] the number of documents having values for this field
    • [int8] if equals to -1, dense – all documents have values for a field. If equals to 0, sparse – some documents missing values.
    • DocIds were encoded by IndexedDISI.writeBitSet(DocIdSetIterator, IndexOutput, byte)
    • OrdToDoc was encoded by DirectMonotonicWriter, note that only in sparse case
    • [vint] the maximum number of connections (neigbours) that each node can have
    • [vint] number of levels in the graph
    • Graph nodes by level. For each level
      • [vint] the number of nodes on this level
      • array[vint] for levels greater than 0 list of nodes on this level, stored as the level 0th delta encoded nodes' ordinals.
    • Field Detail

      • VECTOR_DATA_CODEC_NAME

        static final java.lang.String VECTOR_DATA_CODEC_NAME
        See Also:
        Constant Field Values
      • VECTOR_INDEX_CODEC_NAME

        static final java.lang.String VECTOR_INDEX_CODEC_NAME
        See Also:
        Constant Field Values
      • VECTOR_DATA_EXTENSION

        static final java.lang.String VECTOR_DATA_EXTENSION
        See Also:
        Constant Field Values
      • VECTOR_INDEX_EXTENSION

        static final java.lang.String VECTOR_INDEX_EXTENSION
        See Also:
        Constant Field Values
      • MAXIMUM_MAX_CONN

        private static final int MAXIMUM_MAX_CONN
        A maximum configurable maximum max conn.

        NOTE: We eagerly populate `float[MAX_CONN*2]` and `int[MAX_CONN*2]`, so exceptionally large numbers here will use an inordinate amount of heap

        See Also:
        Constant Field Values
      • DEFAULT_MAX_CONN

        public static final int DEFAULT_MAX_CONN
        Default number of maximum connections per node
        See Also:
        Constant Field Values
      • MAXIMUM_BEAM_WIDTH

        private static final int MAXIMUM_BEAM_WIDTH
        The maximum size of the queue to maintain while searching during graph construction This maximum value preserves the ratio of the DEFAULT_BEAM_WIDTH/DEFAULT_MAX_CONN i.e. `6.25 * 16 = 3200`
        See Also:
        Constant Field Values
      • DEFAULT_BEAM_WIDTH

        public static final int DEFAULT_BEAM_WIDTH
        Default number of the size of the queue maintained while searching during a graph construction.
        See Also:
        Constant Field Values
      • DIRECT_MONOTONIC_BLOCK_SHIFT

        static final int DIRECT_MONOTONIC_BLOCK_SHIFT
        See Also:
        Constant Field Values
      • maxConn

        final int maxConn
        Controls how many of the nearest neighbor candidates are connected to the new node. Defaults to DEFAULT_MAX_CONN. See HnswGraph for more details.
      • beamWidth

        final int beamWidth
        The number of candidate neighbors to track while searching the graph for each newly inserted node. Defaults to to DEFAULT_BEAM_WIDTH. See HnswGraph for details.
    • Constructor Detail

      • Lucene95HnswVectorsFormat

        public Lucene95HnswVectorsFormat()
        Constructs a format using default graph construction parameters
      • Lucene95HnswVectorsFormat

        public Lucene95HnswVectorsFormat​(int maxConn,
                                         int beamWidth)
        Constructs a format using the given graph construction parameters.
        Parameters:
        maxConn - the maximum number of connections to a node in the HNSW graph
        beamWidth - the size of the queue maintained during graph construction.