Module org.apache.lucene.core
Class Lucene99HnswVectorsFormat
- java.lang.Object
-
- org.apache.lucene.codecs.KnnVectorsFormat
-
- org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsFormat
-
- All Implemented Interfaces:
NamedSPILoader.NamedSPI
public final class Lucene99HnswVectorsFormat extends KnnVectorsFormat
Lucene 9.9 vector format, which encodes numeric vector values into an associated graph connecting the documents having values. The graph is used to power HNSW search. The format consists of two files, and requires aFlatVectorsFormat
to store the actual vectors:.vex (vector index)
Stores graphs connecting the documents for each field organized as a list of nodes' neighbours as following:
- For each level:
- For each node:
- [vint] the number of neighbor nodes
- array[vint] the delta encoded neighbor ordinals
- For each node:
- After all levels are encoded memory offsets for each node's neighbor nodes encoded by
DirectMonotonicWriter
are appened to the end of the file.
.vem (vector metadata) file
For each field:
- [int32] field number
- [int32] vector similarity function ordinal
- [vlong] offset to this field's index in the .vex file
- [vlong] length of this field's index data, in bytes
- [vint] dimension of this field's vectors
- [int] the number of documents having values for this field
- [int8] if equals to -1, dense – all documents have values for a field. If equals to 0, sparse – some documents missing values.
- DocIds were encoded by
IndexedDISI.writeBitSet(DocIdSetIterator, IndexOutput, byte)
- OrdToDoc was encoded by
DirectMonotonicWriter
, note that only in sparse case - [vint] the maximum number of connections (neigbours) that each node can have
- [vint] number of levels in the graph
- Graph nodes by level. For each level
- [vint] the number of nodes on this level
- array[vint] for levels greater than 0 list of nodes on this level, stored as the level 0th delta encoded nodes' ordinals.
-
-
Field Summary
Fields Modifier and Type Field Description private int
beamWidth
The number of candidate neighbors to track while searching the graph for each newly inserted node.static int
DEFAULT_BEAM_WIDTH
Default number of the size of the queue maintained while searching during a graph construction.static int
DEFAULT_MAX_CONN
Default number of maximum connections per nodestatic int
DEFAULT_NUM_MERGE_WORKER
Default to use single thread merge(package private) static int
DIRECT_MONOTONIC_BLOCK_SHIFT
private static FlatVectorsFormat
flatVectorsFormat
The format for storing, reading, merging vectors on diskprivate int
maxConn
Controls how many of the nearest neighbor candidates are connected to the new node.(package private) static int
MAXIMUM_BEAM_WIDTH
The maximum size of the queue to maintain while searching during graph construction This maximum value preserves the ratio of the DEFAULT_BEAM_WIDTH/DEFAULT_MAX_CONN i.e.(package private) static int
MAXIMUM_MAX_CONN
A maximum configurable maximum max conn.private TaskExecutor
mergeExec
(package private) static java.lang.String
META_CODEC_NAME
(package private) static java.lang.String
META_EXTENSION
private int
numMergeWorkers
(package private) static java.lang.String
VECTOR_INDEX_CODEC_NAME
(package private) static java.lang.String
VECTOR_INDEX_EXTENSION
static int
VERSION_CURRENT
static int
VERSION_START
-
Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat
DEFAULT_MAX_DIMENSIONS, EMPTY
-
-
Constructor Summary
Constructors Constructor Description Lucene99HnswVectorsFormat()
Constructs a format using default graph construction parametersLucene99HnswVectorsFormat(int maxConn, int beamWidth)
Constructs a format using the given graph construction parameters.Lucene99HnswVectorsFormat(int maxConn, int beamWidth, int numMergeWorkers, java.util.concurrent.ExecutorService mergeExec)
Constructs a format using the given graph construction parameters and scalar quantization.
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description KnnVectorsReader
fieldsReader(SegmentReadState state)
Returns aKnnVectorsReader
to read the vectors from the index.KnnVectorsWriter
fieldsWriter(SegmentWriteState state)
Returns aKnnVectorsWriter
to write the vectors to the index.int
getMaxDimensions(java.lang.String fieldName)
Returns the maximum number of vector dimensions supported by this codec for the given field namejava.lang.String
toString()
-
Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat
forName, getName
-
-
-
-
Field Detail
-
META_CODEC_NAME
static final java.lang.String META_CODEC_NAME
- See Also:
- Constant Field Values
-
VECTOR_INDEX_CODEC_NAME
static final java.lang.String VECTOR_INDEX_CODEC_NAME
- See Also:
- Constant Field Values
-
META_EXTENSION
static final java.lang.String META_EXTENSION
- See Also:
- Constant Field Values
-
VECTOR_INDEX_EXTENSION
static final java.lang.String VECTOR_INDEX_EXTENSION
- See Also:
- Constant Field Values
-
VERSION_START
public static final int VERSION_START
- See Also:
- Constant Field Values
-
VERSION_CURRENT
public static final int VERSION_CURRENT
- See Also:
- Constant Field Values
-
MAXIMUM_MAX_CONN
static final int MAXIMUM_MAX_CONN
A maximum configurable maximum max conn.NOTE: We eagerly populate `float[MAX_CONN*2]` and `int[MAX_CONN*2]`, so exceptionally large numbers here will use an inordinate amount of heap
- See Also:
- Constant Field Values
-
DEFAULT_MAX_CONN
public static final int DEFAULT_MAX_CONN
Default number of maximum connections per node- See Also:
- Constant Field Values
-
MAXIMUM_BEAM_WIDTH
static final int MAXIMUM_BEAM_WIDTH
The maximum size of the queue to maintain while searching during graph construction This maximum value preserves the ratio of the DEFAULT_BEAM_WIDTH/DEFAULT_MAX_CONN i.e. `6.25 * 16 = 3200`- See Also:
- Constant Field Values
-
DEFAULT_BEAM_WIDTH
public static final int DEFAULT_BEAM_WIDTH
Default number of the size of the queue maintained while searching during a graph construction.- See Also:
- Constant Field Values
-
DEFAULT_NUM_MERGE_WORKER
public static final int DEFAULT_NUM_MERGE_WORKER
Default to use single thread merge- See Also:
- Constant Field Values
-
DIRECT_MONOTONIC_BLOCK_SHIFT
static final int DIRECT_MONOTONIC_BLOCK_SHIFT
- See Also:
- Constant Field Values
-
maxConn
private final int maxConn
Controls how many of the nearest neighbor candidates are connected to the new node. Defaults toDEFAULT_MAX_CONN
. SeeHnswGraph
for more details.
-
beamWidth
private final int beamWidth
The number of candidate neighbors to track while searching the graph for each newly inserted node. Defaults to toDEFAULT_BEAM_WIDTH
. SeeHnswGraph
for details.
-
flatVectorsFormat
private static final FlatVectorsFormat flatVectorsFormat
The format for storing, reading, merging vectors on disk
-
numMergeWorkers
private final int numMergeWorkers
-
mergeExec
private final TaskExecutor mergeExec
-
-
Constructor Detail
-
Lucene99HnswVectorsFormat
public Lucene99HnswVectorsFormat()
Constructs a format using default graph construction parameters
-
Lucene99HnswVectorsFormat
public Lucene99HnswVectorsFormat(int maxConn, int beamWidth)
Constructs a format using the given graph construction parameters.- Parameters:
maxConn
- the maximum number of connections to a node in the HNSW graphbeamWidth
- the size of the queue maintained during graph construction.
-
Lucene99HnswVectorsFormat
public Lucene99HnswVectorsFormat(int maxConn, int beamWidth, int numMergeWorkers, java.util.concurrent.ExecutorService mergeExec)
Constructs a format using the given graph construction parameters and scalar quantization.- Parameters:
maxConn
- the maximum number of connections to a node in the HNSW graphbeamWidth
- the size of the queue maintained during graph construction.numMergeWorkers
- number of workers (threads) that will be used when doing merge. If larger than 1, a non-nullExecutorService
must be passed as mergeExecmergeExec
- theExecutorService
that will be used by ALL vector writers that are generated by this format to do the merge
-
-
Method Detail
-
fieldsWriter
public KnnVectorsWriter fieldsWriter(SegmentWriteState state) throws java.io.IOException
Description copied from class:KnnVectorsFormat
Returns aKnnVectorsWriter
to write the vectors to the index.- Specified by:
fieldsWriter
in classKnnVectorsFormat
- Throws:
java.io.IOException
-
fieldsReader
public KnnVectorsReader fieldsReader(SegmentReadState state) throws java.io.IOException
Description copied from class:KnnVectorsFormat
Returns aKnnVectorsReader
to read the vectors from the index.- Specified by:
fieldsReader
in classKnnVectorsFormat
- Throws:
java.io.IOException
-
getMaxDimensions
public int getMaxDimensions(java.lang.String fieldName)
Description copied from class:KnnVectorsFormat
Returns the maximum number of vector dimensions supported by this codec for the given field nameCodecs should override this method to specify the maximum number of dimensions they support.
- Overrides:
getMaxDimensions
in classKnnVectorsFormat
- Parameters:
fieldName
- the field name- Returns:
- the maximum number of vector dimensions.
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
-