Class Lucene99ScalarQuantizedVectorsWriter

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, Accountable

    public final class Lucene99ScalarQuantizedVectorsWriter
    extends FlatVectorsWriter
    Writes quantized vector values and metadata to index segments.
    • Constructor Detail

      • Lucene99ScalarQuantizedVectorsWriter

        Lucene99ScalarQuantizedVectorsWriter​(SegmentWriteState state,
                                             java.lang.Float confidenceInterval,
                                             FlatVectorsWriter rawVectorDelegate)
                                      throws java.io.IOException
        Throws:
        java.io.IOException
    • Method Detail

      • addField

        public FlatFieldVectorsWriter<?> addField​(FieldInfo fieldInfo,
                                                  KnnFieldVectorsWriter<?> indexWriter)
                                           throws java.io.IOException
        Description copied from class: FlatVectorsWriter
        Add a new field for indexing, allowing the user to provide a writer that the flat vectors writer can delegate to if additional indexing logic is required.
        Specified by:
        addField in class FlatVectorsWriter
        Parameters:
        fieldInfo - fieldInfo of the field to add
        indexWriter - the writer to delegate to, can be null
        Returns:
        a writer for the field
        Throws:
        java.io.IOException - if an I/O error occurs when adding the field
      • mergeOneFieldToIndex

        public CloseableRandomVectorScorerSupplier mergeOneFieldToIndex​(FieldInfo fieldInfo,
                                                                        MergeState mergeState)
                                                                 throws java.io.IOException
        Description copied from class: FlatVectorsWriter
        Write the field for merging, providing a scorer over the newly merged flat vectors. This way any additional merging logic can be implemented by the user of this class.
        Specified by:
        mergeOneFieldToIndex in class FlatVectorsWriter
        Parameters:
        fieldInfo - fieldInfo of the field to merge
        mergeState - mergeState of the segments to merge
        Returns:
        a scorer over the newly merged flat vectors, which should be closed as it holds a temporary file handle to read over the newly merged vectors
        Throws:
        java.io.IOException - if an I/O error occurs when merging
      • finish

        public void finish()
                    throws java.io.IOException
        Description copied from class: FlatVectorsWriter
        Called once at the end before close
        Specified by:
        finish in class FlatVectorsWriter
        Throws:
        java.io.IOException
      • ramBytesUsed

        public long ramBytesUsed()
        Description copied from interface: Accountable
        Return the memory usage of this object in bytes. Negative values are illegal.
      • writeMeta

        private void writeMeta​(FieldInfo field,
                               int maxDoc,
                               long vectorDataOffset,
                               long vectorDataLength,
                               java.lang.Float confidenceInterval,
                               java.lang.Float lowerQuantile,
                               java.lang.Float upperQuantile,
                               DocsWithFieldSet docsWithField)
                        throws java.io.IOException
        Throws:
        java.io.IOException
      • mergeQuantiles

        static ScalarQuantizer mergeQuantiles​(java.util.List<ScalarQuantizer> quantizationStates,
                                              java.util.List<java.lang.Integer> segmentSizes,
                                              float confidenceInterval)
      • shouldRecomputeQuantiles

        static boolean shouldRecomputeQuantiles​(ScalarQuantizer mergedQuantizationState,
                                                java.util.List<ScalarQuantizer> quantizationStates)
        Returns true if the quantiles of the merged state are too far from the quantiles of the individual states.
        Parameters:
        mergedQuantizationState - The merged quantization state
        quantizationStates - The quantization states of the individual segments
        Returns:
        true if the quantiles should be recomputed
      • mergeAndRecalculateQuantiles

        static ScalarQuantizer mergeAndRecalculateQuantiles​(MergeState mergeState,
                                                            FieldInfo fieldInfo,
                                                            float confidenceInterval)
                                                     throws java.io.IOException
        Throws:
        java.io.IOException
      • shouldRequantize

        static boolean shouldRequantize​(ScalarQuantizer existingQuantiles,
                                        ScalarQuantizer newQuantiles)
        Returns true if the quantiles of the new quantization state are too far from the quantiles of the existing quantization state. This would imply that floating point values would slightly shift quantization buckets.
        Parameters:
        existingQuantiles - The existing quantiles for a segment
        newQuantiles - The new quantiles for a segment, could be merged, or fully re-calculated
        Returns:
        true if the floating point values should be requantized
      • writeQuantizedVectorData

        private static DocsWithFieldSet writeQuantizedVectorData​(IndexOutput output,
                                                                 QuantizedByteVectorValues quantizedByteVectorValues)
                                                          throws java.io.IOException
        Writes the vector values to the output and returns a set of documents that contains vectors.
        Throws:
        java.io.IOException
      • close

        public void close()
                   throws java.io.IOException
        Throws:
        java.io.IOException