Class Snappy
java.lang.Object
io.netty.handler.codec.compression.Snappy
Uncompresses an input
ByteBuf
encoded with Snappy compression into an
output ByteBuf
.
See snappy format.-
Nested Class Summary
Nested Classes -
Field Summary
FieldsModifier and TypeFieldDescriptionprivate static final int
private static final int
private static final int
private static final boolean
private static final FastThreadLocal
<short[]> private static final int
private static final int
private static final int
private static final int
private static final int
private final boolean
private Snappy.State
private byte
private int
-
Constructor Summary
Constructors -
Method Summary
Modifier and TypeMethodDescriptionprivate static int
bitsToEncode
(int value) Calculates the minimum number of bits required to encode a value.(package private) static int
calculateChecksum
(ByteBuf data) Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum(package private) static int
calculateChecksum
(ByteBuf data, int offset, int length) Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksumvoid
private static int
decodeCopyWith1ByteOffset
(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar) Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.private static int
decodeCopyWith2ByteOffset
(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar) Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.private static int
decodeCopyWith4ByteOffset
(byte tag, ByteBuf in, ByteBuf out, int writtenSoFar) Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.(package private) static int
decodeLiteral
(byte tag, ByteBuf in, ByteBuf out) Reads a literal from the input buffer directly to the output buffer.void
private static void
encodeCopy
(ByteBuf out, int offset, int length) Encodes a series of copies, each at most 64 bytes in length.private static void
encodeCopyWithOffset
(ByteBuf out, int offset, int length) (package private) static void
encodeLiteral
(ByteBuf in, ByteBuf out, int length) Writes a literal to the supplied output buffer by directly copying from the input buffer.private static int
findMatchingLength
(ByteBuf in, int minIndex, int inIndex, int maxIndex) Iterates over the supplied input buffer between the supplied minIndex and maxIndex to find how long our matched copy overlaps with an already-written literal value.private short[]
getHashTable
(int hashTableSize) Returns a short[] to be used as a hashtablestatic short[]
getHashTableFastThreadLocalArrayFill
(int hashTableSize) Returns a short[] from a FastThreadLocal, zeroing for correctness creating a new one and resizing it if necessary(package private) int
getPreamble
(ByteBuf in) Get the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).private static int
Hashes the 4 bytes located at index, shifting the resulting hash into the appropriate range for our hash table.(package private) static int
maskChecksum
(long checksum) From the spec: "Checksums are not stored directly, but masked, as checksumming data and then its own checksum can be problematic.private static int
readPreamble
(ByteBuf in) Reads the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).void
reset()
(package private) static void
validateChecksum
(int expectedChecksum, ByteBuf data) Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.(package private) static void
validateChecksum
(int expectedChecksum, ByteBuf data, int offset, int length) Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.private static void
validateOffset
(int offset, int chunkSizeSoFar) Validates that the offset extracted from a compressed reference is within the permissible bounds of an offset (0 invalid input: '<' offset invalid input: '<' Integer.MAX_VALUE), and does not exceed the length of the chunk currently read so far.static Snappy
-
Field Details
-
MAX_HT_SIZE
private static final int MAX_HT_SIZE- See Also:
-
MIN_COMPRESSIBLE_BYTES
private static final int MIN_COMPRESSIBLE_BYTES- See Also:
-
PREAMBLE_NOT_FULL
private static final int PREAMBLE_NOT_FULL- See Also:
-
NOT_ENOUGH_INPUT
private static final int NOT_ENOUGH_INPUT- See Also:
-
LITERAL
private static final int LITERAL- See Also:
-
COPY_1_BYTE_OFFSET
private static final int COPY_1_BYTE_OFFSET- See Also:
-
COPY_2_BYTE_OFFSET
private static final int COPY_2_BYTE_OFFSET- See Also:
-
COPY_4_BYTE_OFFSET
private static final int COPY_4_BYTE_OFFSET- See Also:
-
HASH_TABLE
-
DEFAULT_REUSE_HASHTABLE
private static final boolean DEFAULT_REUSE_HASHTABLE -
reuseHashtable
private final boolean reuseHashtable -
state
-
tag
private byte tag -
written
private int written
-
-
Constructor Details
-
Snappy
public Snappy() -
Snappy
Snappy(boolean reuseHashtable)
-
-
Method Details
-
withHashTableReuse
-
reset
public void reset() -
encode
-
hash
Hashes the 4 bytes located at index, shifting the resulting hash into the appropriate range for our hash table.- Parameters:
in
- The input buffer to read 4 bytes fromindex
- The index to read atshift
- The shift value, for ensuring that the resulting value is within the range of our hash table size- Returns:
- A 32-bit hash of 4 bytes located at index
-
getHashTable
private short[] getHashTable(int hashTableSize) Returns a short[] to be used as a hashtable- Parameters:
hashTableSize
- the size for the hashtable- Returns:
- An appropriately sized empty hashtable
-
getHashTableFastThreadLocalArrayFill
public static short[] getHashTableFastThreadLocalArrayFill(int hashTableSize) Returns a short[] from a FastThreadLocal, zeroing for correctness creating a new one and resizing it if necessary- Parameters:
hashTableSize
-- Returns:
- An appropriately sized empty hashtable
-
findMatchingLength
Iterates over the supplied input buffer between the supplied minIndex and maxIndex to find how long our matched copy overlaps with an already-written literal value.- Parameters:
in
- The input buffer to scan overminIndex
- The index in the input buffer to start scanning frominIndex
- The index of the start of our copymaxIndex
- The length of our input buffer- Returns:
- The number of bytes for which our candidate copy is a repeat of
-
bitsToEncode
private static int bitsToEncode(int value) Calculates the minimum number of bits required to encode a value. This can then in turn be used to calculate the number of septets or octets (as appropriate) to use to encode a length parameter.- Parameters:
value
- The value to calculate the minimum number of bits required to encode- Returns:
- The minimum number of bits required to encode the supplied value
-
encodeLiteral
Writes a literal to the supplied output buffer by directly copying from the input buffer. The literal is taken from the current readerIndex up to the supplied length.- Parameters:
in
- The input buffer to copy fromout
- The output buffer to copy tolength
- The length of the literal to copy
-
encodeCopyWithOffset
-
encodeCopy
Encodes a series of copies, each at most 64 bytes in length.- Parameters:
out
- The output buffer to write the copy pointer tooffset
- The offset at which the original instance lieslength
- The length of the original instance
-
decode
-
readPreamble
Reads the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).- Parameters:
in
- The input buffer to read the preamble from- Returns:
- The calculated length based on the input buffer, or 0 if no preamble is able to be calculated
-
getPreamble
Get the length varint (a series of bytes, where the lower 7 bits are data and the upper bit is a flag to indicate more bytes to be read).- Parameters:
in
- The input buffer to get the preamble from- Returns:
- The calculated length based on the input buffer, or 0 if no preamble is able to be calculated
-
decodeLiteral
Reads a literal from the input buffer directly to the output buffer. A "literal" is an uncompressed segment of data stored directly in the byte stream.- Parameters:
tag
- The tag that identified this segment as a literal is also used to encode part of the length of the datain
- The input buffer to read the literal fromout
- The output buffer to write the literal to- Returns:
- The number of bytes appended to the output buffer, or -1 to indicate "try again later"
-
decodeCopyWith1ByteOffset
Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.- Parameters:
tag
- The tag used to identify this as a copy is also used to encode the length and part of the offsetin
- The input buffer to read fromout
- The output buffer to write to- Returns:
- The number of bytes appended to the output buffer, or -1 to indicate "try again later"
- Throws:
DecompressionException
- If the read offset is invalid
-
decodeCopyWith2ByteOffset
Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.- Parameters:
tag
- The tag used to identify this as a copy is also used to encode the length and part of the offsetin
- The input buffer to read fromout
- The output buffer to write to- Returns:
- The number of bytes appended to the output buffer, or -1 to indicate "try again later"
- Throws:
DecompressionException
- If the read offset is invalid
-
decodeCopyWith4ByteOffset
Reads a compressed reference offset and length from the supplied input buffer, seeks back to the appropriate place in the input buffer and writes the found data to the supplied output stream.- Parameters:
tag
- The tag used to identify this as a copy is also used to encode the length and part of the offsetin
- The input buffer to read fromout
- The output buffer to write to- Returns:
- The number of bytes appended to the output buffer, or -1 to indicate "try again later"
- Throws:
DecompressionException
- If the read offset is invalid
-
validateOffset
private static void validateOffset(int offset, int chunkSizeSoFar) Validates that the offset extracted from a compressed reference is within the permissible bounds of an offset (0 invalid input: '<' offset invalid input: '<' Integer.MAX_VALUE), and does not exceed the length of the chunk currently read so far.- Parameters:
offset
- The offset extracted from the compressed referencechunkSizeSoFar
- The number of bytes read so far from this chunk- Throws:
DecompressionException
- if the offset is invalid
-
calculateChecksum
Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum- Parameters:
data
- The input data to calculate the CRC32C checksum of
-
calculateChecksum
Computes the CRC32C checksum of the supplied data and performs the "mask" operation on the computed checksum- Parameters:
data
- The input data to calculate the CRC32C checksum of
-
validateChecksum
Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.- Parameters:
expectedChecksum
- The checksum decoded from the stream to compare againstdata
- The input data to calculate the CRC32C checksum of- Throws:
DecompressionException
- If the calculated and supplied checksums do not match
-
validateChecksum
Computes the CRC32C checksum of the supplied data, performs the "mask" operation on the computed checksum, and then compares the resulting masked checksum to the supplied checksum.- Parameters:
expectedChecksum
- The checksum decoded from the stream to compare againstdata
- The input data to calculate the CRC32C checksum of- Throws:
DecompressionException
- If the calculated and supplied checksums do not match
-
maskChecksum
static int maskChecksum(long checksum) From the spec: "Checksums are not stored directly, but masked, as checksumming data and then its own checksum can be problematic. The masking is the same as used in Apache Hadoop: Rotate the checksum by 15 bits, then add the constant 0xa282ead8 (using wraparound as normal for unsigned integers)."- Parameters:
checksum
- The actual checksum of the data- Returns:
- The masked checksum
-