pax_global_header00006660000000000000000000000064127224735710014524gustar00rootroot0000000000000052 comment=d9eb7a3d35ec988b8585d4a0068e462c27d28380 golang-github-golang-snappy-0.0+git20160529.d9eb7a3/000077500000000000000000000000001272247357100214345ustar00rootroot00000000000000golang-github-golang-snappy-0.0+git20160529.d9eb7a3/.gitignore000066400000000000000000000005531272247357100234270ustar00rootroot00000000000000cmd/snappytool/snappytool testdata/bench # These explicitly listed benchmark data files are for an obsolete version of # snappy_test.go. testdata/alice29.txt testdata/asyoulik.txt testdata/fireworks.jpeg testdata/geo.protodata testdata/html testdata/html_x_4 testdata/kppkn.gtb testdata/lcet10.txt testdata/paper-100k.pdf testdata/plrabn12.txt testdata/urls.10K golang-github-golang-snappy-0.0+git20160529.d9eb7a3/AUTHORS000066400000000000000000000007441272247357100225110ustar00rootroot00000000000000# This is the official list of Snappy-Go authors for copyright purposes. # This file is distinct from the CONTRIBUTORS files. # See the latter for an explanation. # Names should be added to this file as # Name or Organization # The email address is not required for organizations. # Please keep the list sorted. Damian Gryski Google Inc. Jan Mercl <0xjnml@gmail.com> Rodolfo Carvalho Sebastien Binet golang-github-golang-snappy-0.0+git20160529.d9eb7a3/CONTRIBUTORS000066400000000000000000000026161272247357100233210ustar00rootroot00000000000000# This is the official list of people who can contribute # (and typically have contributed) code to the Snappy-Go repository. # The AUTHORS file lists the copyright holders; this file # lists people. For example, Google employees are listed here # but not in AUTHORS, because Google holds the copyright. # # The submission process automatically checks to make sure # that people submitting code are listed in this file (by email address). # # Names should be added to this file only after verifying that # the individual or the individual's organization has agreed to # the appropriate Contributor License Agreement, found here: # # http://code.google.com/legal/individual-cla-v1.0.html # http://code.google.com/legal/corporate-cla-v1.0.html # # The agreement for individuals can be filled out on the web. # # When adding J Random Contributor's name to this file, # either J's name or J's organization's name should be # added to the AUTHORS file, depending on whether the # individual or corporate CLA was used. # Names should be added to this file like so: # Name # Please keep the list sorted. Damian Gryski Jan Mercl <0xjnml@gmail.com> Kai Backman Marc-Antoine Ruel Nigel Tao Rob Pike Rodolfo Carvalho Russ Cox Sebastien Binet golang-github-golang-snappy-0.0+git20160529.d9eb7a3/LICENSE000066400000000000000000000027161272247357100224470ustar00rootroot00000000000000Copyright (c) 2011 The Snappy-Go Authors. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang-github-golang-snappy-0.0+git20160529.d9eb7a3/README000066400000000000000000000067251272247357100223260ustar00rootroot00000000000000The Snappy compression format in the Go programming language. To download and install from source: $ go get github.com/golang/snappy Unless otherwise noted, the Snappy-Go source files are distributed under the BSD-style license found in the LICENSE file. Benchmarks. The golang/snappy benchmarks include compressing (Z) and decompressing (U) ten or so files, the same set used by the C++ Snappy code (github.com/google/snappy and note the "google", not "golang"). On an "Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz", Go's GOARCH=amd64 numbers as of 2016-05-29: "go test -test.bench=." _UFlat0-8 2.19GB/s ± 0% html _UFlat1-8 1.41GB/s ± 0% urls _UFlat2-8 23.5GB/s ± 2% jpg _UFlat3-8 1.91GB/s ± 0% jpg_200 _UFlat4-8 14.0GB/s ± 1% pdf _UFlat5-8 1.97GB/s ± 0% html4 _UFlat6-8 814MB/s ± 0% txt1 _UFlat7-8 785MB/s ± 0% txt2 _UFlat8-8 857MB/s ± 0% txt3 _UFlat9-8 719MB/s ± 1% txt4 _UFlat10-8 2.84GB/s ± 0% pb _UFlat11-8 1.05GB/s ± 0% gaviota _ZFlat0-8 1.04GB/s ± 0% html _ZFlat1-8 534MB/s ± 0% urls _ZFlat2-8 15.7GB/s ± 1% jpg _ZFlat3-8 740MB/s ± 3% jpg_200 _ZFlat4-8 9.20GB/s ± 1% pdf _ZFlat5-8 991MB/s ± 0% html4 _ZFlat6-8 379MB/s ± 0% txt1 _ZFlat7-8 352MB/s ± 0% txt2 _ZFlat8-8 396MB/s ± 1% txt3 _ZFlat9-8 327MB/s ± 1% txt4 _ZFlat10-8 1.33GB/s ± 1% pb _ZFlat11-8 605MB/s ± 1% gaviota "go test -test.bench=. -tags=noasm" _UFlat0-8 621MB/s ± 2% html _UFlat1-8 494MB/s ± 1% urls _UFlat2-8 23.2GB/s ± 1% jpg _UFlat3-8 1.12GB/s ± 1% jpg_200 _UFlat4-8 4.35GB/s ± 1% pdf _UFlat5-8 609MB/s ± 0% html4 _UFlat6-8 296MB/s ± 0% txt1 _UFlat7-8 288MB/s ± 0% txt2 _UFlat8-8 309MB/s ± 1% txt3 _UFlat9-8 280MB/s ± 1% txt4 _UFlat10-8 753MB/s ± 0% pb _UFlat11-8 400MB/s ± 0% gaviota _ZFlat0-8 409MB/s ± 1% html _ZFlat1-8 250MB/s ± 1% urls _ZFlat2-8 12.3GB/s ± 1% jpg _ZFlat3-8 132MB/s ± 0% jpg_200 _ZFlat4-8 2.92GB/s ± 0% pdf _ZFlat5-8 405MB/s ± 1% html4 _ZFlat6-8 179MB/s ± 1% txt1 _ZFlat7-8 170MB/s ± 1% txt2 _ZFlat8-8 189MB/s ± 1% txt3 _ZFlat9-8 164MB/s ± 1% txt4 _ZFlat10-8 479MB/s ± 1% pb _ZFlat11-8 270MB/s ± 1% gaviota For comparison (Go's encoded output is byte-for-byte identical to C++'s), here are the numbers from C++ Snappy's make CXXFLAGS="-O2 -DNDEBUG -g" clean snappy_unittest.log && cat snappy_unittest.log BM_UFlat/0 2.4GB/s html BM_UFlat/1 1.4GB/s urls BM_UFlat/2 21.8GB/s jpg BM_UFlat/3 1.5GB/s jpg_200 BM_UFlat/4 13.3GB/s pdf BM_UFlat/5 2.1GB/s html4 BM_UFlat/6 1.0GB/s txt1 BM_UFlat/7 959.4MB/s txt2 BM_UFlat/8 1.0GB/s txt3 BM_UFlat/9 864.5MB/s txt4 BM_UFlat/10 2.9GB/s pb BM_UFlat/11 1.2GB/s gaviota BM_ZFlat/0 944.3MB/s html (22.31 %) BM_ZFlat/1 501.6MB/s urls (47.78 %) BM_ZFlat/2 14.3GB/s jpg (99.95 %) BM_ZFlat/3 538.3MB/s jpg_200 (73.00 %) BM_ZFlat/4 8.3GB/s pdf (83.30 %) BM_ZFlat/5 903.5MB/s html4 (22.52 %) BM_ZFlat/6 336.0MB/s txt1 (57.88 %) BM_ZFlat/7 312.3MB/s txt2 (61.91 %) BM_ZFlat/8 353.1MB/s txt3 (54.99 %) BM_ZFlat/9 289.9MB/s txt4 (66.26 %) BM_ZFlat/10 1.2GB/s pb (19.68 %) BM_ZFlat/11 527.4MB/s gaviota (37.72 %) golang-github-golang-snappy-0.0+git20160529.d9eb7a3/cmd/000077500000000000000000000000001272247357100221775ustar00rootroot00000000000000golang-github-golang-snappy-0.0+git20160529.d9eb7a3/cmd/snappytool/000077500000000000000000000000001272247357100244075ustar00rootroot00000000000000golang-github-golang-snappy-0.0+git20160529.d9eb7a3/cmd/snappytool/main.cpp000066400000000000000000000034471272247357100260470ustar00rootroot00000000000000/* To build the snappytool binary: g++ main.cpp /usr/lib/libsnappy.a -o snappytool or, if you have built the C++ snappy library from source: g++ main.cpp /path/to/your/snappy/.libs/libsnappy.a -o snappytool after running "make" from your snappy checkout directory. */ #include #include #include #include #include "snappy.h" #define N 1000000 char dst[N]; char src[N]; int main(int argc, char** argv) { // Parse args. if (argc != 2) { fprintf(stderr, "exactly one of -d or -e must be given\n"); return 1; } bool decode = strcmp(argv[1], "-d") == 0; bool encode = strcmp(argv[1], "-e") == 0; if (decode == encode) { fprintf(stderr, "exactly one of -d or -e must be given\n"); return 1; } // Read all of stdin into src[:s]. size_t s = 0; while (1) { if (s == N) { fprintf(stderr, "input too large\n"); return 1; } ssize_t n = read(0, src+s, N-s); if (n == 0) { break; } if (n < 0) { fprintf(stderr, "read error: %s\n", strerror(errno)); // TODO: handle EAGAIN, EINTR? return 1; } s += n; } // Encode or decode src[:s] to dst[:d], and write to stdout. size_t d = 0; if (encode) { if (N < snappy::MaxCompressedLength(s)) { fprintf(stderr, "input too large after encoding\n"); return 1; } snappy::RawCompress(src, s, dst, &d); } else { if (!snappy::GetUncompressedLength(src, s, &d)) { fprintf(stderr, "could not get uncompressed length\n"); return 1; } if (N < d) { fprintf(stderr, "input too large after decoding\n"); return 1; } if (!snappy::RawUncompress(src, s, dst)) { fprintf(stderr, "input was not valid Snappy-compressed data\n"); return 1; } } write(1, dst, d); return 0; } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/decode.go000066400000000000000000000135751272247357100232210ustar00rootroot00000000000000// Copyright 2011 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package snappy import ( "encoding/binary" "errors" "io" ) var ( // ErrCorrupt reports that the input is invalid. ErrCorrupt = errors.New("snappy: corrupt input") // ErrTooLarge reports that the uncompressed length is too large. ErrTooLarge = errors.New("snappy: decoded block is too large") // ErrUnsupported reports that the input isn't supported. ErrUnsupported = errors.New("snappy: unsupported input") errUnsupportedLiteralLength = errors.New("snappy: unsupported literal length") ) // DecodedLen returns the length of the decoded block. func DecodedLen(src []byte) (int, error) { v, _, err := decodedLen(src) return v, err } // decodedLen returns the length of the decoded block and the number of bytes // that the length header occupied. func decodedLen(src []byte) (blockLen, headerLen int, err error) { v, n := binary.Uvarint(src) if n <= 0 || v > 0xffffffff { return 0, 0, ErrCorrupt } const wordSize = 32 << (^uint(0) >> 32 & 1) if wordSize == 32 && v > 0x7fffffff { return 0, 0, ErrTooLarge } return int(v), n, nil } const ( decodeErrCodeCorrupt = 1 decodeErrCodeUnsupportedLiteralLength = 2 ) // Decode returns the decoded form of src. The returned slice may be a sub- // slice of dst if dst was large enough to hold the entire decoded block. // Otherwise, a newly allocated slice will be returned. // // The dst and src must not overlap. It is valid to pass a nil dst. func Decode(dst, src []byte) ([]byte, error) { dLen, s, err := decodedLen(src) if err != nil { return nil, err } if dLen <= len(dst) { dst = dst[:dLen] } else { dst = make([]byte, dLen) } switch decode(dst, src[s:]) { case 0: return dst, nil case decodeErrCodeUnsupportedLiteralLength: return nil, errUnsupportedLiteralLength } return nil, ErrCorrupt } // NewReader returns a new Reader that decompresses from r, using the framing // format described at // https://github.com/google/snappy/blob/master/framing_format.txt func NewReader(r io.Reader) *Reader { return &Reader{ r: r, decoded: make([]byte, maxBlockSize), buf: make([]byte, maxEncodedLenOfMaxBlockSize+checksumSize), } } // Reader is an io.Reader that can read Snappy-compressed bytes. type Reader struct { r io.Reader err error decoded []byte buf []byte // decoded[i:j] contains decoded bytes that have not yet been passed on. i, j int readHeader bool } // Reset discards any buffered data, resets all state, and switches the Snappy // reader to read from r. This permits reusing a Reader rather than allocating // a new one. func (r *Reader) Reset(reader io.Reader) { r.r = reader r.err = nil r.i = 0 r.j = 0 r.readHeader = false } func (r *Reader) readFull(p []byte, allowEOF bool) (ok bool) { if _, r.err = io.ReadFull(r.r, p); r.err != nil { if r.err == io.ErrUnexpectedEOF || (r.err == io.EOF && !allowEOF) { r.err = ErrCorrupt } return false } return true } // Read satisfies the io.Reader interface. func (r *Reader) Read(p []byte) (int, error) { if r.err != nil { return 0, r.err } for { if r.i < r.j { n := copy(p, r.decoded[r.i:r.j]) r.i += n return n, nil } if !r.readFull(r.buf[:4], true) { return 0, r.err } chunkType := r.buf[0] if !r.readHeader { if chunkType != chunkTypeStreamIdentifier { r.err = ErrCorrupt return 0, r.err } r.readHeader = true } chunkLen := int(r.buf[1]) | int(r.buf[2])<<8 | int(r.buf[3])<<16 if chunkLen > len(r.buf) { r.err = ErrUnsupported return 0, r.err } // The chunk types are specified at // https://github.com/google/snappy/blob/master/framing_format.txt switch chunkType { case chunkTypeCompressedData: // Section 4.2. Compressed data (chunk type 0x00). if chunkLen < checksumSize { r.err = ErrCorrupt return 0, r.err } buf := r.buf[:chunkLen] if !r.readFull(buf, false) { return 0, r.err } checksum := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2])<<16 | uint32(buf[3])<<24 buf = buf[checksumSize:] n, err := DecodedLen(buf) if err != nil { r.err = err return 0, r.err } if n > len(r.decoded) { r.err = ErrCorrupt return 0, r.err } if _, err := Decode(r.decoded, buf); err != nil { r.err = err return 0, r.err } if crc(r.decoded[:n]) != checksum { r.err = ErrCorrupt return 0, r.err } r.i, r.j = 0, n continue case chunkTypeUncompressedData: // Section 4.3. Uncompressed data (chunk type 0x01). if chunkLen < checksumSize { r.err = ErrCorrupt return 0, r.err } buf := r.buf[:checksumSize] if !r.readFull(buf, false) { return 0, r.err } checksum := uint32(buf[0]) | uint32(buf[1])<<8 | uint32(buf[2])<<16 | uint32(buf[3])<<24 // Read directly into r.decoded instead of via r.buf. n := chunkLen - checksumSize if n > len(r.decoded) { r.err = ErrCorrupt return 0, r.err } if !r.readFull(r.decoded[:n], false) { return 0, r.err } if crc(r.decoded[:n]) != checksum { r.err = ErrCorrupt return 0, r.err } r.i, r.j = 0, n continue case chunkTypeStreamIdentifier: // Section 4.1. Stream identifier (chunk type 0xff). if chunkLen != len(magicBody) { r.err = ErrCorrupt return 0, r.err } if !r.readFull(r.buf[:len(magicBody)], false) { return 0, r.err } for i := 0; i < len(magicBody); i++ { if r.buf[i] != magicBody[i] { r.err = ErrCorrupt return 0, r.err } } continue } if chunkType <= 0x7f { // Section 4.5. Reserved unskippable chunks (chunk types 0x02-0x7f). r.err = ErrUnsupported return 0, r.err } // Section 4.4 Padding (chunk type 0xfe). // Section 4.6. Reserved skippable chunks (chunk types 0x80-0xfd). if !r.readFull(r.buf[:chunkLen], false) { return 0, r.err } } } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/decode_amd64.go000066400000000000000000000005251272247357100242030ustar00rootroot00000000000000// Copyright 2016 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build !appengine // +build gc // +build !noasm package snappy // decode has the same semantics as in decode_other.go. // //go:noescape func decode(dst, src []byte) int golang-github-golang-snappy-0.0+git20160529.d9eb7a3/decode_amd64.s000066400000000000000000000274661272247357100240550ustar00rootroot00000000000000// Copyright 2016 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build !appengine // +build gc // +build !noasm #include "textflag.h" // The asm code generally follows the pure Go code in decode_other.go, except // where marked with a "!!!". // func decode(dst, src []byte) int // // All local variables fit into registers. The non-zero stack size is only to // spill registers and push args when issuing a CALL. The register allocation: // - AX scratch // - BX scratch // - CX length or x // - DX offset // - SI &src[s] // - DI &dst[d] // + R8 dst_base // + R9 dst_len // + R10 dst_base + dst_len // + R11 src_base // + R12 src_len // + R13 src_base + src_len // - R14 used by doCopy // - R15 used by doCopy // // The registers R8-R13 (marked with a "+") are set at the start of the // function, and after a CALL returns, and are not otherwise modified. // // The d variable is implicitly DI - R8, and len(dst)-d is R10 - DI. // The s variable is implicitly SI - R11, and len(src)-s is R13 - SI. TEXT ·decode(SB), NOSPLIT, $48-56 // Initialize SI, DI and R8-R13. MOVQ dst_base+0(FP), R8 MOVQ dst_len+8(FP), R9 MOVQ R8, DI MOVQ R8, R10 ADDQ R9, R10 MOVQ src_base+24(FP), R11 MOVQ src_len+32(FP), R12 MOVQ R11, SI MOVQ R11, R13 ADDQ R12, R13 loop: // for s < len(src) CMPQ SI, R13 JEQ end // CX = uint32(src[s]) // // switch src[s] & 0x03 MOVBLZX (SI), CX MOVL CX, BX ANDL $3, BX CMPL BX, $1 JAE tagCopy // ---------------------------------------- // The code below handles literal tags. // case tagLiteral: // x := uint32(src[s] >> 2) // switch SHRL $2, CX CMPL CX, $60 JAE tagLit60Plus // case x < 60: // s++ INCQ SI doLit: // This is the end of the inner "switch", when we have a literal tag. // // We assume that CX == x and x fits in a uint32, where x is the variable // used in the pure Go decode_other.go code. // length = int(x) + 1 // // Unlike the pure Go code, we don't need to check if length <= 0 because // CX can hold 64 bits, so the increment cannot overflow. INCQ CX // Prepare to check if copying length bytes will run past the end of dst or // src. // // AX = len(dst) - d // BX = len(src) - s MOVQ R10, AX SUBQ DI, AX MOVQ R13, BX SUBQ SI, BX // !!! Try a faster technique for short (16 or fewer bytes) copies. // // if length > 16 || len(dst)-d < 16 || len(src)-s < 16 { // goto callMemmove // Fall back on calling runtime·memmove. // } // // The C++ snappy code calls this TryFastAppend. It also checks len(src)-s // against 21 instead of 16, because it cannot assume that all of its input // is contiguous in memory and so it needs to leave enough source bytes to // read the next tag without refilling buffers, but Go's Decode assumes // contiguousness (the src argument is a []byte). CMPQ CX, $16 JGT callMemmove CMPQ AX, $16 JLT callMemmove CMPQ BX, $16 JLT callMemmove // !!! Implement the copy from src to dst as a 16-byte load and store. // (Decode's documentation says that dst and src must not overlap.) // // This always copies 16 bytes, instead of only length bytes, but that's // OK. If the input is a valid Snappy encoding then subsequent iterations // will fix up the overrun. Otherwise, Decode returns a nil []byte (and a // non-nil error), so the overrun will be ignored. // // Note that on amd64, it is legal and cheap to issue unaligned 8-byte or // 16-byte loads and stores. This technique probably wouldn't be as // effective on architectures that are fussier about alignment. MOVOU 0(SI), X0 MOVOU X0, 0(DI) // d += length // s += length ADDQ CX, DI ADDQ CX, SI JMP loop callMemmove: // if length > len(dst)-d || length > len(src)-s { etc } CMPQ CX, AX JGT errCorrupt CMPQ CX, BX JGT errCorrupt // copy(dst[d:], src[s:s+length]) // // This means calling runtime·memmove(&dst[d], &src[s], length), so we push // DI, SI and CX as arguments. Coincidentally, we also need to spill those // three registers to the stack, to save local variables across the CALL. MOVQ DI, 0(SP) MOVQ SI, 8(SP) MOVQ CX, 16(SP) MOVQ DI, 24(SP) MOVQ SI, 32(SP) MOVQ CX, 40(SP) CALL runtime·memmove(SB) // Restore local variables: unspill registers from the stack and // re-calculate R8-R13. MOVQ 24(SP), DI MOVQ 32(SP), SI MOVQ 40(SP), CX MOVQ dst_base+0(FP), R8 MOVQ dst_len+8(FP), R9 MOVQ R8, R10 ADDQ R9, R10 MOVQ src_base+24(FP), R11 MOVQ src_len+32(FP), R12 MOVQ R11, R13 ADDQ R12, R13 // d += length // s += length ADDQ CX, DI ADDQ CX, SI JMP loop tagLit60Plus: // !!! This fragment does the // // s += x - 58; if uint(s) > uint(len(src)) { etc } // // checks. In the asm version, we code it once instead of once per switch case. ADDQ CX, SI SUBQ $58, SI MOVQ SI, BX SUBQ R11, BX CMPQ BX, R12 JA errCorrupt // case x == 60: CMPL CX, $61 JEQ tagLit61 JA tagLit62Plus // x = uint32(src[s-1]) MOVBLZX -1(SI), CX JMP doLit tagLit61: // case x == 61: // x = uint32(src[s-2]) | uint32(src[s-1])<<8 MOVWLZX -2(SI), CX JMP doLit tagLit62Plus: CMPL CX, $62 JA tagLit63 // case x == 62: // x = uint32(src[s-3]) | uint32(src[s-2])<<8 | uint32(src[s-1])<<16 MOVWLZX -3(SI), CX MOVBLZX -1(SI), BX SHLL $16, BX ORL BX, CX JMP doLit tagLit63: // case x == 63: // x = uint32(src[s-4]) | uint32(src[s-3])<<8 | uint32(src[s-2])<<16 | uint32(src[s-1])<<24 MOVL -4(SI), CX JMP doLit // The code above handles literal tags. // ---------------------------------------- // The code below handles copy tags. tagCopy4: // case tagCopy4: // s += 5 ADDQ $5, SI // if uint(s) > uint(len(src)) { etc } MOVQ SI, BX SUBQ R11, BX CMPQ BX, R12 JA errCorrupt // length = 1 + int(src[s-5])>>2 SHRQ $2, CX INCQ CX // offset = int(uint32(src[s-4]) | uint32(src[s-3])<<8 | uint32(src[s-2])<<16 | uint32(src[s-1])<<24) MOVLQZX -4(SI), DX JMP doCopy tagCopy2: // case tagCopy2: // s += 3 ADDQ $3, SI // if uint(s) > uint(len(src)) { etc } MOVQ SI, BX SUBQ R11, BX CMPQ BX, R12 JA errCorrupt // length = 1 + int(src[s-3])>>2 SHRQ $2, CX INCQ CX // offset = int(uint32(src[s-2]) | uint32(src[s-1])<<8) MOVWQZX -2(SI), DX JMP doCopy tagCopy: // We have a copy tag. We assume that: // - BX == src[s] & 0x03 // - CX == src[s] CMPQ BX, $2 JEQ tagCopy2 JA tagCopy4 // case tagCopy1: // s += 2 ADDQ $2, SI // if uint(s) > uint(len(src)) { etc } MOVQ SI, BX SUBQ R11, BX CMPQ BX, R12 JA errCorrupt // offset = int(uint32(src[s-2])&0xe0<<3 | uint32(src[s-1])) MOVQ CX, DX ANDQ $0xe0, DX SHLQ $3, DX MOVBQZX -1(SI), BX ORQ BX, DX // length = 4 + int(src[s-2])>>2&0x7 SHRQ $2, CX ANDQ $7, CX ADDQ $4, CX doCopy: // This is the end of the outer "switch", when we have a copy tag. // // We assume that: // - CX == length && CX > 0 // - DX == offset // if offset <= 0 { etc } CMPQ DX, $0 JLE errCorrupt // if d < offset { etc } MOVQ DI, BX SUBQ R8, BX CMPQ BX, DX JLT errCorrupt // if length > len(dst)-d { etc } MOVQ R10, BX SUBQ DI, BX CMPQ CX, BX JGT errCorrupt // forwardCopy(dst[d:d+length], dst[d-offset:]); d += length // // Set: // - R14 = len(dst)-d // - R15 = &dst[d-offset] MOVQ R10, R14 SUBQ DI, R14 MOVQ DI, R15 SUBQ DX, R15 // !!! Try a faster technique for short (16 or fewer bytes) forward copies. // // First, try using two 8-byte load/stores, similar to the doLit technique // above. Even if dst[d:d+length] and dst[d-offset:] can overlap, this is // still OK if offset >= 8. Note that this has to be two 8-byte load/stores // and not one 16-byte load/store, and the first store has to be before the // second load, due to the overlap if offset is in the range [8, 16). // // if length > 16 || offset < 8 || len(dst)-d < 16 { // goto slowForwardCopy // } // copy 16 bytes // d += length CMPQ CX, $16 JGT slowForwardCopy CMPQ DX, $8 JLT slowForwardCopy CMPQ R14, $16 JLT slowForwardCopy MOVQ 0(R15), AX MOVQ AX, 0(DI) MOVQ 8(R15), BX MOVQ BX, 8(DI) ADDQ CX, DI JMP loop slowForwardCopy: // !!! If the forward copy is longer than 16 bytes, or if offset < 8, we // can still try 8-byte load stores, provided we can overrun up to 10 extra // bytes. As above, the overrun will be fixed up by subsequent iterations // of the outermost loop. // // The C++ snappy code calls this technique IncrementalCopyFastPath. Its // commentary says: // // ---- // // The main part of this loop is a simple copy of eight bytes at a time // until we've copied (at least) the requested amount of bytes. However, // if d and d-offset are less than eight bytes apart (indicating a // repeating pattern of length < 8), we first need to expand the pattern in // order to get the correct results. For instance, if the buffer looks like // this, with the eight-byte and patterns marked as // intervals: // // abxxxxxxxxxxxx // [------] d-offset // [------] d // // a single eight-byte copy from to will repeat the pattern // once, after which we can move two bytes without moving : // // ababxxxxxxxxxx // [------] d-offset // [------] d // // and repeat the exercise until the two no longer overlap. // // This allows us to do very well in the special case of one single byte // repeated many times, without taking a big hit for more general cases. // // The worst case of extra writing past the end of the match occurs when // offset == 1 and length == 1; the last copy will read from byte positions // [0..7] and write to [4..11], whereas it was only supposed to write to // position 1. Thus, ten excess bytes. // // ---- // // That "10 byte overrun" worst case is confirmed by Go's // TestSlowForwardCopyOverrun, which also tests the fixUpSlowForwardCopy // and finishSlowForwardCopy algorithm. // // if length > len(dst)-d-10 { // goto verySlowForwardCopy // } SUBQ $10, R14 CMPQ CX, R14 JGT verySlowForwardCopy makeOffsetAtLeast8: // !!! As above, expand the pattern so that offset >= 8 and we can use // 8-byte load/stores. // // for offset < 8 { // copy 8 bytes from dst[d-offset:] to dst[d:] // length -= offset // d += offset // offset += offset // // The two previous lines together means that d-offset, and therefore // // R15, is unchanged. // } CMPQ DX, $8 JGE fixUpSlowForwardCopy MOVQ (R15), BX MOVQ BX, (DI) SUBQ DX, CX ADDQ DX, DI ADDQ DX, DX JMP makeOffsetAtLeast8 fixUpSlowForwardCopy: // !!! Add length (which might be negative now) to d (implied by DI being // &dst[d]) so that d ends up at the right place when we jump back to the // top of the loop. Before we do that, though, we save DI to AX so that, if // length is positive, copying the remaining length bytes will write to the // right place. MOVQ DI, AX ADDQ CX, DI finishSlowForwardCopy: // !!! Repeat 8-byte load/stores until length <= 0. Ending with a negative // length means that we overrun, but as above, that will be fixed up by // subsequent iterations of the outermost loop. CMPQ CX, $0 JLE loop MOVQ (R15), BX MOVQ BX, (AX) ADDQ $8, R15 ADDQ $8, AX SUBQ $8, CX JMP finishSlowForwardCopy verySlowForwardCopy: // verySlowForwardCopy is a simple implementation of forward copy. In C // parlance, this is a do/while loop instead of a while loop, since we know // that length > 0. In Go syntax: // // for { // dst[d] = dst[d - offset] // d++ // length-- // if length == 0 { // break // } // } MOVB (R15), BX MOVB BX, (DI) INCQ R15 INCQ DI DECQ CX JNZ verySlowForwardCopy JMP loop // The code above handles copy tags. // ---------------------------------------- end: // This is the end of the "for s < len(src)". // // if d != len(dst) { etc } CMPQ DI, R10 JNE errCorrupt // return 0 MOVQ $0, ret+48(FP) RET errCorrupt: // return decodeErrCodeCorrupt MOVQ $1, ret+48(FP) RET golang-github-golang-snappy-0.0+git20160529.d9eb7a3/decode_other.go000066400000000000000000000060741272247357100244160ustar00rootroot00000000000000// Copyright 2016 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build !amd64 appengine !gc noasm package snappy // decode writes the decoding of src to dst. It assumes that the varint-encoded // length of the decompressed bytes has already been read, and that len(dst) // equals that length. // // It returns 0 on success or a decodeErrCodeXxx error code on failure. func decode(dst, src []byte) int { var d, s, offset, length int for s < len(src) { switch src[s] & 0x03 { case tagLiteral: x := uint32(src[s] >> 2) switch { case x < 60: s++ case x == 60: s += 2 if uint(s) > uint(len(src)) { // The uint conversions catch overflow from the previous line. return decodeErrCodeCorrupt } x = uint32(src[s-1]) case x == 61: s += 3 if uint(s) > uint(len(src)) { // The uint conversions catch overflow from the previous line. return decodeErrCodeCorrupt } x = uint32(src[s-2]) | uint32(src[s-1])<<8 case x == 62: s += 4 if uint(s) > uint(len(src)) { // The uint conversions catch overflow from the previous line. return decodeErrCodeCorrupt } x = uint32(src[s-3]) | uint32(src[s-2])<<8 | uint32(src[s-1])<<16 case x == 63: s += 5 if uint(s) > uint(len(src)) { // The uint conversions catch overflow from the previous line. return decodeErrCodeCorrupt } x = uint32(src[s-4]) | uint32(src[s-3])<<8 | uint32(src[s-2])<<16 | uint32(src[s-1])<<24 } length = int(x) + 1 if length <= 0 { return decodeErrCodeUnsupportedLiteralLength } if length > len(dst)-d || length > len(src)-s { return decodeErrCodeCorrupt } copy(dst[d:], src[s:s+length]) d += length s += length continue case tagCopy1: s += 2 if uint(s) > uint(len(src)) { // The uint conversions catch overflow from the previous line. return decodeErrCodeCorrupt } length = 4 + int(src[s-2])>>2&0x7 offset = int(uint32(src[s-2])&0xe0<<3 | uint32(src[s-1])) case tagCopy2: s += 3 if uint(s) > uint(len(src)) { // The uint conversions catch overflow from the previous line. return decodeErrCodeCorrupt } length = 1 + int(src[s-3])>>2 offset = int(uint32(src[s-2]) | uint32(src[s-1])<<8) case tagCopy4: s += 5 if uint(s) > uint(len(src)) { // The uint conversions catch overflow from the previous line. return decodeErrCodeCorrupt } length = 1 + int(src[s-5])>>2 offset = int(uint32(src[s-4]) | uint32(src[s-3])<<8 | uint32(src[s-2])<<16 | uint32(src[s-1])<<24) } if offset <= 0 || d < offset || length > len(dst)-d { return decodeErrCodeCorrupt } // Copy from an earlier sub-slice of dst to a later sub-slice. Unlike // the built-in copy function, this byte-by-byte copy always runs // forwards, even if the slices overlap. Conceptually, this is: // // d += forwardCopy(dst[d:d+length], dst[d-offset:]) for end := d + length; d != end; d++ { dst[d] = dst[d-offset] } } if d != len(dst) { return decodeErrCodeCorrupt } return 0 } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/encode.go000066400000000000000000000217001272247357100232200ustar00rootroot00000000000000// Copyright 2011 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package snappy import ( "encoding/binary" "errors" "io" ) // Encode returns the encoded form of src. The returned slice may be a sub- // slice of dst if dst was large enough to hold the entire encoded block. // Otherwise, a newly allocated slice will be returned. // // The dst and src must not overlap. It is valid to pass a nil dst. func Encode(dst, src []byte) []byte { if n := MaxEncodedLen(len(src)); n < 0 { panic(ErrTooLarge) } else if len(dst) < n { dst = make([]byte, n) } // The block starts with the varint-encoded length of the decompressed bytes. d := binary.PutUvarint(dst, uint64(len(src))) for len(src) > 0 { p := src src = nil if len(p) > maxBlockSize { p, src = p[:maxBlockSize], p[maxBlockSize:] } if len(p) < minNonLiteralBlockSize { d += emitLiteral(dst[d:], p) } else { d += encodeBlock(dst[d:], p) } } return dst[:d] } // inputMargin is the minimum number of extra input bytes to keep, inside // encodeBlock's inner loop. On some architectures, this margin lets us // implement a fast path for emitLiteral, where the copy of short (<= 16 byte) // literals can be implemented as a single load to and store from a 16-byte // register. That literal's actual length can be as short as 1 byte, so this // can copy up to 15 bytes too much, but that's OK as subsequent iterations of // the encoding loop will fix up the copy overrun, and this inputMargin ensures // that we don't overrun the dst and src buffers. const inputMargin = 16 - 1 // minNonLiteralBlockSize is the minimum size of the input to encodeBlock that // could be encoded with a copy tag. This is the minimum with respect to the // algorithm used by encodeBlock, not a minimum enforced by the file format. // // The encoded output must start with at least a 1 byte literal, as there are // no previous bytes to copy. A minimal (1 byte) copy after that, generated // from an emitCopy call in encodeBlock's main loop, would require at least // another inputMargin bytes, for the reason above: we want any emitLiteral // calls inside encodeBlock's main loop to use the fast path if possible, which // requires being able to overrun by inputMargin bytes. Thus, // minNonLiteralBlockSize equals 1 + 1 + inputMargin. // // The C++ code doesn't use this exact threshold, but it could, as discussed at // https://groups.google.com/d/topic/snappy-compression/oGbhsdIJSJ8/discussion // The difference between Go (2+inputMargin) and C++ (inputMargin) is purely an // optimization. It should not affect the encoded form. This is tested by // TestSameEncodingAsCppShortCopies. const minNonLiteralBlockSize = 1 + 1 + inputMargin // MaxEncodedLen returns the maximum length of a snappy block, given its // uncompressed length. // // It will return a negative value if srcLen is too large to encode. func MaxEncodedLen(srcLen int) int { n := uint64(srcLen) if n > 0xffffffff { return -1 } // Compressed data can be defined as: // compressed := item* literal* // item := literal* copy // // The trailing literal sequence has a space blowup of at most 62/60 // since a literal of length 60 needs one tag byte + one extra byte // for length information. // // Item blowup is trickier to measure. Suppose the "copy" op copies // 4 bytes of data. Because of a special check in the encoding code, // we produce a 4-byte copy only if the offset is < 65536. Therefore // the copy op takes 3 bytes to encode, and this type of item leads // to at most the 62/60 blowup for representing literals. // // Suppose the "copy" op copies 5 bytes of data. If the offset is big // enough, it will take 5 bytes to encode the copy op. Therefore the // worst case here is a one-byte literal followed by a five-byte copy. // That is, 6 bytes of input turn into 7 bytes of "compressed" data. // // This last factor dominates the blowup, so the final estimate is: n = 32 + n + n/6 if n > 0xffffffff { return -1 } return int(n) } var errClosed = errors.New("snappy: Writer is closed") // NewWriter returns a new Writer that compresses to w. // // The Writer returned does not buffer writes. There is no need to Flush or // Close such a Writer. // // Deprecated: the Writer returned is not suitable for many small writes, only // for few large writes. Use NewBufferedWriter instead, which is efficient // regardless of the frequency and shape of the writes, and remember to Close // that Writer when done. func NewWriter(w io.Writer) *Writer { return &Writer{ w: w, obuf: make([]byte, obufLen), } } // NewBufferedWriter returns a new Writer that compresses to w, using the // framing format described at // https://github.com/google/snappy/blob/master/framing_format.txt // // The Writer returned buffers writes. Users must call Close to guarantee all // data has been forwarded to the underlying io.Writer. They may also call // Flush zero or more times before calling Close. func NewBufferedWriter(w io.Writer) *Writer { return &Writer{ w: w, ibuf: make([]byte, 0, maxBlockSize), obuf: make([]byte, obufLen), } } // Writer is an io.Writer than can write Snappy-compressed bytes. type Writer struct { w io.Writer err error // ibuf is a buffer for the incoming (uncompressed) bytes. // // Its use is optional. For backwards compatibility, Writers created by the // NewWriter function have ibuf == nil, do not buffer incoming bytes, and // therefore do not need to be Flush'ed or Close'd. ibuf []byte // obuf is a buffer for the outgoing (compressed) bytes. obuf []byte // wroteStreamHeader is whether we have written the stream header. wroteStreamHeader bool } // Reset discards the writer's state and switches the Snappy writer to write to // w. This permits reusing a Writer rather than allocating a new one. func (w *Writer) Reset(writer io.Writer) { w.w = writer w.err = nil if w.ibuf != nil { w.ibuf = w.ibuf[:0] } w.wroteStreamHeader = false } // Write satisfies the io.Writer interface. func (w *Writer) Write(p []byte) (nRet int, errRet error) { if w.ibuf == nil { // Do not buffer incoming bytes. This does not perform or compress well // if the caller of Writer.Write writes many small slices. This // behavior is therefore deprecated, but still supported for backwards // compatibility with code that doesn't explicitly Flush or Close. return w.write(p) } // The remainder of this method is based on bufio.Writer.Write from the // standard library. for len(p) > (cap(w.ibuf)-len(w.ibuf)) && w.err == nil { var n int if len(w.ibuf) == 0 { // Large write, empty buffer. // Write directly from p to avoid copy. n, _ = w.write(p) } else { n = copy(w.ibuf[len(w.ibuf):cap(w.ibuf)], p) w.ibuf = w.ibuf[:len(w.ibuf)+n] w.Flush() } nRet += n p = p[n:] } if w.err != nil { return nRet, w.err } n := copy(w.ibuf[len(w.ibuf):cap(w.ibuf)], p) w.ibuf = w.ibuf[:len(w.ibuf)+n] nRet += n return nRet, nil } func (w *Writer) write(p []byte) (nRet int, errRet error) { if w.err != nil { return 0, w.err } for len(p) > 0 { obufStart := len(magicChunk) if !w.wroteStreamHeader { w.wroteStreamHeader = true copy(w.obuf, magicChunk) obufStart = 0 } var uncompressed []byte if len(p) > maxBlockSize { uncompressed, p = p[:maxBlockSize], p[maxBlockSize:] } else { uncompressed, p = p, nil } checksum := crc(uncompressed) // Compress the buffer, discarding the result if the improvement // isn't at least 12.5%. compressed := Encode(w.obuf[obufHeaderLen:], uncompressed) chunkType := uint8(chunkTypeCompressedData) chunkLen := 4 + len(compressed) obufEnd := obufHeaderLen + len(compressed) if len(compressed) >= len(uncompressed)-len(uncompressed)/8 { chunkType = chunkTypeUncompressedData chunkLen = 4 + len(uncompressed) obufEnd = obufHeaderLen } // Fill in the per-chunk header that comes before the body. w.obuf[len(magicChunk)+0] = chunkType w.obuf[len(magicChunk)+1] = uint8(chunkLen >> 0) w.obuf[len(magicChunk)+2] = uint8(chunkLen >> 8) w.obuf[len(magicChunk)+3] = uint8(chunkLen >> 16) w.obuf[len(magicChunk)+4] = uint8(checksum >> 0) w.obuf[len(magicChunk)+5] = uint8(checksum >> 8) w.obuf[len(magicChunk)+6] = uint8(checksum >> 16) w.obuf[len(magicChunk)+7] = uint8(checksum >> 24) if _, err := w.w.Write(w.obuf[obufStart:obufEnd]); err != nil { w.err = err return nRet, err } if chunkType == chunkTypeUncompressedData { if _, err := w.w.Write(uncompressed); err != nil { w.err = err return nRet, err } } nRet += len(uncompressed) } return nRet, nil } // Flush flushes the Writer to its underlying io.Writer. func (w *Writer) Flush() error { if w.err != nil { return w.err } if len(w.ibuf) == 0 { return nil } w.write(w.ibuf) w.ibuf = w.ibuf[:0] return w.err } // Close calls Flush and then closes the Writer. func (w *Writer) Close() error { w.Flush() ret := w.err if w.err == nil { w.err = errClosed } return ret } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/encode_amd64.go000066400000000000000000000013171272247357100242150ustar00rootroot00000000000000// Copyright 2016 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build !appengine // +build gc // +build !noasm package snappy // emitLiteral has the same semantics as in encode_other.go. // //go:noescape func emitLiteral(dst, lit []byte) int // emitCopy has the same semantics as in encode_other.go. // //go:noescape func emitCopy(dst []byte, offset, length int) int // extendMatch has the same semantics as in encode_other.go. // //go:noescape func extendMatch(src []byte, i, j int) int // encodeBlock has the same semantics as in encode_other.go. // //go:noescape func encodeBlock(dst, src []byte) (d int)golang-github-golang-snappy-0.0+git20160529.d9eb7a3/encode_amd64.s000066400000000000000000000376461272247357100240700ustar00rootroot00000000000000// Copyright 2016 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build !appengine // +build gc // +build !noasm #include "textflag.h" // The XXX lines assemble on Go 1.4, 1.5 and 1.7, but not 1.6, due to a // Go toolchain regression. See https://github.com/golang/go/issues/15426 and // https://github.com/golang/snappy/issues/29 // // As a workaround, the package was built with a known good assembler, and // those instructions were disassembled by "objdump -d" to yield the // 4e 0f b7 7c 5c 78 movzwq 0x78(%rsp,%r11,2),%r15 // style comments, in AT&T asm syntax. Note that rsp here is a physical // register, not Go/asm's SP pseudo-register (see https://golang.org/doc/asm). // The instructions were then encoded as "BYTE $0x.." sequences, which assemble // fine on Go 1.6. // The asm code generally follows the pure Go code in encode_other.go, except // where marked with a "!!!". // ---------------------------------------------------------------------------- // func emitLiteral(dst, lit []byte) int // // All local variables fit into registers. The register allocation: // - AX len(lit) // - BX n // - DX return value // - DI &dst[i] // - R10 &lit[0] // // The 24 bytes of stack space is to call runtime·memmove. // // The unusual register allocation of local variables, such as R10 for the // source pointer, matches the allocation used at the call site in encodeBlock, // which makes it easier to manually inline this function. TEXT ·emitLiteral(SB), NOSPLIT, $24-56 MOVQ dst_base+0(FP), DI MOVQ lit_base+24(FP), R10 MOVQ lit_len+32(FP), AX MOVQ AX, DX MOVL AX, BX SUBL $1, BX CMPL BX, $60 JLT oneByte CMPL BX, $256 JLT twoBytes threeBytes: MOVB $0xf4, 0(DI) MOVW BX, 1(DI) ADDQ $3, DI ADDQ $3, DX JMP memmove twoBytes: MOVB $0xf0, 0(DI) MOVB BX, 1(DI) ADDQ $2, DI ADDQ $2, DX JMP memmove oneByte: SHLB $2, BX MOVB BX, 0(DI) ADDQ $1, DI ADDQ $1, DX memmove: MOVQ DX, ret+48(FP) // copy(dst[i:], lit) // // This means calling runtime·memmove(&dst[i], &lit[0], len(lit)), so we push // DI, R10 and AX as arguments. MOVQ DI, 0(SP) MOVQ R10, 8(SP) MOVQ AX, 16(SP) CALL runtime·memmove(SB) RET // ---------------------------------------------------------------------------- // func emitCopy(dst []byte, offset, length int) int // // All local variables fit into registers. The register allocation: // - AX length // - SI &dst[0] // - DI &dst[i] // - R11 offset // // The unusual register allocation of local variables, such as R11 for the // offset, matches the allocation used at the call site in encodeBlock, which // makes it easier to manually inline this function. TEXT ·emitCopy(SB), NOSPLIT, $0-48 MOVQ dst_base+0(FP), DI MOVQ DI, SI MOVQ offset+24(FP), R11 MOVQ length+32(FP), AX loop0: // for length >= 68 { etc } CMPL AX, $68 JLT step1 // Emit a length 64 copy, encoded as 3 bytes. MOVB $0xfe, 0(DI) MOVW R11, 1(DI) ADDQ $3, DI SUBL $64, AX JMP loop0 step1: // if length > 64 { etc } CMPL AX, $64 JLE step2 // Emit a length 60 copy, encoded as 3 bytes. MOVB $0xee, 0(DI) MOVW R11, 1(DI) ADDQ $3, DI SUBL $60, AX step2: // if length >= 12 || offset >= 2048 { goto step3 } CMPL AX, $12 JGE step3 CMPL R11, $2048 JGE step3 // Emit the remaining copy, encoded as 2 bytes. MOVB R11, 1(DI) SHRL $8, R11 SHLB $5, R11 SUBB $4, AX SHLB $2, AX ORB AX, R11 ORB $1, R11 MOVB R11, 0(DI) ADDQ $2, DI // Return the number of bytes written. SUBQ SI, DI MOVQ DI, ret+40(FP) RET step3: // Emit the remaining copy, encoded as 3 bytes. SUBL $1, AX SHLB $2, AX ORB $2, AX MOVB AX, 0(DI) MOVW R11, 1(DI) ADDQ $3, DI // Return the number of bytes written. SUBQ SI, DI MOVQ DI, ret+40(FP) RET // ---------------------------------------------------------------------------- // func extendMatch(src []byte, i, j int) int // // All local variables fit into registers. The register allocation: // - DX &src[0] // - SI &src[j] // - R13 &src[len(src) - 8] // - R14 &src[len(src)] // - R15 &src[i] // // The unusual register allocation of local variables, such as R15 for a source // pointer, matches the allocation used at the call site in encodeBlock, which // makes it easier to manually inline this function. TEXT ·extendMatch(SB), NOSPLIT, $0-48 MOVQ src_base+0(FP), DX MOVQ src_len+8(FP), R14 MOVQ i+24(FP), R15 MOVQ j+32(FP), SI ADDQ DX, R14 ADDQ DX, R15 ADDQ DX, SI MOVQ R14, R13 SUBQ $8, R13 cmp8: // As long as we are 8 or more bytes before the end of src, we can load and // compare 8 bytes at a time. If those 8 bytes are equal, repeat. CMPQ SI, R13 JA cmp1 MOVQ (R15), AX MOVQ (SI), BX CMPQ AX, BX JNE bsf ADDQ $8, R15 ADDQ $8, SI JMP cmp8 bsf: // If those 8 bytes were not equal, XOR the two 8 byte values, and return // the index of the first byte that differs. The BSF instruction finds the // least significant 1 bit, the amd64 architecture is little-endian, and // the shift by 3 converts a bit index to a byte index. XORQ AX, BX BSFQ BX, BX SHRQ $3, BX ADDQ BX, SI // Convert from &src[ret] to ret. SUBQ DX, SI MOVQ SI, ret+40(FP) RET cmp1: // In src's tail, compare 1 byte at a time. CMPQ SI, R14 JAE extendMatchEnd MOVB (R15), AX MOVB (SI), BX CMPB AX, BX JNE extendMatchEnd ADDQ $1, R15 ADDQ $1, SI JMP cmp1 extendMatchEnd: // Convert from &src[ret] to ret. SUBQ DX, SI MOVQ SI, ret+40(FP) RET // ---------------------------------------------------------------------------- // func encodeBlock(dst, src []byte) (d int) // // All local variables fit into registers, other than "var table". The register // allocation: // - AX . . // - BX . . // - CX 56 shift (note that amd64 shifts by non-immediates must use CX). // - DX 64 &src[0], tableSize // - SI 72 &src[s] // - DI 80 &dst[d] // - R9 88 sLimit // - R10 . &src[nextEmit] // - R11 96 prevHash, currHash, nextHash, offset // - R12 104 &src[base], skip // - R13 . &src[nextS], &src[len(src) - 8] // - R14 . len(src), bytesBetweenHashLookups, &src[len(src)], x // - R15 112 candidate // // The second column (56, 64, etc) is the stack offset to spill the registers // when calling other functions. We could pack this slightly tighter, but it's // simpler to have a dedicated spill map independent of the function called. // // "var table [maxTableSize]uint16" takes up 32768 bytes of stack space. An // extra 56 bytes, to call other functions, and an extra 64 bytes, to spill // local variables (registers) during calls gives 32768 + 56 + 64 = 32888. TEXT ·encodeBlock(SB), 0, $32888-56 MOVQ dst_base+0(FP), DI MOVQ src_base+24(FP), SI MOVQ src_len+32(FP), R14 // shift, tableSize := uint32(32-8), 1<<8 MOVQ $24, CX MOVQ $256, DX calcShift: // for ; tableSize < maxTableSize && tableSize < len(src); tableSize *= 2 { // shift-- // } CMPQ DX, $16384 JGE varTable CMPQ DX, R14 JGE varTable SUBQ $1, CX SHLQ $1, DX JMP calcShift varTable: // var table [maxTableSize]uint16 // // In the asm code, unlike the Go code, we can zero-initialize only the // first tableSize elements. Each uint16 element is 2 bytes and each MOVOU // writes 16 bytes, so we can do only tableSize/8 writes instead of the // 2048 writes that would zero-initialize all of table's 32768 bytes. SHRQ $3, DX LEAQ table-32768(SP), BX PXOR X0, X0 memclr: MOVOU X0, 0(BX) ADDQ $16, BX SUBQ $1, DX JNZ memclr // !!! DX = &src[0] MOVQ SI, DX // sLimit := len(src) - inputMargin MOVQ R14, R9 SUBQ $15, R9 // !!! Pre-emptively spill CX, DX and R9 to the stack. Their values don't // change for the rest of the function. MOVQ CX, 56(SP) MOVQ DX, 64(SP) MOVQ R9, 88(SP) // nextEmit := 0 MOVQ DX, R10 // s := 1 ADDQ $1, SI // nextHash := hash(load32(src, s), shift) MOVL 0(SI), R11 IMULL $0x1e35a7bd, R11 SHRL CX, R11 outer: // for { etc } // skip := 32 MOVQ $32, R12 // nextS := s MOVQ SI, R13 // candidate := 0 MOVQ $0, R15 inner0: // for { etc } // s := nextS MOVQ R13, SI // bytesBetweenHashLookups := skip >> 5 MOVQ R12, R14 SHRQ $5, R14 // nextS = s + bytesBetweenHashLookups ADDQ R14, R13 // skip += bytesBetweenHashLookups ADDQ R14, R12 // if nextS > sLimit { goto emitRemainder } MOVQ R13, AX SUBQ DX, AX CMPQ AX, R9 JA emitRemainder // candidate = int(table[nextHash]) // XXX: MOVWQZX table-32768(SP)(R11*2), R15 // XXX: 4e 0f b7 7c 5c 78 movzwq 0x78(%rsp,%r11,2),%r15 BYTE $0x4e BYTE $0x0f BYTE $0xb7 BYTE $0x7c BYTE $0x5c BYTE $0x78 // table[nextHash] = uint16(s) MOVQ SI, AX SUBQ DX, AX // XXX: MOVW AX, table-32768(SP)(R11*2) // XXX: 66 42 89 44 5c 78 mov %ax,0x78(%rsp,%r11,2) BYTE $0x66 BYTE $0x42 BYTE $0x89 BYTE $0x44 BYTE $0x5c BYTE $0x78 // nextHash = hash(load32(src, nextS), shift) MOVL 0(R13), R11 IMULL $0x1e35a7bd, R11 SHRL CX, R11 // if load32(src, s) != load32(src, candidate) { continue } break MOVL 0(SI), AX MOVL (DX)(R15*1), BX CMPL AX, BX JNE inner0 fourByteMatch: // As per the encode_other.go code: // // A 4-byte match has been found. We'll later see etc. // !!! Jump to a fast path for short (<= 16 byte) literals. See the comment // on inputMargin in encode.go. MOVQ SI, AX SUBQ R10, AX CMPQ AX, $16 JLE emitLiteralFastPath // ---------------------------------------- // Begin inline of the emitLiteral call. // // d += emitLiteral(dst[d:], src[nextEmit:s]) MOVL AX, BX SUBL $1, BX CMPL BX, $60 JLT inlineEmitLiteralOneByte CMPL BX, $256 JLT inlineEmitLiteralTwoBytes inlineEmitLiteralThreeBytes: MOVB $0xf4, 0(DI) MOVW BX, 1(DI) ADDQ $3, DI JMP inlineEmitLiteralMemmove inlineEmitLiteralTwoBytes: MOVB $0xf0, 0(DI) MOVB BX, 1(DI) ADDQ $2, DI JMP inlineEmitLiteralMemmove inlineEmitLiteralOneByte: SHLB $2, BX MOVB BX, 0(DI) ADDQ $1, DI inlineEmitLiteralMemmove: // Spill local variables (registers) onto the stack; call; unspill. // // copy(dst[i:], lit) // // This means calling runtime·memmove(&dst[i], &lit[0], len(lit)), so we push // DI, R10 and AX as arguments. MOVQ DI, 0(SP) MOVQ R10, 8(SP) MOVQ AX, 16(SP) ADDQ AX, DI // Finish the "d +=" part of "d += emitLiteral(etc)". MOVQ SI, 72(SP) MOVQ DI, 80(SP) MOVQ R15, 112(SP) CALL runtime·memmove(SB) MOVQ 56(SP), CX MOVQ 64(SP), DX MOVQ 72(SP), SI MOVQ 80(SP), DI MOVQ 88(SP), R9 MOVQ 112(SP), R15 JMP inner1 inlineEmitLiteralEnd: // End inline of the emitLiteral call. // ---------------------------------------- emitLiteralFastPath: // !!! Emit the 1-byte encoding "uint8(len(lit)-1)<<2". MOVB AX, BX SUBB $1, BX SHLB $2, BX MOVB BX, (DI) ADDQ $1, DI // !!! Implement the copy from lit to dst as a 16-byte load and store. // (Encode's documentation says that dst and src must not overlap.) // // This always copies 16 bytes, instead of only len(lit) bytes, but that's // OK. Subsequent iterations will fix up the overrun. // // Note that on amd64, it is legal and cheap to issue unaligned 8-byte or // 16-byte loads and stores. This technique probably wouldn't be as // effective on architectures that are fussier about alignment. MOVOU 0(R10), X0 MOVOU X0, 0(DI) ADDQ AX, DI inner1: // for { etc } // base := s MOVQ SI, R12 // !!! offset := base - candidate MOVQ R12, R11 SUBQ R15, R11 SUBQ DX, R11 // ---------------------------------------- // Begin inline of the extendMatch call. // // s = extendMatch(src, candidate+4, s+4) // !!! R14 = &src[len(src)] MOVQ src_len+32(FP), R14 ADDQ DX, R14 // !!! R13 = &src[len(src) - 8] MOVQ R14, R13 SUBQ $8, R13 // !!! R15 = &src[candidate + 4] ADDQ $4, R15 ADDQ DX, R15 // !!! s += 4 ADDQ $4, SI inlineExtendMatchCmp8: // As long as we are 8 or more bytes before the end of src, we can load and // compare 8 bytes at a time. If those 8 bytes are equal, repeat. CMPQ SI, R13 JA inlineExtendMatchCmp1 MOVQ (R15), AX MOVQ (SI), BX CMPQ AX, BX JNE inlineExtendMatchBSF ADDQ $8, R15 ADDQ $8, SI JMP inlineExtendMatchCmp8 inlineExtendMatchBSF: // If those 8 bytes were not equal, XOR the two 8 byte values, and return // the index of the first byte that differs. The BSF instruction finds the // least significant 1 bit, the amd64 architecture is little-endian, and // the shift by 3 converts a bit index to a byte index. XORQ AX, BX BSFQ BX, BX SHRQ $3, BX ADDQ BX, SI JMP inlineExtendMatchEnd inlineExtendMatchCmp1: // In src's tail, compare 1 byte at a time. CMPQ SI, R14 JAE inlineExtendMatchEnd MOVB (R15), AX MOVB (SI), BX CMPB AX, BX JNE inlineExtendMatchEnd ADDQ $1, R15 ADDQ $1, SI JMP inlineExtendMatchCmp1 inlineExtendMatchEnd: // End inline of the extendMatch call. // ---------------------------------------- // ---------------------------------------- // Begin inline of the emitCopy call. // // d += emitCopy(dst[d:], base-candidate, s-base) // !!! length := s - base MOVQ SI, AX SUBQ R12, AX inlineEmitCopyLoop0: // for length >= 68 { etc } CMPL AX, $68 JLT inlineEmitCopyStep1 // Emit a length 64 copy, encoded as 3 bytes. MOVB $0xfe, 0(DI) MOVW R11, 1(DI) ADDQ $3, DI SUBL $64, AX JMP inlineEmitCopyLoop0 inlineEmitCopyStep1: // if length > 64 { etc } CMPL AX, $64 JLE inlineEmitCopyStep2 // Emit a length 60 copy, encoded as 3 bytes. MOVB $0xee, 0(DI) MOVW R11, 1(DI) ADDQ $3, DI SUBL $60, AX inlineEmitCopyStep2: // if length >= 12 || offset >= 2048 { goto inlineEmitCopyStep3 } CMPL AX, $12 JGE inlineEmitCopyStep3 CMPL R11, $2048 JGE inlineEmitCopyStep3 // Emit the remaining copy, encoded as 2 bytes. MOVB R11, 1(DI) SHRL $8, R11 SHLB $5, R11 SUBB $4, AX SHLB $2, AX ORB AX, R11 ORB $1, R11 MOVB R11, 0(DI) ADDQ $2, DI JMP inlineEmitCopyEnd inlineEmitCopyStep3: // Emit the remaining copy, encoded as 3 bytes. SUBL $1, AX SHLB $2, AX ORB $2, AX MOVB AX, 0(DI) MOVW R11, 1(DI) ADDQ $3, DI inlineEmitCopyEnd: // End inline of the emitCopy call. // ---------------------------------------- // nextEmit = s MOVQ SI, R10 // if s >= sLimit { goto emitRemainder } MOVQ SI, AX SUBQ DX, AX CMPQ AX, R9 JAE emitRemainder // As per the encode_other.go code: // // We could immediately etc. // x := load64(src, s-1) MOVQ -1(SI), R14 // prevHash := hash(uint32(x>>0), shift) MOVL R14, R11 IMULL $0x1e35a7bd, R11 SHRL CX, R11 // table[prevHash] = uint16(s-1) MOVQ SI, AX SUBQ DX, AX SUBQ $1, AX // XXX: MOVW AX, table-32768(SP)(R11*2) // XXX: 66 42 89 44 5c 78 mov %ax,0x78(%rsp,%r11,2) BYTE $0x66 BYTE $0x42 BYTE $0x89 BYTE $0x44 BYTE $0x5c BYTE $0x78 // currHash := hash(uint32(x>>8), shift) SHRQ $8, R14 MOVL R14, R11 IMULL $0x1e35a7bd, R11 SHRL CX, R11 // candidate = int(table[currHash]) // XXX: MOVWQZX table-32768(SP)(R11*2), R15 // XXX: 4e 0f b7 7c 5c 78 movzwq 0x78(%rsp,%r11,2),%r15 BYTE $0x4e BYTE $0x0f BYTE $0xb7 BYTE $0x7c BYTE $0x5c BYTE $0x78 // table[currHash] = uint16(s) ADDQ $1, AX // XXX: MOVW AX, table-32768(SP)(R11*2) // XXX: 66 42 89 44 5c 78 mov %ax,0x78(%rsp,%r11,2) BYTE $0x66 BYTE $0x42 BYTE $0x89 BYTE $0x44 BYTE $0x5c BYTE $0x78 // if uint32(x>>8) == load32(src, candidate) { continue } MOVL (DX)(R15*1), BX CMPL R14, BX JEQ inner1 // nextHash = hash(uint32(x>>16), shift) SHRQ $8, R14 MOVL R14, R11 IMULL $0x1e35a7bd, R11 SHRL CX, R11 // s++ ADDQ $1, SI // break out of the inner1 for loop, i.e. continue the outer loop. JMP outer emitRemainder: // if nextEmit < len(src) { etc } MOVQ src_len+32(FP), AX ADDQ DX, AX CMPQ R10, AX JEQ encodeBlockEnd // d += emitLiteral(dst[d:], src[nextEmit:]) // // Push args. MOVQ DI, 0(SP) MOVQ $0, 8(SP) // Unnecessary, as the callee ignores it, but conservative. MOVQ $0, 16(SP) // Unnecessary, as the callee ignores it, but conservative. MOVQ R10, 24(SP) SUBQ R10, AX MOVQ AX, 32(SP) MOVQ AX, 40(SP) // Unnecessary, as the callee ignores it, but conservative. // Spill local variables (registers) onto the stack; call; unspill. MOVQ DI, 80(SP) CALL ·emitLiteral(SB) MOVQ 80(SP), DI // Finish the "d +=" part of "d += emitLiteral(etc)". ADDQ 48(SP), DI encodeBlockEnd: MOVQ dst_base+0(FP), AX SUBQ AX, DI MOVQ DI, d+48(FP) RET golang-github-golang-snappy-0.0+git20160529.d9eb7a3/encode_other.go000066400000000000000000000200211272247357100244140ustar00rootroot00000000000000// Copyright 2016 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // +build !amd64 appengine !gc noasm package snappy func load32(b []byte, i int) uint32 { b = b[i : i+4 : len(b)] // Help the compiler eliminate bounds checks on the next line. return uint32(b[0]) | uint32(b[1])<<8 | uint32(b[2])<<16 | uint32(b[3])<<24 } func load64(b []byte, i int) uint64 { b = b[i : i+8 : len(b)] // Help the compiler eliminate bounds checks on the next line. return uint64(b[0]) | uint64(b[1])<<8 | uint64(b[2])<<16 | uint64(b[3])<<24 | uint64(b[4])<<32 | uint64(b[5])<<40 | uint64(b[6])<<48 | uint64(b[7])<<56 } // emitLiteral writes a literal chunk and returns the number of bytes written. // // It assumes that: // dst is long enough to hold the encoded bytes // 1 <= len(lit) && len(lit) <= 65536 func emitLiteral(dst, lit []byte) int { i, n := 0, uint(len(lit)-1) switch { case n < 60: dst[0] = uint8(n)<<2 | tagLiteral i = 1 case n < 1<<8: dst[0] = 60<<2 | tagLiteral dst[1] = uint8(n) i = 2 default: dst[0] = 61<<2 | tagLiteral dst[1] = uint8(n) dst[2] = uint8(n >> 8) i = 3 } return i + copy(dst[i:], lit) } // emitCopy writes a copy chunk and returns the number of bytes written. // // It assumes that: // dst is long enough to hold the encoded bytes // 1 <= offset && offset <= 65535 // 4 <= length && length <= 65535 func emitCopy(dst []byte, offset, length int) int { i := 0 // The maximum length for a single tagCopy1 or tagCopy2 op is 64 bytes. The // threshold for this loop is a little higher (at 68 = 64 + 4), and the // length emitted down below is is a little lower (at 60 = 64 - 4), because // it's shorter to encode a length 67 copy as a length 60 tagCopy2 followed // by a length 7 tagCopy1 (which encodes as 3+2 bytes) than to encode it as // a length 64 tagCopy2 followed by a length 3 tagCopy2 (which encodes as // 3+3 bytes). The magic 4 in the 64±4 is because the minimum length for a // tagCopy1 op is 4 bytes, which is why a length 3 copy has to be an // encodes-as-3-bytes tagCopy2 instead of an encodes-as-2-bytes tagCopy1. for length >= 68 { // Emit a length 64 copy, encoded as 3 bytes. dst[i+0] = 63<<2 | tagCopy2 dst[i+1] = uint8(offset) dst[i+2] = uint8(offset >> 8) i += 3 length -= 64 } if length > 64 { // Emit a length 60 copy, encoded as 3 bytes. dst[i+0] = 59<<2 | tagCopy2 dst[i+1] = uint8(offset) dst[i+2] = uint8(offset >> 8) i += 3 length -= 60 } if length >= 12 || offset >= 2048 { // Emit the remaining copy, encoded as 3 bytes. dst[i+0] = uint8(length-1)<<2 | tagCopy2 dst[i+1] = uint8(offset) dst[i+2] = uint8(offset >> 8) return i + 3 } // Emit the remaining copy, encoded as 2 bytes. dst[i+0] = uint8(offset>>8)<<5 | uint8(length-4)<<2 | tagCopy1 dst[i+1] = uint8(offset) return i + 2 } // extendMatch returns the largest k such that k <= len(src) and that // src[i:i+k-j] and src[j:k] have the same contents. // // It assumes that: // 0 <= i && i < j && j <= len(src) func extendMatch(src []byte, i, j int) int { for ; j < len(src) && src[i] == src[j]; i, j = i+1, j+1 { } return j } func hash(u, shift uint32) uint32 { return (u * 0x1e35a7bd) >> shift } // encodeBlock encodes a non-empty src to a guaranteed-large-enough dst. It // assumes that the varint-encoded length of the decompressed bytes has already // been written. // // It also assumes that: // len(dst) >= MaxEncodedLen(len(src)) && // minNonLiteralBlockSize <= len(src) && len(src) <= maxBlockSize func encodeBlock(dst, src []byte) (d int) { // Initialize the hash table. Its size ranges from 1<<8 to 1<<14 inclusive. // The table element type is uint16, as s < sLimit and sLimit < len(src) // and len(src) <= maxBlockSize and maxBlockSize == 65536. const ( maxTableSize = 1 << 14 // tableMask is redundant, but helps the compiler eliminate bounds // checks. tableMask = maxTableSize - 1 ) shift := uint32(32 - 8) for tableSize := 1 << 8; tableSize < maxTableSize && tableSize < len(src); tableSize *= 2 { shift-- } // In Go, all array elements are zero-initialized, so there is no advantage // to a smaller tableSize per se. However, it matches the C++ algorithm, // and in the asm versions of this code, we can get away with zeroing only // the first tableSize elements. var table [maxTableSize]uint16 // sLimit is when to stop looking for offset/length copies. The inputMargin // lets us use a fast path for emitLiteral in the main loop, while we are // looking for copies. sLimit := len(src) - inputMargin // nextEmit is where in src the next emitLiteral should start from. nextEmit := 0 // The encoded form must start with a literal, as there are no previous // bytes to copy, so we start looking for hash matches at s == 1. s := 1 nextHash := hash(load32(src, s), shift) for { // Copied from the C++ snappy implementation: // // Heuristic match skipping: If 32 bytes are scanned with no matches // found, start looking only at every other byte. If 32 more bytes are // scanned (or skipped), look at every third byte, etc.. When a match // is found, immediately go back to looking at every byte. This is a // small loss (~5% performance, ~0.1% density) for compressible data // due to more bookkeeping, but for non-compressible data (such as // JPEG) it's a huge win since the compressor quickly "realizes" the // data is incompressible and doesn't bother looking for matches // everywhere. // // The "skip" variable keeps track of how many bytes there are since // the last match; dividing it by 32 (ie. right-shifting by five) gives // the number of bytes to move ahead for each iteration. skip := 32 nextS := s candidate := 0 for { s = nextS bytesBetweenHashLookups := skip >> 5 nextS = s + bytesBetweenHashLookups skip += bytesBetweenHashLookups if nextS > sLimit { goto emitRemainder } candidate = int(table[nextHash&tableMask]) table[nextHash&tableMask] = uint16(s) nextHash = hash(load32(src, nextS), shift) if load32(src, s) == load32(src, candidate) { break } } // A 4-byte match has been found. We'll later see if more than 4 bytes // match. But, prior to the match, src[nextEmit:s] are unmatched. Emit // them as literal bytes. d += emitLiteral(dst[d:], src[nextEmit:s]) // Call emitCopy, and then see if another emitCopy could be our next // move. Repeat until we find no match for the input immediately after // what was consumed by the last emitCopy call. // // If we exit this loop normally then we need to call emitLiteral next, // though we don't yet know how big the literal will be. We handle that // by proceeding to the next iteration of the main loop. We also can // exit this loop via goto if we get close to exhausting the input. for { // Invariant: we have a 4-byte match at s, and no need to emit any // literal bytes prior to s. base := s // Extend the 4-byte match as long as possible. // // This is an inlined version of: // s = extendMatch(src, candidate+4, s+4) s += 4 for i := candidate + 4; s < len(src) && src[i] == src[s]; i, s = i+1, s+1 { } d += emitCopy(dst[d:], base-candidate, s-base) nextEmit = s if s >= sLimit { goto emitRemainder } // We could immediately start working at s now, but to improve // compression we first update the hash table at s-1 and at s. If // another emitCopy is not our next move, also calculate nextHash // at s+1. At least on GOARCH=amd64, these three hash calculations // are faster as one load64 call (with some shifts) instead of // three load32 calls. x := load64(src, s-1) prevHash := hash(uint32(x>>0), shift) table[prevHash&tableMask] = uint16(s - 1) currHash := hash(uint32(x>>8), shift) candidate = int(table[currHash&tableMask]) table[currHash&tableMask] = uint16(s) if uint32(x>>8) != load32(src, candidate) { nextHash = hash(uint32(x>>16), shift) s++ break } } } emitRemainder: if nextEmit < len(src) { d += emitLiteral(dst[d:], src[nextEmit:]) } return d } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/golden_test.go000066400000000000000000001236141272247357100243010ustar00rootroot00000000000000// Copyright 2016 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package snappy // extendMatchGoldenTestCases is the i and j arguments, and the returned value, // for every extendMatch call issued when encoding the // testdata/Mark.Twain-Tom.Sawyer.txt file. It is used to benchmark the // extendMatch implementation. // // It was generated manually by adding some print statements to the (pure Go) // extendMatch implementation: // // func extendMatch(src []byte, i, j int) int { // i0, j0 := i, j // for ; j < len(src) && src[i] == src[j]; i, j = i+1, j+1 { // } // println("{", i0, ",", j0, ",", j, "},") // return j // } // // and running "go test -test.run=EncodeGoldenInput -tags=noasm". var extendMatchGoldenTestCases = []struct { i, j, want int }{ {11, 61, 62}, {80, 81, 82}, {86, 87, 101}, {85, 133, 149}, {152, 153, 162}, {133, 168, 193}, {168, 207, 225}, {81, 255, 275}, {278, 279, 283}, {306, 417, 417}, {373, 428, 430}, {389, 444, 447}, {474, 510, 512}, {465, 533, 533}, {47, 547, 547}, {307, 551, 554}, {420, 582, 587}, {309, 604, 604}, {604, 625, 625}, {538, 629, 629}, {328, 640, 640}, {573, 645, 645}, {319, 657, 657}, {30, 664, 664}, {45, 679, 680}, {621, 684, 684}, {376, 700, 700}, {33, 707, 708}, {601, 733, 733}, {334, 744, 745}, {625, 758, 759}, {382, 763, 763}, {550, 769, 771}, {533, 789, 789}, {804, 813, 813}, {342, 841, 842}, {742, 847, 847}, {74, 852, 852}, {810, 864, 864}, {758, 868, 869}, {714, 883, 883}, {582, 889, 891}, {61, 934, 935}, {894, 942, 942}, {939, 949, 949}, {785, 956, 957}, {886, 978, 978}, {792, 998, 998}, {998, 1005, 1005}, {572, 1032, 1032}, {698, 1051, 1053}, {599, 1067, 1069}, {1056, 1079, 1079}, {942, 1089, 1090}, {831, 1094, 1096}, {1088, 1100, 1103}, {732, 1113, 1114}, {1037, 1118, 1118}, {872, 1128, 1130}, {1079, 1140, 1142}, {332, 1162, 1162}, {207, 1168, 1186}, {1189, 1190, 1225}, {105, 1229, 1230}, {79, 1256, 1257}, {1190, 1261, 1283}, {255, 1306, 1306}, {1319, 1339, 1358}, {364, 1370, 1370}, {955, 1378, 1380}, {122, 1403, 1403}, {1325, 1407, 1419}, {664, 1423, 1424}, {941, 1461, 1463}, {867, 1477, 1478}, {757, 1488, 1489}, {1140, 1499, 1499}, {31, 1506, 1506}, {1487, 1510, 1512}, {1089, 1520, 1521}, {1467, 1525, 1529}, {1394, 1537, 1537}, {1499, 1541, 1541}, {367, 1558, 1558}, {1475, 1564, 1564}, {1525, 1568, 1571}, {1541, 1582, 1583}, {864, 1587, 1588}, {704, 1597, 1597}, {336, 1602, 1602}, {1383, 1613, 1613}, {1498, 1617, 1618}, {1051, 1623, 1625}, {401, 1643, 1645}, {1072, 1654, 1655}, {1067, 1667, 1669}, {699, 1673, 1674}, {1587, 1683, 1684}, {920, 1696, 1696}, {1505, 1710, 1710}, {1550, 1723, 1723}, {996, 1727, 1727}, {833, 1733, 1734}, {1638, 1739, 1740}, {1654, 1744, 1744}, {753, 1761, 1761}, {1548, 1773, 1773}, {1568, 1777, 1780}, {1683, 1793, 1794}, {948, 1801, 1801}, {1666, 1805, 1808}, {1502, 1814, 1814}, {1696, 1822, 1822}, {502, 1836, 1837}, {917, 1843, 1843}, {1733, 1854, 1855}, {970, 1859, 1859}, {310, 1863, 1863}, {657, 1872, 1872}, {1005, 1876, 1876}, {1662, 1880, 1880}, {904, 1892, 1892}, {1427, 1910, 1910}, {1772, 1929, 1930}, {1822, 1937, 1940}, {1858, 1949, 1950}, {1602, 1956, 1956}, {1150, 1962, 1962}, {1504, 1966, 1967}, {51, 1971, 1971}, {1605, 1979, 1979}, {1458, 1983, 1988}, {1536, 2001, 2006}, {1373, 2014, 2018}, {1494, 2025, 2025}, {1667, 2029, 2031}, {1592, 2035, 2035}, {330, 2045, 2045}, {1376, 2053, 2053}, {1991, 2058, 2059}, {1635, 2065, 2065}, {1992, 2073, 2074}, {2014, 2080, 2081}, {1546, 2085, 2087}, {59, 2099, 2099}, {1996, 2106, 2106}, {1836, 2110, 2110}, {2068, 2114, 2114}, {1338, 2122, 2122}, {1562, 2128, 2130}, {1934, 2134, 2134}, {2114, 2141, 2142}, {977, 2149, 2150}, {956, 2154, 2155}, {1407, 2162, 2162}, {1773, 2166, 2166}, {883, 2171, 2171}, {623, 2175, 2178}, {1520, 2191, 2192}, {1162, 2200, 2200}, {912, 2204, 2204}, {733, 2208, 2208}, {1777, 2212, 2215}, {1532, 2219, 2219}, {718, 2223, 2225}, {2069, 2229, 2229}, {2207, 2245, 2246}, {1139, 2264, 2264}, {677, 2274, 2274}, {2099, 2279, 2279}, {1863, 2283, 2283}, {1966, 2305, 2306}, {2279, 2313, 2313}, {1628, 2319, 2319}, {755, 2329, 2329}, {1461, 2334, 2334}, {2117, 2340, 2340}, {2313, 2349, 2349}, {1859, 2353, 2353}, {1048, 2362, 2362}, {895, 2366, 2366}, {2278, 2373, 2373}, {1884, 2377, 2377}, {1402, 2387, 2392}, {700, 2398, 2398}, {1971, 2402, 2402}, {2009, 2419, 2419}, {1441, 2426, 2428}, {2208, 2432, 2432}, {2038, 2436, 2436}, {932, 2443, 2443}, {1759, 2447, 2448}, {744, 2452, 2452}, {1875, 2458, 2458}, {2405, 2468, 2468}, {1596, 2472, 2473}, {1953, 2480, 2482}, {736, 2487, 2487}, {1913, 2493, 2493}, {774, 2497, 2497}, {1484, 2506, 2508}, {2432, 2512, 2512}, {752, 2519, 2519}, {2497, 2523, 2523}, {2409, 2528, 2529}, {2122, 2533, 2533}, {2396, 2537, 2538}, {2410, 2547, 2548}, {1093, 2555, 2560}, {551, 2564, 2565}, {2268, 2569, 2569}, {1362, 2580, 2580}, {1916, 2584, 2585}, {994, 2589, 2590}, {1979, 2596, 2596}, {1041, 2602, 2602}, {2104, 2614, 2616}, {2609, 2621, 2628}, {2329, 2638, 2638}, {2211, 2657, 2658}, {2638, 2662, 2667}, {2578, 2676, 2679}, {2153, 2685, 2686}, {2608, 2696, 2697}, {598, 2712, 2712}, {2620, 2719, 2720}, {1888, 2724, 2728}, {2709, 2732, 2732}, {1365, 2739, 2739}, {784, 2747, 2748}, {424, 2753, 2753}, {2204, 2759, 2759}, {812, 2768, 2769}, {2455, 2773, 2773}, {1722, 2781, 2781}, {1917, 2792, 2792}, {2705, 2799, 2799}, {2685, 2806, 2807}, {2742, 2811, 2811}, {1370, 2818, 2818}, {2641, 2830, 2830}, {2512, 2837, 2837}, {2457, 2841, 2841}, {2756, 2845, 2845}, {2719, 2855, 2855}, {1423, 2859, 2859}, {2849, 2863, 2865}, {1474, 2871, 2871}, {1161, 2875, 2876}, {2282, 2880, 2881}, {2746, 2888, 2888}, {1783, 2893, 2893}, {2401, 2899, 2900}, {2632, 2920, 2923}, {2422, 2928, 2930}, {2715, 2939, 2939}, {2162, 2943, 2943}, {2859, 2947, 2947}, {1910, 2951, 2951}, {1431, 2955, 2956}, {1439, 2964, 2964}, {2501, 2968, 2969}, {2029, 2973, 2976}, {689, 2983, 2984}, {1658, 2988, 2988}, {1031, 2996, 2996}, {2149, 3001, 3002}, {25, 3009, 3013}, {2964, 3023, 3023}, {953, 3027, 3028}, {2359, 3036, 3036}, {3023, 3049, 3049}, {2880, 3055, 3056}, {2973, 3076, 3077}, {2874, 3090, 3090}, {2871, 3094, 3094}, {2532, 3100, 3100}, {2938, 3107, 3108}, {350, 3115, 3115}, {2196, 3119, 3121}, {1133, 3127, 3129}, {1797, 3134, 3150}, {3032, 3158, 3158}, {3016, 3172, 3172}, {2533, 3179, 3179}, {3055, 3187, 3188}, {1384, 3192, 3193}, {2799, 3199, 3199}, {2126, 3203, 3207}, {2334, 3215, 3215}, {2105, 3220, 3221}, {3199, 3229, 3229}, {2891, 3233, 3233}, {855, 3240, 3240}, {1852, 3253, 3256}, {2140, 3263, 3263}, {1682, 3268, 3270}, {3243, 3274, 3274}, {924, 3279, 3279}, {2212, 3283, 3283}, {2596, 3287, 3287}, {2999, 3291, 3291}, {2353, 3295, 3295}, {2480, 3302, 3304}, {1959, 3308, 3311}, {3000, 3318, 3318}, {845, 3330, 3330}, {2283, 3334, 3334}, {2519, 3342, 3342}, {3325, 3346, 3348}, {2397, 3353, 3354}, {2763, 3358, 3358}, {3198, 3363, 3364}, {3211, 3368, 3372}, {2950, 3376, 3377}, {3245, 3388, 3391}, {2264, 3398, 3398}, {795, 3403, 3403}, {3287, 3407, 3407}, {3358, 3411, 3411}, {3317, 3415, 3415}, {3232, 3431, 3431}, {2128, 3435, 3437}, {3236, 3441, 3441}, {3398, 3445, 3446}, {2814, 3450, 3450}, {3394, 3466, 3466}, {2425, 3470, 3470}, {3330, 3476, 3476}, {1612, 3480, 3480}, {1004, 3485, 3486}, {2732, 3490, 3490}, {1117, 3494, 3495}, {629, 3501, 3501}, {3087, 3514, 3514}, {684, 3518, 3518}, {3489, 3522, 3524}, {1760, 3529, 3529}, {617, 3537, 3537}, {3431, 3541, 3541}, {997, 3547, 3547}, {882, 3552, 3553}, {2419, 3558, 3558}, {610, 3562, 3563}, {1903, 3567, 3569}, {3005, 3575, 3575}, {3076, 3585, 3586}, {3541, 3590, 3590}, {3490, 3594, 3594}, {1899, 3599, 3599}, {3545, 3606, 3606}, {3290, 3614, 3615}, {2056, 3619, 3620}, {3556, 3625, 3625}, {3294, 3632, 3633}, {637, 3643, 3644}, {3609, 3648, 3650}, {3175, 3658, 3658}, {3498, 3665, 3665}, {1597, 3669, 3669}, {1983, 3673, 3673}, {3215, 3682, 3682}, {3544, 3689, 3689}, {3694, 3698, 3698}, {3228, 3715, 3716}, {2594, 3720, 3722}, {3573, 3726, 3726}, {2479, 3732, 3735}, {3191, 3741, 3742}, {1113, 3746, 3747}, {2844, 3751, 3751}, {3445, 3756, 3757}, {3755, 3766, 3766}, {3421, 3775, 3780}, {3593, 3784, 3786}, {3263, 3796, 3796}, {3469, 3806, 3806}, {2602, 3815, 3815}, {723, 3819, 3821}, {1608, 3826, 3826}, {3334, 3830, 3830}, {2198, 3835, 3835}, {2635, 3840, 3840}, {3702, 3852, 3853}, {3406, 3858, 3859}, {3681, 3867, 3870}, {3407, 3880, 3880}, {340, 3889, 3889}, {3772, 3893, 3893}, {593, 3897, 3897}, {2563, 3914, 3916}, {2981, 3929, 3929}, {1835, 3933, 3934}, {3906, 3951, 3951}, {1459, 3958, 3958}, {3889, 3974, 3974}, {2188, 3982, 3982}, {3220, 3986, 3987}, {3585, 3991, 3993}, {3712, 3997, 4001}, {2805, 4007, 4007}, {1879, 4012, 4013}, {3618, 4018, 4018}, {1145, 4031, 4032}, {3901, 4037, 4037}, {2772, 4046, 4047}, {2802, 4053, 4054}, {3299, 4058, 4058}, {3725, 4066, 4066}, {2271, 4070, 4070}, {385, 4075, 4076}, {3624, 4089, 4090}, {3745, 4096, 4098}, {1563, 4102, 4102}, {4045, 4106, 4111}, {3696, 4115, 4119}, {3376, 4125, 4126}, {1880, 4130, 4130}, {2048, 4140, 4141}, {2724, 4149, 4149}, {1767, 4156, 4156}, {2601, 4164, 4164}, {2757, 4168, 4168}, {3974, 4172, 4172}, {3914, 4178, 4178}, {516, 4185, 4185}, {1032, 4189, 4190}, {3462, 4197, 4198}, {3805, 4202, 4203}, {3910, 4207, 4212}, {3075, 4221, 4221}, {3756, 4225, 4226}, {1872, 4236, 4237}, {3844, 4241, 4241}, {3991, 4245, 4249}, {2203, 4258, 4258}, {3903, 4267, 4268}, {705, 4272, 4272}, {1896, 4276, 4276}, {1955, 4285, 4288}, {3746, 4302, 4303}, {2672, 4311, 4311}, {3969, 4317, 4317}, {3883, 4322, 4322}, {1920, 4339, 4340}, {3527, 4344, 4346}, {1160, 4358, 4358}, {3648, 4364, 4366}, {2711, 4387, 4387}, {3619, 4391, 4392}, {1944, 4396, 4396}, {4369, 4400, 4400}, {2736, 4404, 4407}, {2546, 4411, 4412}, {4390, 4422, 4422}, {3610, 4426, 4427}, {4058, 4431, 4431}, {4374, 4435, 4435}, {3463, 4445, 4446}, {1813, 4452, 4452}, {3669, 4456, 4456}, {3830, 4460, 4460}, {421, 4464, 4465}, {1719, 4471, 4471}, {3880, 4475, 4475}, {1834, 4485, 4487}, {3590, 4491, 4491}, {442, 4496, 4497}, {4435, 4501, 4501}, {3814, 4509, 4509}, {987, 4513, 4513}, {4494, 4518, 4521}, {3218, 4526, 4529}, {4221, 4537, 4537}, {2778, 4543, 4545}, {4422, 4552, 4552}, {4031, 4558, 4559}, {4178, 4563, 4563}, {3726, 4567, 4574}, {4027, 4578, 4578}, {4339, 4585, 4587}, {3796, 4592, 4595}, {543, 4600, 4613}, {2855, 4620, 4621}, {2795, 4627, 4627}, {3440, 4631, 4632}, {4279, 4636, 4639}, {4245, 4643, 4645}, {4516, 4649, 4650}, {3133, 4654, 4654}, {4042, 4658, 4659}, {3422, 4663, 4663}, {4046, 4667, 4668}, {4267, 4672, 4672}, {4004, 4676, 4677}, {2490, 4682, 4682}, {2451, 4697, 4697}, {3027, 4705, 4705}, {4028, 4717, 4717}, {4460, 4721, 4721}, {2471, 4725, 4727}, {3090, 4735, 4735}, {3192, 4739, 4740}, {3835, 4760, 4760}, {4540, 4764, 4764}, {4007, 4772, 4774}, {619, 4784, 4784}, {3561, 4789, 4791}, {3367, 4805, 4805}, {4490, 4810, 4811}, {2402, 4815, 4815}, {3352, 4819, 4822}, {2773, 4828, 4828}, {4552, 4832, 4832}, {2522, 4840, 4841}, {316, 4847, 4852}, {4715, 4858, 4858}, {2959, 4862, 4862}, {4858, 4868, 4869}, {2134, 4873, 4873}, {578, 4878, 4878}, {4189, 4889, 4890}, {2229, 4894, 4894}, {4501, 4898, 4898}, {2297, 4903, 4903}, {2933, 4909, 4909}, {3008, 4913, 4913}, {3153, 4917, 4917}, {4819, 4921, 4921}, {4921, 4932, 4933}, {4920, 4944, 4945}, {4814, 4954, 4955}, {576, 4966, 4966}, {1854, 4970, 4971}, {1374, 4975, 4976}, {3307, 4980, 4980}, {974, 4984, 4988}, {4721, 4992, 4992}, {4898, 4996, 4996}, {4475, 5006, 5006}, {3819, 5012, 5012}, {1948, 5019, 5021}, {4954, 5027, 5029}, {3740, 5038, 5040}, {4763, 5044, 5045}, {1936, 5051, 5051}, {4844, 5055, 5060}, {4215, 5069, 5072}, {1146, 5076, 5076}, {3845, 5082, 5082}, {4865, 5090, 5090}, {4624, 5094, 5094}, {4815, 5098, 5098}, {5006, 5105, 5105}, {4980, 5109, 5109}, {4795, 5113, 5115}, {5043, 5119, 5121}, {4782, 5129, 5129}, {3826, 5139, 5139}, {3876, 5156, 5156}, {3111, 5167, 5171}, {1470, 5177, 5177}, {4431, 5181, 5181}, {546, 5189, 5189}, {4225, 5193, 5193}, {1672, 5199, 5201}, {4207, 5205, 5209}, {4220, 5216, 5217}, {4658, 5224, 5225}, {3295, 5235, 5235}, {2436, 5239, 5239}, {2349, 5246, 5246}, {2175, 5250, 5250}, {5180, 5257, 5258}, {3161, 5263, 5263}, {5105, 5272, 5272}, {3552, 5282, 5282}, {4944, 5299, 5300}, {4130, 5312, 5313}, {902, 5323, 5323}, {913, 5327, 5327}, {2987, 5333, 5334}, {5150, 5344, 5344}, {5249, 5348, 5348}, {1965, 5358, 5359}, {5330, 5364, 5364}, {2012, 5373, 5377}, {712, 5384, 5386}, {5235, 5390, 5390}, {5044, 5398, 5399}, {564, 5406, 5406}, {39, 5410, 5410}, {4642, 5422, 5425}, {4421, 5437, 5438}, {2347, 5449, 5449}, {5333, 5453, 5454}, {4136, 5458, 5459}, {3793, 5468, 5468}, {2243, 5480, 5480}, {4889, 5492, 5493}, {4295, 5504, 5504}, {2785, 5511, 5511}, {2377, 5518, 5518}, {3662, 5525, 5525}, {5097, 5529, 5530}, {4781, 5537, 5538}, {4697, 5547, 5548}, {436, 5552, 5553}, {5542, 5558, 5558}, {3692, 5562, 5562}, {2696, 5568, 5569}, {4620, 5578, 5578}, {2898, 5590, 5590}, {5557, 5596, 5618}, {2797, 5623, 5625}, {2792, 5629, 5629}, {5243, 5633, 5633}, {5348, 5637, 5637}, {5547, 5643, 5643}, {4296, 5654, 5655}, {5568, 5662, 5662}, {3001, 5670, 5671}, {3794, 5679, 5679}, {4006, 5685, 5686}, {4969, 5690, 5692}, {687, 5704, 5704}, {4563, 5708, 5708}, {1723, 5738, 5738}, {649, 5742, 5742}, {5163, 5748, 5755}, {3907, 5759, 5759}, {3074, 5764, 5764}, {5326, 5771, 5771}, {2951, 5776, 5776}, {5181, 5780, 5780}, {2614, 5785, 5788}, {4709, 5794, 5794}, {2784, 5799, 5799}, {5518, 5803, 5803}, {4155, 5812, 5815}, {921, 5819, 5819}, {5224, 5823, 5824}, {2853, 5830, 5836}, {5776, 5840, 5840}, {2955, 5844, 5845}, {5745, 5853, 5853}, {3291, 5857, 5857}, {2988, 5861, 5861}, {2647, 5865, 5865}, {5398, 5869, 5870}, {1085, 5874, 5875}, {4906, 5881, 5881}, {802, 5886, 5886}, {5119, 5890, 5893}, {5802, 5899, 5900}, {3415, 5904, 5904}, {5629, 5908, 5908}, {3714, 5912, 5914}, {5558, 5921, 5921}, {2710, 5927, 5928}, {1094, 5932, 5934}, {2653, 5940, 5941}, {4735, 5954, 5954}, {5861, 5958, 5958}, {1040, 5971, 5971}, {5514, 5977, 5977}, {5048, 5981, 5982}, {5953, 5992, 5993}, {3751, 5997, 5997}, {4991, 6001, 6002}, {5885, 6006, 6007}, {5529, 6011, 6012}, {4974, 6019, 6020}, {5857, 6024, 6024}, {3483, 6032, 6032}, {3594, 6036, 6036}, {1997, 6040, 6040}, {5997, 6044, 6047}, {5197, 6051, 6051}, {1764, 6055, 6055}, {6050, 6059, 6059}, {5239, 6063, 6063}, {5049, 6067, 6067}, {5957, 6073, 6074}, {1022, 6078, 6078}, {3414, 6083, 6084}, {3809, 6090, 6090}, {4562, 6095, 6096}, {5878, 6104, 6104}, {594, 6108, 6109}, {3353, 6115, 6116}, {4992, 6120, 6121}, {2424, 6125, 6125}, {4484, 6130, 6130}, {3900, 6134, 6135}, {5793, 6139, 6141}, {3562, 6145, 6145}, {1438, 6152, 6153}, {6058, 6157, 6158}, {4411, 6162, 6163}, {4590, 6167, 6171}, {4748, 6175, 6175}, {5517, 6183, 6184}, {6095, 6191, 6192}, {1471, 6203, 6203}, {2643, 6209, 6210}, {450, 6220, 6220}, {5266, 6226, 6226}, {2576, 6233, 6233}, {2607, 6239, 6240}, {5164, 6244, 6251}, {6054, 6255, 6255}, {1789, 6260, 6261}, {5250, 6265, 6265}, {6062, 6273, 6278}, {5990, 6282, 6282}, {3283, 6286, 6286}, {5436, 6290, 6290}, {6059, 6294, 6294}, {5668, 6298, 6300}, {3072, 6324, 6329}, {3132, 6338, 6339}, {3246, 6343, 6344}, {28, 6348, 6349}, {1503, 6353, 6355}, {6067, 6359, 6359}, {3384, 6364, 6364}, {545, 6375, 6376}, {5803, 6380, 6380}, {5522, 6384, 6385}, {5908, 6389, 6389}, {2796, 6393, 6396}, {4831, 6403, 6404}, {6388, 6412, 6412}, {6005, 6417, 6420}, {4450, 6430, 6430}, {4050, 6435, 6435}, {5372, 6441, 6441}, {4378, 6447, 6447}, {6199, 6452, 6452}, {3026, 6456, 6456}, {2642, 6460, 6462}, {6392, 6470, 6470}, {6459, 6474, 6474}, {2829, 6487, 6488}, {2942, 6499, 6504}, {5069, 6508, 6511}, {5341, 6515, 6516}, {5853, 6521, 6525}, {6104, 6531, 6531}, {5759, 6535, 6538}, {4672, 6542, 6543}, {2443, 6550, 6550}, {5109, 6554, 6554}, {6494, 6558, 6560}, {6006, 6570, 6572}, {6424, 6576, 6580}, {4693, 6591, 6592}, {6439, 6596, 6597}, {3179, 6601, 6601}, {5299, 6606, 6607}, {4148, 6612, 6613}, {3774, 6617, 6617}, {3537, 6623, 6624}, {4975, 6628, 6629}, {3848, 6636, 6636}, {856, 6640, 6640}, {5724, 6645, 6645}, {6632, 6651, 6651}, {4630, 6656, 6658}, {1440, 6662, 6662}, {4281, 6666, 6667}, {4302, 6671, 6672}, {2589, 6676, 6677}, {5647, 6681, 6687}, {6082, 6691, 6693}, {6144, 6698, 6698}, {6103, 6709, 6710}, {3710, 6714, 6714}, {4253, 6718, 6721}, {2467, 6730, 6730}, {4778, 6734, 6734}, {6528, 6738, 6738}, {4358, 6747, 6747}, {5889, 6753, 6753}, {5193, 6757, 6757}, {5797, 6761, 6761}, {3858, 6765, 6766}, {5951, 6776, 6776}, {6487, 6781, 6782}, {3282, 6786, 6787}, {4667, 6797, 6799}, {1927, 6803, 6806}, {6583, 6810, 6810}, {4937, 6814, 6814}, {6099, 6824, 6824}, {4415, 6835, 6836}, {6332, 6840, 6841}, {5160, 6850, 6850}, {4764, 6854, 6854}, {6814, 6858, 6859}, {3018, 6864, 6864}, {6293, 6868, 6869}, {6359, 6877, 6877}, {3047, 6884, 6886}, {5262, 6890, 6891}, {5471, 6900, 6900}, {3268, 6910, 6912}, {1047, 6916, 6916}, {5904, 6923, 6923}, {5798, 6933, 6938}, {4149, 6942, 6942}, {1821, 6946, 6946}, {3599, 6952, 6952}, {6470, 6957, 6957}, {5562, 6961, 6961}, {6268, 6965, 6967}, {6389, 6971, 6971}, {6596, 6975, 6976}, {6553, 6980, 6981}, {6576, 6985, 6989}, {1375, 6993, 6993}, {652, 6998, 6998}, {4876, 7002, 7003}, {5768, 7011, 7013}, {3973, 7017, 7017}, {6802, 7025, 7025}, {6955, 7034, 7036}, {6974, 7040, 7040}, {5944, 7044, 7044}, {6992, 7048, 7054}, {6872, 7059, 7059}, {2943, 7063, 7063}, {6923, 7067, 7067}, {5094, 7071, 7071}, {4873, 7075, 7075}, {5819, 7079, 7079}, {5945, 7085, 7085}, {1540, 7090, 7091}, {2090, 7095, 7095}, {5024, 7104, 7105}, {6900, 7109, 7109}, {6024, 7113, 7114}, {6000, 7118, 7120}, {2187, 7124, 7125}, {6760, 7129, 7130}, {5898, 7134, 7136}, {7032, 7144, 7144}, {4271, 7148, 7148}, {3706, 7152, 7152}, {6970, 7156, 7157}, {7088, 7161, 7163}, {2718, 7168, 7169}, {5674, 7175, 7175}, {4631, 7182, 7182}, {7070, 7188, 7189}, {6220, 7196, 7196}, {3458, 7201, 7202}, {2041, 7211, 7212}, {1454, 7216, 7216}, {5199, 7225, 7227}, {3529, 7234, 7234}, {6890, 7238, 7238}, {3815, 7242, 7243}, {5490, 7250, 7253}, {6554, 7257, 7263}, {5890, 7267, 7269}, {6877, 7273, 7273}, {4877, 7277, 7277}, {2502, 7285, 7285}, {1483, 7289, 7295}, {7210, 7304, 7308}, {6845, 7313, 7316}, {7219, 7320, 7320}, {7001, 7325, 7329}, {6853, 7333, 7334}, {6120, 7338, 7338}, {6606, 7342, 7343}, {7020, 7348, 7350}, {3509, 7354, 7354}, {7133, 7359, 7363}, {3434, 7371, 7374}, {2787, 7384, 7384}, {7044, 7388, 7388}, {6960, 7394, 7395}, {6676, 7399, 7400}, {7161, 7404, 7404}, {7285, 7417, 7418}, {4558, 7425, 7426}, {4828, 7430, 7430}, {6063, 7436, 7436}, {3597, 7442, 7442}, {914, 7446, 7446}, {7320, 7452, 7454}, {7267, 7458, 7460}, {5076, 7464, 7464}, {7430, 7468, 7469}, {6273, 7473, 7474}, {7440, 7478, 7487}, {7348, 7491, 7494}, {1021, 7510, 7510}, {7473, 7515, 7515}, {2823, 7519, 7519}, {6264, 7527, 7527}, {7302, 7531, 7531}, {7089, 7535, 7535}, {7342, 7540, 7541}, {3688, 7547, 7551}, {3054, 7558, 7560}, {4177, 7566, 7567}, {6691, 7574, 7575}, {7156, 7585, 7586}, {7147, 7590, 7592}, {7407, 7598, 7598}, {7403, 7602, 7603}, {6868, 7607, 7607}, {6636, 7611, 7611}, {4805, 7617, 7617}, {5779, 7623, 7623}, {7063, 7627, 7627}, {5079, 7632, 7632}, {7377, 7637, 7637}, {7337, 7641, 7642}, {6738, 7655, 7655}, {7338, 7659, 7659}, {6541, 7669, 7671}, {595, 7675, 7675}, {7658, 7679, 7680}, {7647, 7685, 7686}, {2477, 7690, 7690}, {5823, 7694, 7694}, {4156, 7699, 7699}, {5931, 7703, 7706}, {6854, 7712, 7712}, {4931, 7718, 7718}, {6979, 7722, 7722}, {5085, 7727, 7727}, {6965, 7732, 7732}, {7201, 7736, 7737}, {3639, 7741, 7743}, {7534, 7749, 7749}, {4292, 7753, 7753}, {3427, 7759, 7763}, {7273, 7767, 7767}, {940, 7778, 7778}, {4838, 7782, 7785}, {4216, 7790, 7792}, {922, 7800, 7801}, {7256, 7810, 7811}, {7789, 7815, 7819}, {7225, 7823, 7825}, {7531, 7829, 7829}, {6997, 7833, 7833}, {7757, 7837, 7838}, {4129, 7842, 7842}, {7333, 7848, 7849}, {6776, 7855, 7855}, {7527, 7859, 7859}, {4370, 7863, 7863}, {4512, 7868, 7868}, {5679, 7880, 7880}, {3162, 7884, 7885}, {3933, 7892, 7894}, {7804, 7899, 7902}, {6363, 7906, 7907}, {7848, 7911, 7912}, {5584, 7917, 7921}, {874, 7926, 7926}, {3342, 7930, 7930}, {4507, 7935, 7937}, {3672, 7943, 7944}, {7911, 7948, 7949}, {6402, 7956, 7956}, {7940, 7960, 7960}, {7113, 7964, 7964}, {1073, 7968, 7968}, {7740, 7974, 7974}, {7601, 7978, 7982}, {6797, 7987, 7988}, {3528, 7994, 7995}, {5483, 7999, 7999}, {5717, 8011, 8011}, {5480, 8017, 8017}, {7770, 8023, 8030}, {2452, 8034, 8034}, {5282, 8047, 8047}, {7967, 8051, 8051}, {1128, 8058, 8066}, {6348, 8070, 8070}, {8055, 8077, 8077}, {7925, 8081, 8086}, {6810, 8090, 8090}, {5051, 8101, 8101}, {4696, 8109, 8110}, {5129, 8119, 8119}, {4449, 8123, 8123}, {7222, 8127, 8127}, {4649, 8131, 8134}, {7994, 8138, 8138}, {5954, 8148, 8148}, {475, 8152, 8153}, {7906, 8157, 8157}, {7458, 8164, 8166}, {7632, 8171, 8173}, {3874, 8177, 8183}, {4391, 8187, 8187}, {561, 8191, 8191}, {2417, 8195, 8195}, {2357, 8204, 8204}, {2269, 8216, 8218}, {3968, 8222, 8222}, {2200, 8226, 8227}, {3453, 8247, 8247}, {2439, 8251, 8252}, {7175, 8257, 8257}, {976, 8262, 8264}, {4953, 8273, 8273}, {4219, 8278, 8278}, {6, 8285, 8291}, {5703, 8295, 8296}, {5272, 8300, 8300}, {8037, 8304, 8304}, {8186, 8314, 8314}, {8304, 8318, 8318}, {8051, 8326, 8326}, {8318, 8330, 8330}, {2671, 8334, 8335}, {2662, 8339, 8339}, {8081, 8349, 8350}, {3328, 8356, 8356}, {2879, 8360, 8362}, {8050, 8370, 8371}, {8330, 8375, 8376}, {8375, 8386, 8386}, {4961, 8390, 8390}, {1017, 8403, 8405}, {3533, 8416, 8416}, {4555, 8422, 8422}, {6445, 8426, 8426}, {8169, 8432, 8432}, {990, 8436, 8436}, {4102, 8440, 8440}, {7398, 8444, 8446}, {3480, 8450, 8450}, {6324, 8462, 8462}, {7948, 8466, 8467}, {5950, 8471, 8471}, {5189, 8476, 8476}, {4026, 8490, 8490}, {8374, 8494, 8495}, {4682, 8501, 8501}, {7387, 8506, 8506}, {8164, 8510, 8515}, {4079, 8524, 8524}, {8360, 8529, 8531}, {7446, 8540, 8543}, {7971, 8547, 8548}, {4311, 8552, 8552}, {5204, 8556, 8557}, {7968, 8562, 8562}, {7847, 8571, 8573}, {8547, 8577, 8577}, {5320, 8581, 8581}, {8556, 8585, 8586}, {8504, 8590, 8590}, {7669, 8602, 8604}, {5874, 8608, 8609}, {5828, 8613, 8613}, {7998, 8617, 8617}, {8519, 8625, 8625}, {7250, 8637, 8637}, {426, 8641, 8641}, {8436, 8645, 8645}, {5986, 8649, 8656}, {8157, 8660, 8660}, {7182, 8665, 8665}, {8421, 8675, 8675}, {8509, 8681, 8681}, {5137, 8688, 8689}, {8625, 8694, 8695}, {5228, 8701, 8702}, {6661, 8714, 8714}, {1010, 8719, 8719}, {6648, 8723, 8723}, {3500, 8728, 8728}, {2442, 8735, 8735}, {8494, 8740, 8741}, {8171, 8753, 8755}, {7242, 8763, 8764}, {4739, 8768, 8769}, {7079, 8773, 8773}, {8386, 8777, 8777}, {8624, 8781, 8787}, {661, 8791, 8794}, {8631, 8801, 8801}, {7753, 8805, 8805}, {4783, 8809, 8810}, {1673, 8814, 8815}, {6623, 8819, 8819}, {4404, 8823, 8823}, {8089, 8827, 8828}, {8773, 8832, 8832}, {5394, 8836, 8836}, {6231, 8841, 8843}, {1015, 8852, 8853}, {6873, 8857, 8857}, {6289, 8865, 8865}, {8577, 8869, 8869}, {8114, 8873, 8875}, {8534, 8883, 8883}, {3007, 8887, 8888}, {8827, 8892, 8893}, {4788, 8897, 8900}, {5698, 8906, 8907}, {7690, 8911, 8911}, {6643, 8919, 8919}, {7206, 8923, 8924}, {7866, 8929, 8931}, {8880, 8942, 8942}, {8630, 8951, 8952}, {6027, 8958, 8958}, {7749, 8966, 8967}, {4932, 8972, 8973}, {8892, 8980, 8981}, {634, 9003, 9003}, {8109, 9007, 9008}, {8777, 9012, 9012}, {3981, 9016, 9017}, {5723, 9025, 9025}, {7662, 9034, 9038}, {8955, 9042, 9042}, {8070, 9060, 9062}, {8910, 9066, 9066}, {5363, 9070, 9071}, {7699, 9075, 9076}, {8991, 9081, 9081}, {6850, 9085, 9085}, {5811, 9092, 9094}, {9079, 9098, 9102}, {6456, 9106, 9106}, {2259, 9111, 9111}, {4752, 9116, 9116}, {9060, 9120, 9123}, {8090, 9127, 9127}, {5305, 9131, 9132}, {8623, 9137, 9137}, {7417, 9141, 9141}, {6564, 9148, 9149}, {9126, 9157, 9158}, {4285, 9169, 9170}, {8698, 9174, 9174}, {8869, 9178, 9178}, {2572, 9182, 9183}, {6482, 9188, 9190}, {9181, 9201, 9201}, {2968, 9208, 9209}, {2506, 9213, 9215}, {9127, 9219, 9219}, {7910, 9225, 9227}, {5422, 9235, 9239}, {8813, 9244, 9246}, {9178, 9250, 9250}, {8748, 9255, 9255}, {7354, 9265, 9265}, {7767, 9269, 9269}, {7710, 9281, 9283}, {8826, 9288, 9290}, {861, 9295, 9295}, {4482, 9301, 9301}, {9264, 9305, 9306}, {8805, 9310, 9310}, {4995, 9314, 9314}, {6730, 9318, 9318}, {7457, 9328, 9328}, {2547, 9335, 9336}, {6298, 9340, 9343}, {9305, 9353, 9354}, {9269, 9358, 9358}, {6338, 9370, 9370}, {7289, 9376, 9379}, {5780, 9383, 9383}, {7607, 9387, 9387}, {2065, 9392, 9392}, {7238, 9396, 9396}, {8856, 9400, 9400}, {8069, 9412, 9413}, {611, 9420, 9420}, {7071, 9424, 9424}, {3089, 9430, 9431}, {7117, 9435, 9438}, {1976, 9445, 9445}, {6640, 9449, 9449}, {5488, 9453, 9453}, {8739, 9457, 9459}, {5958, 9466, 9466}, {7985, 9470, 9470}, {8735, 9475, 9475}, {5009, 9479, 9479}, {8073, 9483, 9484}, {2328, 9490, 9491}, {9250, 9495, 9495}, {4043, 9502, 9502}, {7712, 9506, 9506}, {9012, 9510, 9510}, {9028, 9514, 9515}, {2190, 9521, 9524}, {9029, 9528, 9528}, {9519, 9532, 9532}, {9495, 9536, 9536}, {8527, 9540, 9540}, {2137, 9550, 9550}, {8419, 9557, 9557}, {9383, 9561, 9562}, {8970, 9575, 9578}, {8911, 9582, 9582}, {7828, 9595, 9596}, {6180, 9600, 9600}, {8738, 9604, 9607}, {7540, 9611, 9612}, {9599, 9616, 9618}, {9187, 9623, 9623}, {9294, 9628, 9629}, {4536, 9639, 9639}, {3867, 9643, 9643}, {6305, 9648, 9648}, {1617, 9654, 9657}, {5762, 9666, 9666}, {8314, 9670, 9670}, {9666, 9674, 9675}, {9506, 9679, 9679}, {9669, 9685, 9686}, {9683, 9690, 9690}, {8763, 9697, 9698}, {7468, 9702, 9702}, {460, 9707, 9707}, {3115, 9712, 9712}, {9424, 9716, 9717}, {7359, 9721, 9724}, {7547, 9728, 9729}, {7151, 9733, 9738}, {7627, 9742, 9742}, {2822, 9747, 9747}, {8247, 9751, 9753}, {9550, 9758, 9758}, {7585, 9762, 9763}, {1002, 9767, 9767}, {7168, 9772, 9773}, {6941, 9777, 9780}, {9728, 9784, 9786}, {9770, 9792, 9796}, {6411, 9801, 9802}, {3689, 9806, 9808}, {9575, 9814, 9816}, {7025, 9820, 9821}, {2776, 9826, 9826}, {9806, 9830, 9830}, {9820, 9834, 9835}, {9800, 9839, 9847}, {9834, 9851, 9852}, {9829, 9856, 9862}, {1400, 9866, 9866}, {3197, 9870, 9871}, {9851, 9875, 9876}, {9742, 9883, 9884}, {3362, 9888, 9889}, {9883, 9893, 9893}, {5711, 9899, 9910}, {7806, 9915, 9915}, {9120, 9919, 9919}, {9715, 9925, 9934}, {2580, 9938, 9938}, {4907, 9942, 9944}, {6239, 9953, 9954}, {6961, 9963, 9963}, {5295, 9967, 9968}, {1915, 9972, 9973}, {3426, 9983, 9985}, {9875, 9994, 9995}, {6942, 9999, 9999}, {6621, 10005, 10005}, {7589, 10010, 10012}, {9286, 10020, 10020}, {838, 10024, 10024}, {9980, 10028, 10031}, {9994, 10035, 10041}, {2702, 10048, 10051}, {2621, 10059, 10059}, {10054, 10065, 10065}, {8612, 10073, 10074}, {7033, 10078, 10078}, {916, 10082, 10082}, {10035, 10086, 10087}, {8613, 10097, 10097}, {9919, 10107, 10108}, {6133, 10114, 10115}, {10059, 10119, 10119}, {10065, 10126, 10127}, {7732, 10131, 10131}, {7155, 10135, 10136}, {6728, 10140, 10140}, {6162, 10144, 10145}, {4724, 10150, 10150}, {1665, 10154, 10154}, {10126, 10163, 10163}, {9783, 10168, 10168}, {1715, 10172, 10173}, {7152, 10177, 10182}, {8760, 10187, 10187}, {7829, 10191, 10191}, {9679, 10196, 10196}, {9369, 10201, 10201}, {2928, 10206, 10208}, {6951, 10214, 10217}, {5633, 10221, 10221}, {7199, 10225, 10225}, {10118, 10230, 10231}, {9999, 10235, 10236}, {10045, 10240, 10249}, {5565, 10256, 10256}, {9866, 10261, 10261}, {10163, 10268, 10268}, {9869, 10272, 10272}, {9789, 10276, 10283}, {10235, 10287, 10288}, {10214, 10298, 10299}, {6971, 10303, 10303}, {3346, 10307, 10307}, {10185, 10311, 10312}, {9993, 10318, 10320}, {2779, 10332, 10334}, {1726, 10338, 10338}, {741, 10354, 10360}, {10230, 10372, 10373}, {10260, 10384, 10385}, {10131, 10389, 10398}, {6946, 10406, 10409}, {10158, 10413, 10420}, {10123, 10424, 10424}, {6157, 10428, 10429}, {4518, 10434, 10434}, {9893, 10438, 10438}, {9865, 10442, 10446}, {7558, 10454, 10454}, {10434, 10460, 10460}, {10064, 10466, 10468}, {2703, 10472, 10474}, {9751, 10478, 10479}, {6714, 10485, 10485}, {8020, 10490, 10490}, {10303, 10494, 10494}, {3521, 10499, 10500}, {9281, 10513, 10515}, {6028, 10519, 10523}, {9387, 10527, 10527}, {7614, 10531, 10531}, {3611, 10536, 10536}, {9162, 10540, 10540}, {10081, 10546, 10547}, {10034, 10560, 10562}, {6726, 10567, 10571}, {8237, 10575, 10575}, {10438, 10579, 10583}, {10140, 10587, 10587}, {5784, 10592, 10592}, {9819, 10597, 10600}, {10567, 10604, 10608}, {9335, 10613, 10613}, {8300, 10617, 10617}, {10575, 10621, 10621}, {9678, 10625, 10626}, {9962, 10632, 10633}, {10535, 10637, 10638}, {8199, 10642, 10642}, {10372, 10647, 10648}, {10637, 10656, 10657}, {10579, 10667, 10668}, {10465, 10677, 10680}, {6702, 10684, 10685}, {10073, 10691, 10692}, {4505, 10696, 10697}, {9042, 10701, 10701}, {6460, 10705, 10706}, {10010, 10714, 10716}, {10656, 10720, 10722}, {7282, 10727, 10729}, {2327, 10733, 10733}, {2491, 10740, 10741}, {10704, 10748, 10750}, {6465, 10754, 10754}, {10647, 10758, 10759}, {10424, 10763, 10763}, {10748, 10776, 10776}, {10546, 10780, 10781}, {10758, 10785, 10786}, {10287, 10790, 10797}, {10785, 10801, 10807}, {10240, 10811, 10826}, {9509, 10830, 10830}, {2579, 10836, 10838}, {9801, 10843, 10845}, {7555, 10849, 10850}, {10776, 10860, 10865}, {8023, 10869, 10869}, {10046, 10876, 10884}, {10253, 10888, 10892}, {9941, 10897, 10897}, {7898, 10901, 10905}, {6725, 10909, 10913}, {10757, 10921, 10923}, {10160, 10931, 10931}, {10916, 10935, 10942}, {10261, 10946, 10946}, {10318, 10952, 10954}, {5911, 10959, 10961}, {10801, 10965, 10966}, {10946, 10970, 10977}, {10592, 10982, 10984}, {9913, 10988, 10990}, {8510, 10994, 10996}, {9419, 11000, 11001}, {6765, 11006, 11007}, {10725, 11011, 11011}, {5537, 11017, 11019}, {9208, 11024, 11025}, {5850, 11030, 11030}, {9610, 11034, 11036}, {8846, 11041, 11047}, {9697, 11051, 11051}, {1622, 11055, 11058}, {2370, 11062, 11062}, {8393, 11067, 11067}, {9756, 11071, 11071}, {10172, 11076, 11076}, {27, 11081, 11081}, {7357, 11087, 11092}, {8151, 11104, 11106}, {6115, 11110, 11110}, {10667, 11114, 11115}, {11099, 11121, 11123}, {10705, 11127, 11127}, {8938, 11131, 11131}, {11114, 11135, 11136}, {1390, 11140, 11141}, {10964, 11146, 11148}, {11140, 11152, 11155}, {9813, 11159, 11166}, {624, 11171, 11172}, {3118, 11177, 11179}, {11029, 11184, 11186}, {10186, 11190, 11190}, {10306, 11196, 11196}, {8665, 11201, 11201}, {7382, 11205, 11205}, {1100, 11210, 11210}, {2337, 11216, 11217}, {1609, 11221, 11223}, {5763, 11228, 11229}, {5220, 11233, 11233}, {11061, 11241, 11241}, {10617, 11246, 11246}, {11190, 11250, 11251}, {10144, 11255, 11256}, {11232, 11260, 11260}, {857, 11264, 11265}, {10994, 11269, 11271}, {3879, 11280, 11281}, {11184, 11287, 11289}, {9611, 11293, 11295}, {11250, 11299, 11299}, {4495, 11304, 11304}, {7574, 11308, 11309}, {9814, 11315, 11317}, {1713, 11321, 11324}, {1905, 11328, 11328}, {8745, 11335, 11340}, {8883, 11351, 11351}, {8119, 11358, 11358}, {1842, 11363, 11364}, {11237, 11368, 11368}, {8814, 11373, 11374}, {5684, 11378, 11378}, {11011, 11382, 11382}, {6520, 11389, 11389}, {11183, 11393, 11396}, {1790, 11404, 11404}, {9536, 11408, 11408}, {11298, 11418, 11419}, {3929, 11425, 11425}, {5588, 11429, 11429}, {8476, 11436, 11436}, {4096, 11440, 11442}, {11084, 11446, 11454}, {10603, 11458, 11463}, {7332, 11472, 11474}, {7611, 11483, 11486}, {4836, 11490, 11491}, {10024, 11495, 11495}, {4917, 11501, 11506}, {6486, 11510, 11512}, {11269, 11516, 11518}, {3603, 11522, 11525}, {11126, 11535, 11535}, {11418, 11539, 11541}, {11408, 11545, 11545}, {9021, 11549, 11552}, {6745, 11557, 11557}, {5118, 11561, 11564}, {7590, 11568, 11569}, {4426, 11573, 11578}, {9790, 11582, 11583}, {6447, 11587, 11587}, {10229, 11591, 11594}, {10457, 11598, 11598}, {10168, 11604, 11604}, {10543, 11608, 11608}, {7404, 11612, 11612}, {11127, 11616, 11616}, {3337, 11620, 11620}, {11501, 11624, 11628}, {4543, 11633, 11635}, {8449, 11642, 11642}, {4943, 11646, 11648}, {10526, 11652, 11654}, {11620, 11659, 11659}, {8927, 11664, 11669}, {532, 11673, 11673}, {10513, 11677, 11679}, {10428, 11683, 11683}, {10999, 11689, 11690}, {9469, 11695, 11695}, {3606, 11699, 11699}, {9560, 11708, 11709}, {1564, 11714, 11714}, {10527, 11718, 11718}, {3071, 11723, 11726}, {11590, 11731, 11732}, {6605, 11737, 11737}, {11624, 11741, 11745}, {7822, 11749, 11752}, {5269, 11757, 11758}, {1339, 11767, 11767}, {1363, 11771, 11773}, {3704, 11777, 11777}, {10952, 11781, 11783}, {6764, 11793, 11795}, {8675, 11800, 11800}, {9963, 11804, 11804}, {11573, 11808, 11809}, {9548, 11813, 11813}, {11591, 11817, 11818}, {11446, 11822, 11822}, {9224, 11828, 11828}, {3158, 11836, 11836}, {10830, 11840, 11840}, {7234, 11846, 11846}, {11299, 11850, 11850}, {11544, 11854, 11855}, {11498, 11859, 11859}, {10993, 11865, 11868}, {9720, 11872, 11878}, {10489, 11882, 11890}, {11712, 11898, 11904}, {11516, 11908, 11910}, {11568, 11914, 11915}, {10177, 11919, 11924}, {11363, 11928, 11929}, {10494, 11933, 11933}, {9870, 11937, 11938}, {9427, 11942, 11942}, {11481, 11949, 11949}, {6030, 11955, 11957}, {11718, 11961, 11961}, {10531, 11965, 11983}, {5126, 11987, 11987}, {7515, 11991, 11991}, {10646, 11996, 11997}, {2947, 12001, 12001}, {9582, 12009, 12010}, {6202, 12017, 12018}, {11714, 12022, 12022}, {9235, 12033, 12037}, {9721, 12041, 12044}, {11932, 12051, 12052}, {12040, 12056, 12056}, {12051, 12060, 12060}, {11601, 12066, 12066}, {8426, 12070, 12070}, {4053, 12077, 12077}, {4262, 12081, 12081}, {9761, 12086, 12088}, {11582, 12092, 12093}, {10965, 12097, 12098}, {11803, 12103, 12104}, {11933, 12108, 12109}, {10688, 12117, 12117}, {12107, 12125, 12126}, {6774, 12130, 12132}, {6286, 12137, 12137}, {9543, 12141, 12141}, {12097, 12145, 12146}, {10790, 12150, 12150}, {10125, 12154, 12156}, {12125, 12164, 12164}, {12064, 12168, 12172}, {10811, 12178, 12188}, {12092, 12192, 12193}, {10058, 12197, 12198}, {11611, 12211, 12212}, {3459, 12216, 12216}, {10291, 12225, 12228}, {12191, 12232, 12234}, {12145, 12238, 12238}, {12001, 12242, 12250}, {3840, 12255, 12255}, {12216, 12259, 12259}, {674, 12272, 12272}, {12141, 12276, 12276}, {10766, 12280, 12280}, {11545, 12284, 12284}, {6496, 12290, 12290}, {11381, 12294, 12295}, {603, 12302, 12303}, {12276, 12308, 12308}, {11850, 12313, 12314}, {565, 12319, 12319}, {9351, 12324, 12324}, {11822, 12328, 12328}, {2691, 12333, 12334}, {11840, 12338, 12338}, {11070, 12343, 12343}, {9510, 12347, 12347}, {11024, 12352, 12353}, {7173, 12359, 12359}, {517, 12363, 12363}, {6311, 12367, 12368}, {11367, 12372, 12373}, {12008, 12377, 12377}, {11372, 12382, 12384}, {11358, 12391, 12392}, {11382, 12396, 12396}, {6882, 12400, 12401}, {11246, 12405, 12405}, {8359, 12409, 12412}, {10154, 12418, 12418}, {12016, 12425, 12426}, {8972, 12434, 12435}, {10478, 12439, 12440}, {12395, 12449, 12449}, {11612, 12454, 12454}, {12347, 12458, 12458}, {10700, 12466, 12467}, {3637, 12471, 12476}, {1042, 12480, 12481}, {6747, 12488, 12488}, {12396, 12492, 12493}, {9420, 12497, 12497}, {11285, 12501, 12510}, {4470, 12515, 12515}, {9374, 12519, 12519}, {11293, 12528, 12528}, {2058, 12534, 12535}, {6521, 12539, 12539}, {12492, 12543, 12543}, {3043, 12547, 12547}, {2982, 12551, 12553}, {11030, 12557, 12563}, {7636, 12568, 12568}, {9639, 12572, 12572}, {12543, 12576, 12576}, {5989, 12580, 12583}, {11051, 12587, 12587}, {1061, 12592, 12594}, {12313, 12599, 12601}, {11846, 12605, 12605}, {12576, 12609, 12609}, {11040, 12618, 12625}, {12479, 12629, 12629}, {6903, 12633, 12633}, {12322, 12639, 12639}, {12253, 12643, 12645}, {5594, 12651, 12651}, {12522, 12655, 12655}, {11703, 12659, 12659}, {1377, 12665, 12665}, {8022, 12669, 12669}, {12280, 12674, 12674}, {9023, 12680, 12681}, {12328, 12685, 12685}, {3085, 12689, 12693}, {4700, 12698, 12698}, {10224, 12702, 12702}, {8781, 12706, 12706}, {1651, 12710, 12710}, {12458, 12714, 12714}, {12005, 12718, 12721}, {11908, 12725, 12726}, {8202, 12733, 12733}, {11708, 12739, 12740}, {12599, 12744, 12745}, {12284, 12749, 12749}, {5285, 12756, 12756}, {12055, 12775, 12777}, {6919, 12782, 12782}, {12242, 12786, 12786}, {12009, 12790, 12790}, {9628, 12794, 12796}, {11354, 12801, 12802}, {10225, 12806, 12807}, {579, 12813, 12813}, {8935, 12817, 12822}, {8753, 12827, 12829}, {11006, 12835, 12835}, {858, 12841, 12845}, {476, 12849, 12849}, {7667, 12854, 12854}, {12760, 12860, 12871}, {11677, 12875, 12877}, {12714, 12881, 12881}, {12731, 12885, 12890}, {7108, 12894, 12896}, {1165, 12900, 12900}, {4021, 12906, 12906}, {10829, 12910, 12911}, {12331, 12915, 12915}, {8887, 12919, 12921}, {11639, 12925, 12925}, {7964, 12929, 12929}, {12528, 12937, 12937}, {8148, 12941, 12941}, {12770, 12948, 12950}, {12609, 12954, 12954}, {12685, 12958, 12958}, {2803, 12962, 12962}, {9561, 12966, 12966}, {6671, 12972, 12973}, {12056, 12977, 12977}, {6380, 12981, 12981}, {12048, 12985, 12985}, {11961, 12989, 12993}, {3368, 12997, 12999}, {6634, 13004, 13004}, {6775, 13009, 13010}, {12136, 13014, 13019}, {10341, 13023, 13023}, {13002, 13027, 13027}, {10587, 13031, 13031}, {10307, 13035, 13035}, {12736, 13039, 13039}, {12744, 13043, 13044}, {6175, 13048, 13048}, {9702, 13053, 13054}, {662, 13059, 13061}, {12718, 13065, 13068}, {12893, 13072, 13075}, {8299, 13086, 13091}, {12604, 13095, 13096}, {12848, 13100, 13101}, {12749, 13105, 13105}, {12526, 13109, 13114}, {9173, 13122, 13122}, {12769, 13128, 13128}, {13038, 13132, 13132}, {12725, 13136, 13137}, {12639, 13146, 13146}, {9711, 13150, 13151}, {12137, 13155, 13155}, {13039, 13159, 13159}, {4681, 13163, 13164}, {12954, 13168, 13168}, {13158, 13175, 13176}, {13105, 13180, 13180}, {10754, 13184, 13184}, {13167, 13188, 13188}, {12658, 13192, 13192}, {4294, 13199, 13200}, {11682, 13204, 13205}, {11695, 13209, 13209}, {11076, 13214, 13214}, {12232, 13218, 13218}, {9399, 13223, 13224}, {12880, 13228, 13229}, {13048, 13234, 13234}, {9701, 13238, 13239}, {13209, 13243, 13243}, {3658, 13248, 13248}, {3698, 13252, 13254}, {12237, 13260, 13260}, {8872, 13266, 13266}, {12957, 13272, 13273}, {1393, 13281, 13281}, {2013, 13285, 13288}, {4244, 13296, 13299}, {9428, 13303, 13303}, {12702, 13307, 13307}, {13078, 13311, 13311}, {6071, 13315, 13315}, {3061, 13319, 13319}, {2051, 13324, 13324}, {11560, 13328, 13331}, {6584, 13336, 13336}, {8482, 13340, 13340}, {5331, 13344, 13344}, {4171, 13348, 13348}, {8501, 13352, 13352}, {9219, 13356, 13356}, {9473, 13360, 13363}, {12881, 13367, 13367}, {13065, 13371, 13375}, {2979, 13379, 13384}, {1518, 13388, 13388}, {11177, 13392, 13392}, {9457, 13398, 13398}, {12293, 13407, 13410}, {3697, 13414, 13417}, {10338, 13425, 13425}, {13367, 13429, 13429}, {11074, 13433, 13437}, {4201, 13441, 13443}, {1812, 13447, 13448}, {13360, 13452, 13456}, {13188, 13463, 13463}, {9732, 13470, 13470}, {11332, 13477, 13477}, {9918, 13487, 13487}, {6337, 13497, 13497}, {13429, 13501, 13501}, {11413, 13505, 13505}, {4685, 13512, 13513}, {13136, 13517, 13519}, {7416, 13528, 13530}, {12929, 13534, 13534}, {11110, 13539, 13539}, {11521, 13543, 13543}, {12825, 13553, 13553}, {13447, 13557, 13558}, {12299, 13562, 13563}, {9003, 13570, 13570}, {12500, 13577, 13577}, {13501, 13581, 13581}, {9392, 13586, 13586}, {12454, 13590, 13590}, {6189, 13595, 13595}, {13053, 13599, 13599}, {11881, 13604, 13604}, {13159, 13608, 13608}, {4894, 13612, 13612}, {13221, 13621, 13621}, {8950, 13625, 13625}, {13533, 13629, 13629}, {9633, 13633, 13633}, {7892, 13637, 13639}, {13581, 13643, 13643}, {13616, 13647, 13649}, {12794, 13653, 13654}, {8919, 13659, 13659}, {9674, 13663, 13663}, {13577, 13668, 13668}, {12966, 13672, 13672}, {12659, 13676, 13683}, {6124, 13688, 13688}, {9225, 13693, 13695}, {11833, 13702, 13702}, {12904, 13709, 13717}, {13647, 13721, 13722}, {11687, 13726, 13727}, {12434, 13731, 13732}, {12689, 13736, 13742}, {13168, 13746, 13746}, {6151, 13751, 13752}, {11821, 13756, 13757}, {6467, 13764, 13764}, {5730, 13769, 13769}, {5136, 13780, 13780}, {724, 13784, 13785}, {13517, 13789, 13791}, {640, 13795, 13796}, {7721, 13800, 13802}, {11121, 13806, 13807}, {5791, 13811, 13815}, {12894, 13819, 13819}, {11100, 13824, 13824}, {7011, 13830, 13830}, {7129, 13834, 13837}, {13833, 13841, 13841}, {11276, 13847, 13847}, {13621, 13853, 13853}, {13589, 13862, 13863}, {12989, 13867, 13867}, {12789, 13871, 13871}, {1239, 13875, 13875}, {4675, 13879, 13881}, {4686, 13885, 13885}, {707, 13889, 13889}, {5449, 13897, 13898}, {13867, 13902, 13903}, {10613, 13908, 13908}, {13789, 13912, 13914}, {4451, 13918, 13919}, {9200, 13924, 13924}, {2011, 13930, 13930}, {11433, 13934, 13936}, {4695, 13942, 13943}, {9435, 13948, 13951}, {13688, 13955, 13957}, {11694, 13961, 13962}, {5712, 13966, 13966}, {5991, 13970, 13972}, {13477, 13976, 13976}, {10213, 13987, 13987}, {11839, 13991, 13993}, {12272, 13997, 13997}, {6206, 14001, 14001}, {13179, 14006, 14007}, {2939, 14011, 14011}, {12972, 14016, 14017}, {13918, 14021, 14022}, {7436, 14026, 14027}, {7678, 14032, 14034}, {13586, 14040, 14040}, {13347, 14044, 14044}, {13109, 14048, 14051}, {9244, 14055, 14057}, {13315, 14061, 14061}, {13276, 14067, 14067}, {11435, 14073, 14074}, {13853, 14078, 14078}, {13452, 14082, 14082}, {14044, 14087, 14087}, {4440, 14091, 14095}, {4479, 14100, 14103}, {9395, 14107, 14109}, {6834, 14119, 14119}, {10458, 14123, 14124}, {1429, 14129, 14129}, {8443, 14135, 14135}, {10365, 14140, 14140}, {5267, 14145, 14145}, {11834, 14151, 14153}, } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/snappy.go000066400000000000000000000064261272247357100233050ustar00rootroot00000000000000// Copyright 2011 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Package snappy implements the snappy block-based compression format. // It aims for very high speeds and reasonable compression. // // The C++ snappy implementation is at https://github.com/google/snappy package snappy // import "github.com/golang/snappy" import ( "hash/crc32" ) /* Each encoded block begins with the varint-encoded length of the decoded data, followed by a sequence of chunks. Chunks begin and end on byte boundaries. The first byte of each chunk is broken into its 2 least and 6 most significant bits called l and m: l ranges in [0, 4) and m ranges in [0, 64). l is the chunk tag. Zero means a literal tag. All other values mean a copy tag. For literal tags: - If m < 60, the next 1 + m bytes are literal bytes. - Otherwise, let n be the little-endian unsigned integer denoted by the next m - 59 bytes. The next 1 + n bytes after that are literal bytes. For copy tags, length bytes are copied from offset bytes ago, in the style of Lempel-Ziv compression algorithms. In particular: - For l == 1, the offset ranges in [0, 1<<11) and the length in [4, 12). The length is 4 + the low 3 bits of m. The high 3 bits of m form bits 8-10 of the offset. The next byte is bits 0-7 of the offset. - For l == 2, the offset ranges in [0, 1<<16) and the length in [1, 65). The length is 1 + m. The offset is the little-endian unsigned integer denoted by the next 2 bytes. - For l == 3, this tag is a legacy format that is no longer issued by most encoders. Nonetheless, the offset ranges in [0, 1<<32) and the length in [1, 65). The length is 1 + m. The offset is the little-endian unsigned integer denoted by the next 4 bytes. */ const ( tagLiteral = 0x00 tagCopy1 = 0x01 tagCopy2 = 0x02 tagCopy4 = 0x03 ) const ( checksumSize = 4 chunkHeaderSize = 4 magicChunk = "\xff\x06\x00\x00" + magicBody magicBody = "sNaPpY" // maxBlockSize is the maximum size of the input to encodeBlock. It is not // part of the wire format per se, but some parts of the encoder assume // that an offset fits into a uint16. // // Also, for the framing format (Writer type instead of Encode function), // https://github.com/google/snappy/blob/master/framing_format.txt says // that "the uncompressed data in a chunk must be no longer than 65536 // bytes". maxBlockSize = 65536 // maxEncodedLenOfMaxBlockSize equals MaxEncodedLen(maxBlockSize), but is // hard coded to be a const instead of a variable, so that obufLen can also // be a const. Their equivalence is confirmed by // TestMaxEncodedLenOfMaxBlockSize. maxEncodedLenOfMaxBlockSize = 76490 obufHeaderLen = len(magicChunk) + checksumSize + chunkHeaderSize obufLen = obufHeaderLen + maxEncodedLenOfMaxBlockSize ) const ( chunkTypeCompressedData = 0x00 chunkTypeUncompressedData = 0x01 chunkTypePadding = 0xfe chunkTypeStreamIdentifier = 0xff ) var crcTable = crc32.MakeTable(crc32.Castagnoli) // crc implements the checksum specified in section 3 of // https://github.com/google/snappy/blob/master/framing_format.txt func crc(b []byte) uint32 { c := crc32.Update(0, crcTable, b) return uint32(c>>15|c<<17) + 0xa282ead8 } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/snappy_test.go000066400000000000000000001152621272247357100243430ustar00rootroot00000000000000// Copyright 2011 The Snappy-Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. package snappy import ( "bytes" "encoding/binary" "flag" "fmt" "io" "io/ioutil" "math/rand" "net/http" "os" "os/exec" "path/filepath" "runtime" "strings" "testing" ) var ( download = flag.Bool("download", false, "If true, download any missing files before running benchmarks") testdataDir = flag.String("testdataDir", "testdata", "Directory containing the test data") benchdataDir = flag.String("benchdataDir", "testdata/bench", "Directory containing the benchmark data") ) // goEncoderShouldMatchCppEncoder is whether to test that the algorithm used by // Go's encoder matches byte-for-byte what the C++ snappy encoder produces, on // this GOARCH. There is more than one valid encoding of any given input, and // there is more than one good algorithm along the frontier of trading off // throughput for output size. Nonetheless, we presume that the C++ encoder's // algorithm is a good one and has been tested on a wide range of inputs, so // matching that exactly should mean that the Go encoder's algorithm is also // good, without needing to gather our own corpus of test data. // // The exact algorithm used by the C++ code is potentially endian dependent, as // it puns a byte pointer to a uint32 pointer to load, hash and compare 4 bytes // at a time. The Go implementation is endian agnostic, in that its output is // the same (as little-endian C++ code), regardless of the CPU's endianness. // // Thus, when comparing Go's output to C++ output generated beforehand, such as // the "testdata/pi.txt.rawsnappy" file generated by C++ code on a little- // endian system, we can run that test regardless of the runtime.GOARCH value. // // When comparing Go's output to dynamically generated C++ output, i.e. the // result of fork/exec'ing a C++ program, we can run that test only on // little-endian systems, because the C++ output might be different on // big-endian systems. The runtime package doesn't export endianness per se, // but we can restrict this match-C++ test to common little-endian systems. const goEncoderShouldMatchCppEncoder = runtime.GOARCH == "386" || runtime.GOARCH == "amd64" || runtime.GOARCH == "arm" func TestMaxEncodedLenOfMaxBlockSize(t *testing.T) { got := maxEncodedLenOfMaxBlockSize want := MaxEncodedLen(maxBlockSize) if got != want { t.Fatalf("got %d, want %d", got, want) } } func cmp(a, b []byte) error { if bytes.Equal(a, b) { return nil } if len(a) != len(b) { return fmt.Errorf("got %d bytes, want %d", len(a), len(b)) } for i := range a { if a[i] != b[i] { return fmt.Errorf("byte #%d: got 0x%02x, want 0x%02x", i, a[i], b[i]) } } return nil } func roundtrip(b, ebuf, dbuf []byte) error { d, err := Decode(dbuf, Encode(ebuf, b)) if err != nil { return fmt.Errorf("decoding error: %v", err) } if err := cmp(d, b); err != nil { return fmt.Errorf("roundtrip mismatch: %v", err) } return nil } func TestEmpty(t *testing.T) { if err := roundtrip(nil, nil, nil); err != nil { t.Fatal(err) } } func TestSmallCopy(t *testing.T) { for _, ebuf := range [][]byte{nil, make([]byte, 20), make([]byte, 64)} { for _, dbuf := range [][]byte{nil, make([]byte, 20), make([]byte, 64)} { for i := 0; i < 32; i++ { s := "aaaa" + strings.Repeat("b", i) + "aaaabbbb" if err := roundtrip([]byte(s), ebuf, dbuf); err != nil { t.Errorf("len(ebuf)=%d, len(dbuf)=%d, i=%d: %v", len(ebuf), len(dbuf), i, err) } } } } } func TestSmallRand(t *testing.T) { rng := rand.New(rand.NewSource(1)) for n := 1; n < 20000; n += 23 { b := make([]byte, n) for i := range b { b[i] = uint8(rng.Intn(256)) } if err := roundtrip(b, nil, nil); err != nil { t.Fatal(err) } } } func TestSmallRegular(t *testing.T) { for n := 1; n < 20000; n += 23 { b := make([]byte, n) for i := range b { b[i] = uint8(i%10 + 'a') } if err := roundtrip(b, nil, nil); err != nil { t.Fatal(err) } } } func TestInvalidVarint(t *testing.T) { testCases := []struct { desc string input string }{{ "invalid varint, final byte has continuation bit set", "\xff", }, { "invalid varint, value overflows uint64", "\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\x00", }, { // https://github.com/google/snappy/blob/master/format_description.txt // says that "the stream starts with the uncompressed length [as a // varint] (up to a maximum of 2^32 - 1)". "valid varint (as uint64), but value overflows uint32", "\x80\x80\x80\x80\x10", }} for _, tc := range testCases { input := []byte(tc.input) if _, err := DecodedLen(input); err != ErrCorrupt { t.Errorf("%s: DecodedLen: got %v, want ErrCorrupt", tc.desc, err) } if _, err := Decode(nil, input); err != ErrCorrupt { t.Errorf("%s: Decode: got %v, want ErrCorrupt", tc.desc, err) } } } func TestDecode(t *testing.T) { lit40Bytes := make([]byte, 40) for i := range lit40Bytes { lit40Bytes[i] = byte(i) } lit40 := string(lit40Bytes) testCases := []struct { desc string input string want string wantErr error }{{ `decodedLen=0; valid input`, "\x00", "", nil, }, { `decodedLen=3; tagLiteral, 0-byte length; length=3; valid input`, "\x03" + "\x08\xff\xff\xff", "\xff\xff\xff", nil, }, { `decodedLen=2; tagLiteral, 0-byte length; length=3; not enough dst bytes`, "\x02" + "\x08\xff\xff\xff", "", ErrCorrupt, }, { `decodedLen=3; tagLiteral, 0-byte length; length=3; not enough src bytes`, "\x03" + "\x08\xff\xff", "", ErrCorrupt, }, { `decodedLen=40; tagLiteral, 0-byte length; length=40; valid input`, "\x28" + "\x9c" + lit40, lit40, nil, }, { `decodedLen=1; tagLiteral, 1-byte length; not enough length bytes`, "\x01" + "\xf0", "", ErrCorrupt, }, { `decodedLen=3; tagLiteral, 1-byte length; length=3; valid input`, "\x03" + "\xf0\x02\xff\xff\xff", "\xff\xff\xff", nil, }, { `decodedLen=1; tagLiteral, 2-byte length; not enough length bytes`, "\x01" + "\xf4\x00", "", ErrCorrupt, }, { `decodedLen=3; tagLiteral, 2-byte length; length=3; valid input`, "\x03" + "\xf4\x02\x00\xff\xff\xff", "\xff\xff\xff", nil, }, { `decodedLen=1; tagLiteral, 3-byte length; not enough length bytes`, "\x01" + "\xf8\x00\x00", "", ErrCorrupt, }, { `decodedLen=3; tagLiteral, 3-byte length; length=3; valid input`, "\x03" + "\xf8\x02\x00\x00\xff\xff\xff", "\xff\xff\xff", nil, }, { `decodedLen=1; tagLiteral, 4-byte length; not enough length bytes`, "\x01" + "\xfc\x00\x00\x00", "", ErrCorrupt, }, { `decodedLen=1; tagLiteral, 4-byte length; length=3; not enough dst bytes`, "\x01" + "\xfc\x02\x00\x00\x00\xff\xff\xff", "", ErrCorrupt, }, { `decodedLen=4; tagLiteral, 4-byte length; length=3; not enough src bytes`, "\x04" + "\xfc\x02\x00\x00\x00\xff", "", ErrCorrupt, }, { `decodedLen=3; tagLiteral, 4-byte length; length=3; valid input`, "\x03" + "\xfc\x02\x00\x00\x00\xff\xff\xff", "\xff\xff\xff", nil, }, { `decodedLen=4; tagCopy1, 1 extra length|offset byte; not enough extra bytes`, "\x04" + "\x01", "", ErrCorrupt, }, { `decodedLen=4; tagCopy2, 2 extra length|offset bytes; not enough extra bytes`, "\x04" + "\x02\x00", "", ErrCorrupt, }, { `decodedLen=4; tagCopy4, 4 extra length|offset bytes; not enough extra bytes`, "\x04" + "\x03\x00\x00\x00", "", ErrCorrupt, }, { `decodedLen=4; tagLiteral (4 bytes "abcd"); valid input`, "\x04" + "\x0cabcd", "abcd", nil, }, { `decodedLen=13; tagLiteral (4 bytes "abcd"); tagCopy1; length=9 offset=4; valid input`, "\x0d" + "\x0cabcd" + "\x15\x04", "abcdabcdabcda", nil, }, { `decodedLen=8; tagLiteral (4 bytes "abcd"); tagCopy1; length=4 offset=4; valid input`, "\x08" + "\x0cabcd" + "\x01\x04", "abcdabcd", nil, }, { `decodedLen=8; tagLiteral (4 bytes "abcd"); tagCopy1; length=4 offset=2; valid input`, "\x08" + "\x0cabcd" + "\x01\x02", "abcdcdcd", nil, }, { `decodedLen=8; tagLiteral (4 bytes "abcd"); tagCopy1; length=4 offset=1; valid input`, "\x08" + "\x0cabcd" + "\x01\x01", "abcddddd", nil, }, { `decodedLen=8; tagLiteral (4 bytes "abcd"); tagCopy1; length=4 offset=0; zero offset`, "\x08" + "\x0cabcd" + "\x01\x00", "", ErrCorrupt, }, { `decodedLen=9; tagLiteral (4 bytes "abcd"); tagCopy1; length=4 offset=4; inconsistent dLen`, "\x09" + "\x0cabcd" + "\x01\x04", "", ErrCorrupt, }, { `decodedLen=8; tagLiteral (4 bytes "abcd"); tagCopy1; length=4 offset=5; offset too large`, "\x08" + "\x0cabcd" + "\x01\x05", "", ErrCorrupt, }, { `decodedLen=7; tagLiteral (4 bytes "abcd"); tagCopy1; length=4 offset=4; length too large`, "\x07" + "\x0cabcd" + "\x01\x04", "", ErrCorrupt, }, { `decodedLen=6; tagLiteral (4 bytes "abcd"); tagCopy2; length=2 offset=3; valid input`, "\x06" + "\x0cabcd" + "\x06\x03\x00", "abcdbc", nil, }, { `decodedLen=6; tagLiteral (4 bytes "abcd"); tagCopy4; length=2 offset=3; valid input`, "\x06" + "\x0cabcd" + "\x07\x03\x00\x00\x00", "abcdbc", nil, }} const ( // notPresentXxx defines a range of byte values [0xa0, 0xc5) that are // not present in either the input or the output. It is written to dBuf // to check that Decode does not write bytes past the end of // dBuf[:dLen]. // // The magic number 37 was chosen because it is prime. A more 'natural' // number like 32 might lead to a false negative if, for example, a // byte was incorrectly copied 4*8 bytes later. notPresentBase = 0xa0 notPresentLen = 37 ) var dBuf [100]byte loop: for i, tc := range testCases { input := []byte(tc.input) for _, x := range input { if notPresentBase <= x && x < notPresentBase+notPresentLen { t.Errorf("#%d (%s): input shouldn't contain %#02x\ninput: % x", i, tc.desc, x, input) continue loop } } dLen, n := binary.Uvarint(input) if n <= 0 { t.Errorf("#%d (%s): invalid varint-encoded dLen", i, tc.desc) continue } if dLen > uint64(len(dBuf)) { t.Errorf("#%d (%s): dLen %d is too large", i, tc.desc, dLen) continue } for j := range dBuf { dBuf[j] = byte(notPresentBase + j%notPresentLen) } g, gotErr := Decode(dBuf[:], input) if got := string(g); got != tc.want || gotErr != tc.wantErr { t.Errorf("#%d (%s):\ngot %q, %v\nwant %q, %v", i, tc.desc, got, gotErr, tc.want, tc.wantErr) continue } for j, x := range dBuf { if uint64(j) < dLen { continue } if w := byte(notPresentBase + j%notPresentLen); x != w { t.Errorf("#%d (%s): Decode overrun: dBuf[%d] was modified: got %#02x, want %#02x\ndBuf: % x", i, tc.desc, j, x, w, dBuf) continue loop } } } } func TestDecodeCopy4(t *testing.T) { dots := strings.Repeat(".", 65536) input := strings.Join([]string{ "\x89\x80\x04", // decodedLen = 65545. "\x0cpqrs", // 4-byte literal "pqrs". "\xf4\xff\xff" + dots, // 65536-byte literal dots. "\x13\x04\x00\x01\x00", // tagCopy4; length=5 offset=65540. }, "") gotBytes, err := Decode(nil, []byte(input)) if err != nil { t.Fatal(err) } got := string(gotBytes) want := "pqrs" + dots + "pqrs." if len(got) != len(want) { t.Fatalf("got %d bytes, want %d", len(got), len(want)) } if got != want { for i := 0; i < len(got); i++ { if g, w := got[i], want[i]; g != w { t.Fatalf("byte #%d: got %#02x, want %#02x", i, g, w) } } } } // TestDecodeLengthOffset tests decoding an encoding of the form literal + // copy-length-offset + literal. For example: "abcdefghijkl" + "efghij" + "AB". func TestDecodeLengthOffset(t *testing.T) { const ( prefix = "abcdefghijklmnopqr" suffix = "ABCDEFGHIJKLMNOPQR" // notPresentXxx defines a range of byte values [0xa0, 0xc5) that are // not present in either the input or the output. It is written to // gotBuf to check that Decode does not write bytes past the end of // gotBuf[:totalLen]. // // The magic number 37 was chosen because it is prime. A more 'natural' // number like 32 might lead to a false negative if, for example, a // byte was incorrectly copied 4*8 bytes later. notPresentBase = 0xa0 notPresentLen = 37 ) var gotBuf, wantBuf, inputBuf [128]byte for length := 1; length <= 18; length++ { for offset := 1; offset <= 18; offset++ { loop: for suffixLen := 0; suffixLen <= 18; suffixLen++ { totalLen := len(prefix) + length + suffixLen inputLen := binary.PutUvarint(inputBuf[:], uint64(totalLen)) inputBuf[inputLen] = tagLiteral + 4*byte(len(prefix)-1) inputLen++ inputLen += copy(inputBuf[inputLen:], prefix) inputBuf[inputLen+0] = tagCopy2 + 4*byte(length-1) inputBuf[inputLen+1] = byte(offset) inputBuf[inputLen+2] = 0x00 inputLen += 3 if suffixLen > 0 { inputBuf[inputLen] = tagLiteral + 4*byte(suffixLen-1) inputLen++ inputLen += copy(inputBuf[inputLen:], suffix[:suffixLen]) } input := inputBuf[:inputLen] for i := range gotBuf { gotBuf[i] = byte(notPresentBase + i%notPresentLen) } got, err := Decode(gotBuf[:], input) if err != nil { t.Errorf("length=%d, offset=%d; suffixLen=%d: %v", length, offset, suffixLen, err) continue } wantLen := 0 wantLen += copy(wantBuf[wantLen:], prefix) for i := 0; i < length; i++ { wantBuf[wantLen] = wantBuf[wantLen-offset] wantLen++ } wantLen += copy(wantBuf[wantLen:], suffix[:suffixLen]) want := wantBuf[:wantLen] for _, x := range input { if notPresentBase <= x && x < notPresentBase+notPresentLen { t.Errorf("length=%d, offset=%d; suffixLen=%d: input shouldn't contain %#02x\ninput: % x", length, offset, suffixLen, x, input) continue loop } } for i, x := range gotBuf { if i < totalLen { continue } if w := byte(notPresentBase + i%notPresentLen); x != w { t.Errorf("length=%d, offset=%d; suffixLen=%d; totalLen=%d: "+ "Decode overrun: gotBuf[%d] was modified: got %#02x, want %#02x\ngotBuf: % x", length, offset, suffixLen, totalLen, i, x, w, gotBuf) continue loop } } for _, x := range want { if notPresentBase <= x && x < notPresentBase+notPresentLen { t.Errorf("length=%d, offset=%d; suffixLen=%d: want shouldn't contain %#02x\nwant: % x", length, offset, suffixLen, x, want) continue loop } } if !bytes.Equal(got, want) { t.Errorf("length=%d, offset=%d; suffixLen=%d:\ninput % x\ngot % x\nwant % x", length, offset, suffixLen, input, got, want) continue } } } } } const ( goldenText = "Mark.Twain-Tom.Sawyer.txt" goldenCompressed = goldenText + ".rawsnappy" ) func TestDecodeGoldenInput(t *testing.T) { tDir := filepath.FromSlash(*testdataDir) src, err := ioutil.ReadFile(filepath.Join(tDir, goldenCompressed)) if err != nil { t.Fatalf("ReadFile: %v", err) } got, err := Decode(nil, src) if err != nil { t.Fatalf("Decode: %v", err) } want, err := ioutil.ReadFile(filepath.Join(tDir, goldenText)) if err != nil { t.Fatalf("ReadFile: %v", err) } if err := cmp(got, want); err != nil { t.Fatal(err) } } func TestEncodeGoldenInput(t *testing.T) { tDir := filepath.FromSlash(*testdataDir) src, err := ioutil.ReadFile(filepath.Join(tDir, goldenText)) if err != nil { t.Fatalf("ReadFile: %v", err) } got := Encode(nil, src) want, err := ioutil.ReadFile(filepath.Join(tDir, goldenCompressed)) if err != nil { t.Fatalf("ReadFile: %v", err) } if err := cmp(got, want); err != nil { t.Fatal(err) } } func TestExtendMatchGoldenInput(t *testing.T) { tDir := filepath.FromSlash(*testdataDir) src, err := ioutil.ReadFile(filepath.Join(tDir, goldenText)) if err != nil { t.Fatalf("ReadFile: %v", err) } for i, tc := range extendMatchGoldenTestCases { got := extendMatch(src, tc.i, tc.j) if got != tc.want { t.Errorf("test #%d: i, j = %5d, %5d: got %5d (= j + %6d), want %5d (= j + %6d)", i, tc.i, tc.j, got, got-tc.j, tc.want, tc.want-tc.j) } } } func TestExtendMatch(t *testing.T) { // ref is a simple, reference implementation of extendMatch. ref := func(src []byte, i, j int) int { for ; j < len(src) && src[i] == src[j]; i, j = i+1, j+1 { } return j } nums := []int{0, 1, 2, 7, 8, 9, 29, 30, 31, 32, 33, 34, 38, 39, 40} for yIndex := 40; yIndex > 30; yIndex-- { xxx := bytes.Repeat([]byte("x"), 40) if yIndex < len(xxx) { xxx[yIndex] = 'y' } for _, i := range nums { for _, j := range nums { if i >= j { continue } got := extendMatch(xxx, i, j) want := ref(xxx, i, j) if got != want { t.Errorf("yIndex=%d, i=%d, j=%d: got %d, want %d", yIndex, i, j, got, want) } } } } } const snappytoolCmdName = "cmd/snappytool/snappytool" func skipTestSameEncodingAsCpp() (msg string) { if !goEncoderShouldMatchCppEncoder { return fmt.Sprintf("skipping testing that the encoding is byte-for-byte identical to C++: GOARCH=%s", runtime.GOARCH) } if _, err := os.Stat(snappytoolCmdName); err != nil { return fmt.Sprintf("could not find snappytool: %v", err) } return "" } func runTestSameEncodingAsCpp(src []byte) error { got := Encode(nil, src) cmd := exec.Command(snappytoolCmdName, "-e") cmd.Stdin = bytes.NewReader(src) want, err := cmd.Output() if err != nil { return fmt.Errorf("could not run snappytool: %v", err) } return cmp(got, want) } func TestSameEncodingAsCppShortCopies(t *testing.T) { if msg := skipTestSameEncodingAsCpp(); msg != "" { t.Skip(msg) } src := bytes.Repeat([]byte{'a'}, 20) for i := 0; i <= len(src); i++ { if err := runTestSameEncodingAsCpp(src[:i]); err != nil { t.Errorf("i=%d: %v", i, err) } } } func TestSameEncodingAsCppLongFiles(t *testing.T) { if msg := skipTestSameEncodingAsCpp(); msg != "" { t.Skip(msg) } bDir := filepath.FromSlash(*benchdataDir) failed := false for i, tf := range testFiles { if err := downloadBenchmarkFiles(t, tf.filename); err != nil { t.Fatalf("failed to download testdata: %s", err) } data := readFile(t, filepath.Join(bDir, tf.filename)) if n := tf.sizeLimit; 0 < n && n < len(data) { data = data[:n] } if err := runTestSameEncodingAsCpp(data); err != nil { t.Errorf("i=%d: %v", i, err) failed = true } } if failed { t.Errorf("was the snappytool program built against the C++ snappy library version " + "d53de187 or later, commited on 2016-04-05? See " + "https://github.com/google/snappy/commit/d53de18799418e113e44444252a39b12a0e4e0cc") } } // TestSlowForwardCopyOverrun tests the "expand the pattern" algorithm // described in decode_amd64.s and its claim of a 10 byte overrun worst case. func TestSlowForwardCopyOverrun(t *testing.T) { const base = 100 for length := 1; length < 18; length++ { for offset := 1; offset < 18; offset++ { highWaterMark := base d := base l := length o := offset // makeOffsetAtLeast8 for o < 8 { if end := d + 8; highWaterMark < end { highWaterMark = end } l -= o d += o o += o } // fixUpSlowForwardCopy a := d d += l // finishSlowForwardCopy for l > 0 { if end := a + 8; highWaterMark < end { highWaterMark = end } a += 8 l -= 8 } dWant := base + length overrun := highWaterMark - dWant if d != dWant || overrun < 0 || 10 < overrun { t.Errorf("length=%d, offset=%d: d and overrun: got (%d, %d), want (%d, something in [0, 10])", length, offset, d, overrun, dWant) } } } } // TestEncodeNoiseThenRepeats encodes input for which the first half is very // incompressible and the second half is very compressible. The encoded form's // length should be closer to 50% of the original length than 100%. func TestEncodeNoiseThenRepeats(t *testing.T) { for _, origLen := range []int{256 * 1024, 2048 * 1024} { src := make([]byte, origLen) rng := rand.New(rand.NewSource(1)) firstHalf, secondHalf := src[:origLen/2], src[origLen/2:] for i := range firstHalf { firstHalf[i] = uint8(rng.Intn(256)) } for i := range secondHalf { secondHalf[i] = uint8(i >> 8) } dst := Encode(nil, src) if got, want := len(dst), origLen*3/4; got >= want { t.Errorf("origLen=%d: got %d encoded bytes, want less than %d", origLen, got, want) } } } func TestFramingFormat(t *testing.T) { // src is comprised of alternating 1e5-sized sequences of random // (incompressible) bytes and repeated (compressible) bytes. 1e5 was chosen // because it is larger than maxBlockSize (64k). src := make([]byte, 1e6) rng := rand.New(rand.NewSource(1)) for i := 0; i < 10; i++ { if i%2 == 0 { for j := 0; j < 1e5; j++ { src[1e5*i+j] = uint8(rng.Intn(256)) } } else { for j := 0; j < 1e5; j++ { src[1e5*i+j] = uint8(i) } } } buf := new(bytes.Buffer) if _, err := NewWriter(buf).Write(src); err != nil { t.Fatalf("Write: encoding: %v", err) } dst, err := ioutil.ReadAll(NewReader(buf)) if err != nil { t.Fatalf("ReadAll: decoding: %v", err) } if err := cmp(dst, src); err != nil { t.Fatal(err) } } func TestWriterGoldenOutput(t *testing.T) { buf := new(bytes.Buffer) w := NewBufferedWriter(buf) defer w.Close() w.Write([]byte("abcd")) // Not compressible. w.Flush() w.Write(bytes.Repeat([]byte{'A'}, 150)) // Compressible. w.Flush() // The next chunk is also compressible, but a naive, greedy encoding of the // overall length 67 copy as a length 64 copy (the longest expressible as a // tagCopy1 or tagCopy2) plus a length 3 remainder would be two 3-byte // tagCopy2 tags (6 bytes), since the minimum length for a tagCopy1 is 4 // bytes. Instead, we could do it shorter, in 5 bytes: a 3-byte tagCopy2 // (of length 60) and a 2-byte tagCopy1 (of length 7). w.Write(bytes.Repeat([]byte{'B'}, 68)) w.Write([]byte("efC")) // Not compressible. w.Write(bytes.Repeat([]byte{'C'}, 20)) // Compressible. w.Write(bytes.Repeat([]byte{'B'}, 20)) // Compressible. w.Write([]byte("g")) // Not compressible. w.Flush() got := buf.String() want := strings.Join([]string{ magicChunk, "\x01\x08\x00\x00", // Uncompressed chunk, 8 bytes long (including 4 byte checksum). "\x68\x10\xe6\xb6", // Checksum. "\x61\x62\x63\x64", // Uncompressed payload: "abcd". "\x00\x11\x00\x00", // Compressed chunk, 17 bytes long (including 4 byte checksum). "\x5f\xeb\xf2\x10", // Checksum. "\x96\x01", // Compressed payload: Uncompressed length (varint encoded): 150. "\x00\x41", // Compressed payload: tagLiteral, length=1, "A". "\xfe\x01\x00", // Compressed payload: tagCopy2, length=64, offset=1. "\xfe\x01\x00", // Compressed payload: tagCopy2, length=64, offset=1. "\x52\x01\x00", // Compressed payload: tagCopy2, length=21, offset=1. "\x00\x18\x00\x00", // Compressed chunk, 24 bytes long (including 4 byte checksum). "\x30\x85\x69\xeb", // Checksum. "\x70", // Compressed payload: Uncompressed length (varint encoded): 112. "\x00\x42", // Compressed payload: tagLiteral, length=1, "B". "\xee\x01\x00", // Compressed payload: tagCopy2, length=60, offset=1. "\x0d\x01", // Compressed payload: tagCopy1, length=7, offset=1. "\x08\x65\x66\x43", // Compressed payload: tagLiteral, length=3, "efC". "\x4e\x01\x00", // Compressed payload: tagCopy2, length=20, offset=1. "\x4e\x5a\x00", // Compressed payload: tagCopy2, length=20, offset=90. "\x00\x67", // Compressed payload: tagLiteral, length=1, "g". }, "") if got != want { t.Fatalf("\ngot: % x\nwant: % x", got, want) } } func TestEmitLiteral(t *testing.T) { testCases := []struct { length int want string }{ {1, "\x00"}, {2, "\x04"}, {59, "\xe8"}, {60, "\xec"}, {61, "\xf0\x3c"}, {62, "\xf0\x3d"}, {254, "\xf0\xfd"}, {255, "\xf0\xfe"}, {256, "\xf0\xff"}, {257, "\xf4\x00\x01"}, {65534, "\xf4\xfd\xff"}, {65535, "\xf4\xfe\xff"}, {65536, "\xf4\xff\xff"}, } dst := make([]byte, 70000) nines := bytes.Repeat([]byte{0x99}, 65536) for _, tc := range testCases { lit := nines[:tc.length] n := emitLiteral(dst, lit) if !bytes.HasSuffix(dst[:n], lit) { t.Errorf("length=%d: did not end with that many literal bytes", tc.length) continue } got := string(dst[:n-tc.length]) if got != tc.want { t.Errorf("length=%d:\ngot % x\nwant % x", tc.length, got, tc.want) continue } } } func TestEmitCopy(t *testing.T) { testCases := []struct { offset int length int want string }{ {8, 04, "\x01\x08"}, {8, 11, "\x1d\x08"}, {8, 12, "\x2e\x08\x00"}, {8, 13, "\x32\x08\x00"}, {8, 59, "\xea\x08\x00"}, {8, 60, "\xee\x08\x00"}, {8, 61, "\xf2\x08\x00"}, {8, 62, "\xf6\x08\x00"}, {8, 63, "\xfa\x08\x00"}, {8, 64, "\xfe\x08\x00"}, {8, 65, "\xee\x08\x00\x05\x08"}, {8, 66, "\xee\x08\x00\x09\x08"}, {8, 67, "\xee\x08\x00\x0d\x08"}, {8, 68, "\xfe\x08\x00\x01\x08"}, {8, 69, "\xfe\x08\x00\x05\x08"}, {8, 80, "\xfe\x08\x00\x3e\x08\x00"}, {256, 04, "\x21\x00"}, {256, 11, "\x3d\x00"}, {256, 12, "\x2e\x00\x01"}, {256, 13, "\x32\x00\x01"}, {256, 59, "\xea\x00\x01"}, {256, 60, "\xee\x00\x01"}, {256, 61, "\xf2\x00\x01"}, {256, 62, "\xf6\x00\x01"}, {256, 63, "\xfa\x00\x01"}, {256, 64, "\xfe\x00\x01"}, {256, 65, "\xee\x00\x01\x25\x00"}, {256, 66, "\xee\x00\x01\x29\x00"}, {256, 67, "\xee\x00\x01\x2d\x00"}, {256, 68, "\xfe\x00\x01\x21\x00"}, {256, 69, "\xfe\x00\x01\x25\x00"}, {256, 80, "\xfe\x00\x01\x3e\x00\x01"}, {2048, 04, "\x0e\x00\x08"}, {2048, 11, "\x2a\x00\x08"}, {2048, 12, "\x2e\x00\x08"}, {2048, 13, "\x32\x00\x08"}, {2048, 59, "\xea\x00\x08"}, {2048, 60, "\xee\x00\x08"}, {2048, 61, "\xf2\x00\x08"}, {2048, 62, "\xf6\x00\x08"}, {2048, 63, "\xfa\x00\x08"}, {2048, 64, "\xfe\x00\x08"}, {2048, 65, "\xee\x00\x08\x12\x00\x08"}, {2048, 66, "\xee\x00\x08\x16\x00\x08"}, {2048, 67, "\xee\x00\x08\x1a\x00\x08"}, {2048, 68, "\xfe\x00\x08\x0e\x00\x08"}, {2048, 69, "\xfe\x00\x08\x12\x00\x08"}, {2048, 80, "\xfe\x00\x08\x3e\x00\x08"}, } dst := make([]byte, 1024) for _, tc := range testCases { n := emitCopy(dst, tc.offset, tc.length) got := string(dst[:n]) if got != tc.want { t.Errorf("offset=%d, length=%d:\ngot % x\nwant % x", tc.offset, tc.length, got, tc.want) } } } func TestNewBufferedWriter(t *testing.T) { // Test all 32 possible sub-sequences of these 5 input slices. // // Their lengths sum to 400,000, which is over 6 times the Writer ibuf // capacity: 6 * maxBlockSize is 393,216. inputs := [][]byte{ bytes.Repeat([]byte{'a'}, 40000), bytes.Repeat([]byte{'b'}, 150000), bytes.Repeat([]byte{'c'}, 60000), bytes.Repeat([]byte{'d'}, 120000), bytes.Repeat([]byte{'e'}, 30000), } loop: for i := 0; i < 1< 0; { i := copy(x, src) x = x[i:] } return dst } func benchWords(b *testing.B, n int, decode bool) { // Note: the file is OS-language dependent so the resulting values are not // directly comparable for non-US-English OS installations. data := expand(readFile(b, "/usr/share/dict/words"), n) if decode { benchDecode(b, data) } else { benchEncode(b, data) } } func BenchmarkWordsDecode1e1(b *testing.B) { benchWords(b, 1e1, true) } func BenchmarkWordsDecode1e2(b *testing.B) { benchWords(b, 1e2, true) } func BenchmarkWordsDecode1e3(b *testing.B) { benchWords(b, 1e3, true) } func BenchmarkWordsDecode1e4(b *testing.B) { benchWords(b, 1e4, true) } func BenchmarkWordsDecode1e5(b *testing.B) { benchWords(b, 1e5, true) } func BenchmarkWordsDecode1e6(b *testing.B) { benchWords(b, 1e6, true) } func BenchmarkWordsEncode1e1(b *testing.B) { benchWords(b, 1e1, false) } func BenchmarkWordsEncode1e2(b *testing.B) { benchWords(b, 1e2, false) } func BenchmarkWordsEncode1e3(b *testing.B) { benchWords(b, 1e3, false) } func BenchmarkWordsEncode1e4(b *testing.B) { benchWords(b, 1e4, false) } func BenchmarkWordsEncode1e5(b *testing.B) { benchWords(b, 1e5, false) } func BenchmarkWordsEncode1e6(b *testing.B) { benchWords(b, 1e6, false) } func BenchmarkRandomEncode(b *testing.B) { rng := rand.New(rand.NewSource(1)) data := make([]byte, 1<<20) for i := range data { data[i] = uint8(rng.Intn(256)) } benchEncode(b, data) } // testFiles' values are copied directly from // https://raw.githubusercontent.com/google/snappy/master/snappy_unittest.cc // The label field is unused in snappy-go. var testFiles = []struct { label string filename string sizeLimit int }{ {"html", "html", 0}, {"urls", "urls.10K", 0}, {"jpg", "fireworks.jpeg", 0}, {"jpg_200", "fireworks.jpeg", 200}, {"pdf", "paper-100k.pdf", 0}, {"html4", "html_x_4", 0}, {"txt1", "alice29.txt", 0}, {"txt2", "asyoulik.txt", 0}, {"txt3", "lcet10.txt", 0}, {"txt4", "plrabn12.txt", 0}, {"pb", "geo.protodata", 0}, {"gaviota", "kppkn.gtb", 0}, } const ( // The benchmark data files are at this canonical URL. benchURL = "https://raw.githubusercontent.com/google/snappy/master/testdata/" ) func downloadBenchmarkFiles(b testing.TB, basename string) (errRet error) { bDir := filepath.FromSlash(*benchdataDir) filename := filepath.Join(bDir, basename) if stat, err := os.Stat(filename); err == nil && stat.Size() != 0 { return nil } if !*download { b.Skipf("test data not found; skipping %s without the -download flag", testOrBenchmark(b)) } // Download the official snappy C++ implementation reference test data // files for benchmarking. if err := os.MkdirAll(bDir, 0777); err != nil && !os.IsExist(err) { return fmt.Errorf("failed to create %s: %s", bDir, err) } f, err := os.Create(filename) if err != nil { return fmt.Errorf("failed to create %s: %s", filename, err) } defer f.Close() defer func() { if errRet != nil { os.Remove(filename) } }() url := benchURL + basename resp, err := http.Get(url) if err != nil { return fmt.Errorf("failed to download %s: %s", url, err) } defer resp.Body.Close() if s := resp.StatusCode; s != http.StatusOK { return fmt.Errorf("downloading %s: HTTP status code %d (%s)", url, s, http.StatusText(s)) } _, err = io.Copy(f, resp.Body) if err != nil { return fmt.Errorf("failed to download %s to %s: %s", url, filename, err) } return nil } func benchFile(b *testing.B, i int, decode bool) { if err := downloadBenchmarkFiles(b, testFiles[i].filename); err != nil { b.Fatalf("failed to download testdata: %s", err) } bDir := filepath.FromSlash(*benchdataDir) data := readFile(b, filepath.Join(bDir, testFiles[i].filename)) if n := testFiles[i].sizeLimit; 0 < n && n < len(data) { data = data[:n] } if decode { benchDecode(b, data) } else { benchEncode(b, data) } } // Naming convention is kept similar to what snappy's C++ implementation uses. func Benchmark_UFlat0(b *testing.B) { benchFile(b, 0, true) } func Benchmark_UFlat1(b *testing.B) { benchFile(b, 1, true) } func Benchmark_UFlat2(b *testing.B) { benchFile(b, 2, true) } func Benchmark_UFlat3(b *testing.B) { benchFile(b, 3, true) } func Benchmark_UFlat4(b *testing.B) { benchFile(b, 4, true) } func Benchmark_UFlat5(b *testing.B) { benchFile(b, 5, true) } func Benchmark_UFlat6(b *testing.B) { benchFile(b, 6, true) } func Benchmark_UFlat7(b *testing.B) { benchFile(b, 7, true) } func Benchmark_UFlat8(b *testing.B) { benchFile(b, 8, true) } func Benchmark_UFlat9(b *testing.B) { benchFile(b, 9, true) } func Benchmark_UFlat10(b *testing.B) { benchFile(b, 10, true) } func Benchmark_UFlat11(b *testing.B) { benchFile(b, 11, true) } func Benchmark_ZFlat0(b *testing.B) { benchFile(b, 0, false) } func Benchmark_ZFlat1(b *testing.B) { benchFile(b, 1, false) } func Benchmark_ZFlat2(b *testing.B) { benchFile(b, 2, false) } func Benchmark_ZFlat3(b *testing.B) { benchFile(b, 3, false) } func Benchmark_ZFlat4(b *testing.B) { benchFile(b, 4, false) } func Benchmark_ZFlat5(b *testing.B) { benchFile(b, 5, false) } func Benchmark_ZFlat6(b *testing.B) { benchFile(b, 6, false) } func Benchmark_ZFlat7(b *testing.B) { benchFile(b, 7, false) } func Benchmark_ZFlat8(b *testing.B) { benchFile(b, 8, false) } func Benchmark_ZFlat9(b *testing.B) { benchFile(b, 9, false) } func Benchmark_ZFlat10(b *testing.B) { benchFile(b, 10, false) } func Benchmark_ZFlat11(b *testing.B) { benchFile(b, 11, false) } func BenchmarkExtendMatch(b *testing.B) { tDir := filepath.FromSlash(*testdataDir) src, err := ioutil.ReadFile(filepath.Join(tDir, goldenText)) if err != nil { b.Fatalf("ReadFile: %v", err) } b.ResetTimer() for i := 0; i < b.N; i++ { for _, tc := range extendMatchGoldenTestCases { extendMatch(src, tc.i, tc.j) } } } golang-github-golang-snappy-0.0+git20160529.d9eb7a3/testdata/000077500000000000000000000000001272247357100232455ustar00rootroot00000000000000golang-github-golang-snappy-0.0+git20160529.d9eb7a3/testdata/Mark.Twain-Tom.Sawyer.txt000066400000000000000000000335301272247357100277730ustar00rootroot00000000000000Produced by David Widger. The previous edition was updated by Jose Menendez. THE ADVENTURES OF TOM SAWYER BY MARK TWAIN (Samuel Langhorne Clemens) P R E F A C E MOST of the adventures recorded in this book really occurred; one or two were experiences of my own, the rest those of boys who were schoolmates of mine. Huck Finn is drawn from life; Tom Sawyer also, but not from an individual--he is a combination of the characteristics of three boys whom I knew, and therefore belongs to the composite order of architecture. The odd superstitions touched upon were all prevalent among children and slaves in the West at the period of this story--that is to say, thirty or forty years ago. Although my book is intended mainly for the entertainment of boys and girls, I hope it will not be shunned by men and women on that account, for part of my plan has been to try to pleasantly remind adults of what they once were themselves, and of how they felt and thought and talked, and what queer enterprises they sometimes engaged in. THE AUTHOR. HARTFORD, 1876. T O M S A W Y E R CHAPTER I "TOM!" No answer. "TOM!" No answer. "What's gone with that boy, I wonder? You TOM!" No answer. The old lady pulled her spectacles down and looked over them about the room; then she put them up and looked out under them. She seldom or never looked THROUGH them for so small a thing as a boy; they were her state pair, the pride of her heart, and were built for "style," not service--she could have seen through a pair of stove-lids just as well. She looked perplexed for a moment, and then said, not fiercely, but still loud enough for the furniture to hear: "Well, I lay if I get hold of you I'll--" She did not finish, for by this time she was bending down and punching under the bed with the broom, and so she needed breath to punctuate the punches with. She resurrected nothing but the cat. "I never did see the beat of that boy!" She went to the open door and stood in it and looked out among the tomato vines and "jimpson" weeds that constituted the garden. No Tom. So she lifted up her voice at an angle calculated for distance and shouted: "Y-o-u-u TOM!" There was a slight noise behind her and she turned just in time to seize a small boy by the slack of his roundabout and arrest his flight. "There! I might 'a' thought of that closet. What you been doing in there?" "Nothing." "Nothing! Look at your hands. And look at your mouth. What IS that truck?" "I don't know, aunt." "Well, I know. It's jam--that's what it is. Forty times I've said if you didn't let that jam alone I'd skin you. Hand me that switch." The switch hovered in the air--the peril was desperate-- "My! Look behind you, aunt!" The old lady whirled round, and snatched her skirts out of danger. The lad fled on the instant, scrambled up the high board-fence, and disappeared over it. His aunt Polly stood surprised a moment, and then broke into a gentle laugh. "Hang the boy, can't I never learn anything? Ain't he played me tricks enough like that for me to be looking out for him by this time? But old fools is the biggest fools there is. Can't learn an old dog new tricks, as the saying is. But my goodness, he never plays them alike, two days, and how is a body to know what's coming? He 'pears to know just how long he can torment me before I get my dander up, and he knows if he can make out to put me off for a minute or make me laugh, it's all down again and I can't hit him a lick. I ain't doing my duty by that boy, and that's the Lord's truth, goodness knows. Spare the rod and spile the child, as the Good Book says. I'm a laying up sin and suffering for us both, I know. He's full of the Old Scratch, but laws-a-me! he's my own dead sister's boy, poor thing, and I ain't got the heart to lash him, somehow. Every time I let him off, my conscience does hurt me so, and every time I hit him my old heart most breaks. Well-a-well, man that is born of woman is of few days and full of trouble, as the Scripture says, and I reckon it's so. He'll play hookey this evening, * and [* Southwestern for "afternoon"] I'll just be obleeged to make him work, to-morrow, to punish him. It's mighty hard to make him work Saturdays, when all the boys is having holiday, but he hates work more than he hates anything else, and I've GOT to do some of my duty by him, or I'll be the ruination of the child." Tom did play hookey, and he had a very good time. He got back home barely in season to help Jim, the small colored boy, saw next-day's wood and split the kindlings before supper--at least he was there in time to tell his adventures to Jim while Jim did three-fourths of the work. Tom's younger brother (or rather half-brother) Sid was already through with his part of the work (picking up chips), for he was a quiet boy, and had no adventurous, troublesome ways. While Tom was eating his supper, and stealing sugar as opportunity offered, Aunt Polly asked him questions that were full of guile, and very deep--for she wanted to trap him into damaging revealments. Like many other simple-hearted souls, it was her pet vanity to believe she was endowed with a talent for dark and mysterious diplomacy, and she loved to contemplate her most transparent devices as marvels of low cunning. Said she: "Tom, it was middling warm in school, warn't it?" "Yes'm." "Powerful warm, warn't it?" "Yes'm." "Didn't you want to go in a-swimming, Tom?" A bit of a scare shot through Tom--a touch of uncomfortable suspicion. He searched Aunt Polly's face, but it told him nothing. So he said: "No'm--well, not very much." The old lady reached out her hand and felt Tom's shirt, and said: "But you ain't too warm now, though." And it flattered her to reflect that she had discovered that the shirt was dry without anybody knowing that that was what she had in her mind. But in spite of her, Tom knew where the wind lay, now. So he forestalled what might be the next move: "Some of us pumped on our heads--mine's damp yet. See?" Aunt Polly was vexed to think she had overlooked that bit of circumstantial evidence, and missed a trick. Then she had a new inspiration: "Tom, you didn't have to undo your shirt collar where I sewed it, to pump on your head, did you? Unbutton your jacket!" The trouble vanished out of Tom's face. He opened his jacket. His shirt collar was securely sewed. "Bother! Well, go 'long with you. I'd made sure you'd played hookey and been a-swimming. But I forgive ye, Tom. I reckon you're a kind of a singed cat, as the saying is--better'n you look. THIS time." She was half sorry her sagacity had miscarried, and half glad that Tom had stumbled into obedient conduct for once. But Sidney said: "Well, now, if I didn't think you sewed his collar with white thread, but it's black." "Why, I did sew it with white! Tom!" But Tom did not wait for the rest. As he went out at the door he said: "Siddy, I'll lick you for that." In a safe place Tom examined two large needles which were thrust into the lapels of his jacket, and had thread bound about them--one needle carried white thread and the other black. He said: "She'd never noticed if it hadn't been for Sid. Confound it! sometimes she sews it with white, and sometimes she sews it with black. I wish to geeminy she'd stick to one or t'other--I can't keep the run of 'em. But I bet you I'll lam Sid for that. I'll learn him!" He was not the Model Boy of the village. He knew the model boy very well though--and loathed him. Within two minutes, or even less, he had forgotten all his troubles. Not because his troubles were one whit less heavy and bitter to him than a man's are to a man, but because a new and powerful interest bore them down and drove them out of his mind for the time--just as men's misfortunes are forgotten in the excitement of new enterprises. This new interest was a valued novelty in whistling, which he had just acquired from a negro, and he was suffering to practise it undisturbed. It consisted in a peculiar bird-like turn, a sort of liquid warble, produced by touching the tongue to the roof of the mouth at short intervals in the midst of the music--the reader probably remembers how to do it, if he has ever been a boy. Diligence and attention soon gave him the knack of it, and he strode down the street with his mouth full of harmony and his soul full of gratitude. He felt much as an astronomer feels who has discovered a new planet--no doubt, as far as strong, deep, unalloyed pleasure is concerned, the advantage was with the boy, not the astronomer. The summer evenings were long. It was not dark, yet. Presently Tom checked his whistle. A stranger was before him--a boy a shade larger than himself. A new-comer of any age or either sex was an impressive curiosity in the poor little shabby village of St. Petersburg. This boy was well dressed, too--well dressed on a week-day. This was simply astounding. His cap was a dainty thing, his close-buttoned blue cloth roundabout was new and natty, and so were his pantaloons. He had shoes on--and it was only Friday. He even wore a necktie, a bright bit of ribbon. He had a citified air about him that ate into Tom's vitals. The more Tom stared at the splendid marvel, the higher he turned up his nose at his finery and the shabbier and shabbier his own outfit seemed to him to grow. Neither boy spoke. If one moved, the other moved--but only sidewise, in a circle; they kept face to face and eye to eye all the time. Finally Tom said: "I can lick you!" "I'd like to see you try it." "Well, I can do it." "No you can't, either." "Yes I can." "No you can't." "I can." "You can't." "Can!" "Can't!" An uncomfortable pause. Then Tom said: "What's your name?" "'Tisn't any of your business, maybe." "Well I 'low I'll MAKE it my business." "Well why don't you?" "If you say much, I will." "Much--much--MUCH. There now." "Oh, you think you're mighty smart, DON'T you? I could lick you with one hand tied behind me, if I wanted to." "Well why don't you DO it? You SAY you can do it." "Well I WILL, if you fool with me." "Oh yes--I've seen whole families in the same fix." "Smarty! You think you're SOME, now, DON'T you? Oh, what a hat!" "You can lump that hat if you don't like it. I dare you to knock it off--and anybody that'll take a dare will suck eggs." "You're a liar!" "You're another." "You're a fighting liar and dasn't take it up." "Aw--take a walk!" "Say--if you give me much more of your sass I'll take and bounce a rock off'n your head." "Oh, of COURSE you will." "Well I WILL." "Well why don't you DO it then? What do you keep SAYING you will for? Why don't you DO it? It's because you're afraid." "I AIN'T afraid." "You are." "I ain't." "You are." Another pause, and more eying and sidling around each other. Presently they were shoulder to shoulder. Tom said: "Get away from here!" "Go away yourself!" "I won't." "I won't either." So they stood, each with a foot placed at an angle as a brace, and both shoving with might and main, and glowering at each other with hate. But neither could get an advantage. After struggling till both were hot and flushed, each relaxed his strain with watchful caution, and Tom said: "You're a coward and a pup. I'll tell my big brother on you, and he can thrash you with his little finger, and I'll make him do it, too." "What do I care for your big brother? I've got a brother that's bigger than he is--and what's more, he can throw him over that fence, too." [Both brothers were imaginary.] "That's a lie." "YOUR saying so don't make it so." Tom drew a line in the dust with his big toe, and said: "I dare you to step over that, and I'll lick you till you can't stand up. Anybody that'll take a dare will steal sheep." The new boy stepped over promptly, and said: "Now you said you'd do it, now let's see you do it." "Don't you crowd me now; you better look out." "Well, you SAID you'd do it--why don't you do it?" "By jingo! for two cents I WILL do it." The new boy took two broad coppers out of his pocket and held them out with derision. Tom struck them to the ground. In an instant both boys were rolling and tumbling in the dirt, gripped together like cats; and for the space of a minute they tugged and tore at each other's hair and clothes, punched and scratched each other's nose, and covered themselves with dust and glory. Presently the confusion took form, and through the fog of battle Tom appeared, seated astride the new boy, and pounding him with his fists. "Holler 'nuff!" said he. The boy only struggled to free himself. He was crying--mainly from rage. "Holler 'nuff!"--and the pounding went on. At last the stranger got out a smothered "'Nuff!" and Tom let him up and said: "Now that'll learn you. Better look out who you're fooling with next time." The new boy went off brushing the dust from his clothes, sobbing, snuffling, and occasionally looking back and shaking his head and threatening what he would do to Tom the "next time he caught him out." To which Tom responded with jeers, and started off in high feather, and as soon as his back was turned the new boy snatched up a stone, threw it and hit him between the shoulders and then turned tail and ran like an antelope. Tom chased the traitor home, and thus found out where he lived. He then held a position at the gate for some time, daring the enemy to come outside, but the enemy only made faces at him through the window and declined. At last the enemy's mother appeared, and called Tom a bad, vicious, vulgar child, and ordered him away. So he went away; but he said he "'lowed" to "lay" for that boy. He got home pretty late that night, and when he climbed cautiously in at the window, he uncovered an ambuscade, in the person of his aunt; and when she saw the state his clothes were in her resolution to turn his Saturday holiday into captivity at hard labor became adamantine in its firmness. golang-github-golang-snappy-0.0+git20160529.d9eb7a3/testdata/Mark.Twain-Tom.Sawyer.txt.rawsnappy000066400000000000000000000232171272247357100320170ustar00rootroot00000000000000nProduced by David Widger. The previous edition was update28Jose Menendez.  FlTHE ADVENTURES OF TOM SAWYERN02BYr#$MARK TWAINV'd(Samuel Langhorne Clemens)^P R E F A C E MOST of the adventures recorded in this book really occurred; one or two were experiences of my own, the rest thoseoboys wh 7$ schoolmat 7ine. Huck Finn is drawn from life; Tom Sawyer also, but not $@an individual--heD$a combinat! \characteristics of three0m I knew, and!'@refore belongs to[posite !8rHarchitec!R. Az(odd superstEz?,uched upon w!DallEPalent among children slaves % e West at!}od 4is story--that!Lto say, thirty or fo \years ago. Although my %iita d mainly6n$entertainmof)3and girls, I hope it will not be shunneeimen0 wo on @account, for part\. VdTUTHOR. HARTFORD, 1876fGHT O M S A W Y E RpCHAPTER I "TOM!" No answer.ZWhat's ga with)Hboy, I wonder? You>REld lady pulled her spectacles dowI$looked oveEbm abouE room;!gn s  m up%: ut u*0m. She seldomneY +THROUGH)Eso sma}ang as awyI<4her state pairprideEFer heartIXebuilt`"style,"a$ service-- could havAhreaeZ0stove-lids juas well.  perplexedna maU th!8 aid,~$ fiercely,6staloud enyayfurniag: "Wella4lay if I get h!8of you I'll--" did snish,[by!bs a,% bend!vU punching 5 bedQe bA)j!he nee breaAoCtu! the QesBIresurrecnotn!. cataI I6seebeaboyA!w open doorE stoo -A,tomato vines&4"jimpson" weedeat con=u!Dgarden. No Tom. So%Slif"upA voice&aaigllcula$!dista"sho_!Y-o-u-uTh!0a slight nois!hini!turEimAG seize?elboyIeckADroundiParrfw!! I m'a'!-loset. E;doAiinre?" "N). ! Look!58your hands. And%mouth bISE truckX(I don't knoBuntcqD. I^ jam-' atA+is. F!> s I'#aid if akdid^letyE alI'd skin. H!E!Yswitch  huŲEVaireVil%ld fools  bigga7 teASC%dog new , ane0 x5b,my goodness,ǩ/A|,ike, two dayHowLd AI 's8 ing? He '! !j how h nca rn ms ɀmyA:der up%1hshe = make%DtEoff%Ra minu ' me l!, i agai!d I't hi0m a lick. I a%futatE&I a'%7 Lord ruth,5b . SparAe rod !Qspile ,!GeBsays. I'a% up s suffer!for u y 0ow. He's fullG e Old Scra20 laws-a-me! h-my  ,dead sister'U y, pE)1goeUt! lash him,F hEvery IA off,!Ucj does hur%so)_ =1myEt most, ks. -a-U , marn!womUU fewE%5)roublz% Scrip< !recko's so%l L hooke $evening, *E, [* Sgwe!\n!0 "afternoon"]s i1be oblee~ toI@him work, to-morre ̈́Iy hard e0!u= Sature wO aAvis ASholida[ ahB morA e a els!<ndGOTdoE!}IA' or bmuBildom(-e)EhEm!Eb(home barely seaso help JA! colom L stea![ sugar!0opportunity oed, Aunas{A ques#aat qguiend E6deep-- wanQ  rapMi6damagreveal0s. Like many %c simple-ed souE>as* pet vacbelievA "endow! a t@ dark%bmysdiplomacm she loveecontemp x*transnt devic s marvelE[low cun. S sheE TomG%midE warmeR,Nit8 Yes'ma"Powerfu rmf'D  !!!go`a-swimmNTom^ A bim a sc]shŏITom--a tyduncomfortable suspicion. Hed]I<a but!t AWnc  S=  !No'm-y"EWmu& @I reacl 9  %Toasmaia !too!km  ." it flattea to reflecCat!ediscov'ey%dry anyb   /aV a8aint.m int her, w huln)Z stajc) movESom`us pump|n  eads--'sap yISe08vwa hink!$ aVIvL circumstantial evid"  miss  !$a  inspirAAe^! hav$ undor-collarrQ se-it f  ead,N ? UnbuttoJr jacke" sQof!mNope  @. His I4as secuj^ B!   ' uyou dare'fQ A  IaIA*give yeE^  you'ra sinU caa`a[ is--bea9'%& . THISR U sorryA sagacit %carri*,gE? Tom Astu\obediconduc:. aSidneyo w, I!wIAF%{%1wN ad,  lack"Why, OBL8!watot. As хA^I%q Siddp= v  IabInafe cevexaadlarge2whichthry!\c lapYa!` ad b&m--on^ -et1D%E )Hs he'danotice!Xha%E Sid. Confit!3 * s ]seth T &2& ,I wish to geYy*X tick!'--keeru= 'emes I bet%)amAall rn4A4H s!%0 Model Boya!Avillaghm&]O  --aZlo AmR iAEs,  l!forgott t s. NcauseE*IV!*aDPvyEbi7!L t a man's rmanwb _?p ibd em%droaSis1ye--r en's misun re he excite Tnew.is a value veltU whist Za  acquigroIhI&it unurbed. I;new-c%Aofs ageisexXDan impressive curi e litt habby\W4St. Petersburgi{`dZtoo Z ak  < yAing capa daint!!5- ed blueth `3\ne# natto)Hp!loons.wshoes on#i)o Fr)! necktiO br ribbon0Ya citifi ir' nA 's vital?e mi"1 st spl- }Ihig uAvosfinS!%ie! )aoutfitmedr o grow. NM]A spoke. If\ mb -! %Nsidewi  leekept @Leyalle.$a%$: " C" o="i,  8doN> ,  Y.'!Can~ An:\p=aen2r nam"'Tis @yr busimaybew  I8 wu MAKEA# 0)why you?If say #3Much----MUCHre<Oh,=_  T y2)!DON'T%?! !" w: hAta@bnme  op2 DOS !SAYi!=4I WILLT 1~ m)EOh yes--!,whole famili&%same fixSmarty!|2 SOME !-aA!QA lumP at i)YEitd tB ck it offte t bre% suck eggIY" !aem %f .edaEf p% Aw--a walkXSay-- meEj/{ sassI @nu   ro9 f'n ,o!S of COURSE=J;)n?A d SAYINGT ? W.>Q{Ia X frai I AIaAarIzIA)3 -e!siha <.&J$!lr #hae.+om"Get away %Go !&' BS3)y{, a aazt a~' a"%rY} bothvAu!sflu rela%Pstrainx wHcau .juWcowar,a pup ig" thrash! y f+"Rudo)p![ {?goty /fs'g" +'sE,throwdo' !t[ uWiXary.]((i=YOUR ss5 1!xdr, lin!a dA'%6!itomh&h.q step)%ZE5 nd !An%V|eF^# new { tep!4promptl  Noww 'd!<nowXe %lD%, rowd now; & 0P SAID'h--6Wd[ By jingo!EX5"cents )].t + broad copN-Apoehel-m %d-i !s%!m g0. BF.e!qsirol adigri% toge cats;6 aJo spac"",tugm` 2'mI nd cls,(3 %a%'Du !m - )A!glory*bfu!=)foaug,'g!baI!e%,> U3+!Me1 pong!fis/8"Holler 'nuff!"IhA a ^ t  f/"*Hcr%--.U0rC :d-At"!AHe a sm!"'N !Y!'M upaa%yBZunz   !/%+  "m0-[ ff brus"%!dUG sobbm s!g^% occa!o ax"!shaK~ Aeat"wZat%\" "cv%R%ouaTo*ch%; respp., jeer\##A]L(f , `M#)/!12"(^.neerew iZ% betw >7 $s-\ail! rake antelop om chaH,tor{")}thus A]w} lived.Ana pion5PgbpsA",!"enemy!C`q> e[ms[A} otdecMd. .%JABca bvgus, vul!3)c3} Ta  ; 1"' d""lay"%an:\1##~3 tty !#n {$htim. us$ ) > +!mbuscadhsfe;@+;gast!AmAeraolN Av +% %hcaptivuQr1bo<ca"nt 8 its firmness.