pax_global_header00006660000000000000000000000064141070751750014521gustar00rootroot0000000000000052 comment=ae8d9b870b19ddd1cae2d86cfff5278d931d94b8 murmur3-1.1.6/000077500000000000000000000000001410707517500131405ustar00rootroot00000000000000murmur3-1.1.6/.gitignore000066400000000000000000000000061410707517500151240ustar00rootroot00000000000000*.swp murmur3-1.1.6/.travis.yml000066400000000000000000000001241410707517500152460ustar00rootroot00000000000000sudo: false language: go go: - "1.10" - "1.11" notifications: email: false murmur3-1.1.6/LICENSE000066400000000000000000000056541410707517500141570ustar00rootroot00000000000000Copyright 2013, Sébastien Paolacci. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the library nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. Copyright 2018, Travis Bischel. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the library nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. murmur3-1.1.6/README.md000066400000000000000000000142261410707517500144240ustar00rootroot00000000000000murmur3 ======= Native Go implementation of Austin Appleby's third MurmurHash revision (aka MurmurHash3). Includes assembly for amd64 for 64/128 bit hashes, seeding functions, and string functions to avoid string to slice conversions. Hand rolled 32 bit assembly was removed during 1.11, but may be reintroduced if the compiler slows down any more. As is, the compiler generates marginally slower code (by one instruction in the hot loop). The reference algorithm has been slightly hacked as to support the streaming mode required by Go's standard [Hash interface](http://golang.org/pkg/hash/#Hash). Endianness ========== Unlike the canonical source, this library **always** reads bytes as little endian numbers. This makes the hashes portable across architectures, although does mean that hashing is a bit slower on big endian architectures. Safety ====== This library used to use `unsafe` to convert four bytes to a `uint32` and eight bytes to a `uint64`, but Go 1.14 introduced checks around those types of conversions that flagged that code as erroneous when hashing on unaligned input. While the code would not be problematic on amd64, it could be problematic on some architectures. As of Go 1.14, those conversions were removed at the expense of a very minor performance hit. This hit affects all cpu architectures on for `Sum32`, and non-amd64 architectures for `Sum64` and `Sum128`. For 64 and 128, custom assembly exists for amd64 that preserves performance. Testing ======= [![Build Status](https://travis-ci.org/twmb/murmur3.svg?branch=master)](https://travis-ci.org/twmb/murmur3) Testing includes comparing random inputs against the [canonical implementation](https://github.com/aappleby/smhasher/blob/master/src/MurmurHash3.cpp), and testing length 0 through 17 inputs to force all branches. Because this code always reads input as little endian, testing against the canonical source is skipped for big endian architectures. The canonical source just converts bytes to numbers, meaning on big endian architectures, it will use different numbers for its hashing. Documentation ============= [![GoDoc](https://godoc.org/github.com/twmb/murmur3?status.svg)](https://godoc.org/github.com/twmb/murmur3) Full documentation can be found on `godoc`. Benchmarks ========== Benchmarks below were run on an amd64 machine with _and_ without the custom assembly. The following numbers are for Go 1.14.1 and are comparing against [spaolacci/murmur3](https://github.com/spaolacci/murmur3). You will notice that at small sizes, the other library is better. This is due to this library converting to safe code for Go 1.14. At large sizes, this library is nearly identical to the other. On amd64, the 64 bit and 128 bit sums come out to ~9% faster. 32 bit sums: ``` 32Sizes/32-12 3.00GB/s ± 1% 2.12GB/s ±11% -29.24% (p=0.000 n=9+10) 32Sizes/64-12 3.61GB/s ± 3% 2.79GB/s ± 8% -22.62% (p=0.000 n=10+10) 32Sizes/128-12 3.47GB/s ± 8% 2.79GB/s ± 4% -19.47% (p=0.000 n=10+10) 32Sizes/256-12 3.66GB/s ± 4% 3.25GB/s ± 6% -11.09% (p=0.000 n=10+10) 32Sizes/512-12 3.78GB/s ± 3% 3.54GB/s ± 4% -6.30% (p=0.000 n=9+9) 32Sizes/1024-12 3.86GB/s ± 3% 3.69GB/s ± 5% -4.46% (p=0.000 n=10+10) 32Sizes/2048-12 3.85GB/s ± 3% 3.81GB/s ± 3% ~ (p=0.079 n=10+9) 32Sizes/4096-12 3.90GB/s ± 3% 3.82GB/s ± 2% -2.14% (p=0.029 n=10+10) 32Sizes/8192-12 3.82GB/s ± 3% 3.78GB/s ± 7% ~ (p=0.529 n=10+10) ``` 64/128 bit sums, non-amd64: ``` 64Sizes/32-12 2.34GB/s ± 5% 2.64GB/s ± 9% +12.87% (p=0.000 n=10+10) 64Sizes/64-12 3.62GB/s ± 5% 3.96GB/s ± 4% +9.41% (p=0.000 n=10+10) 64Sizes/128-12 5.12GB/s ± 3% 5.44GB/s ± 4% +6.09% (p=0.000 n=10+9) 64Sizes/256-12 6.35GB/s ± 2% 6.27GB/s ± 9% ~ (p=0.796 n=10+10) 64Sizes/512-12 6.58GB/s ± 7% 6.79GB/s ± 3% ~ (p=0.075 n=10+10) 64Sizes/1024-12 7.49GB/s ± 3% 7.55GB/s ± 9% ~ (p=0.393 n=10+10) 64Sizes/2048-12 8.06GB/s ± 2% 7.90GB/s ± 6% ~ (p=0.156 n=9+10) 64Sizes/4096-12 8.27GB/s ± 6% 8.22GB/s ± 5% ~ (p=0.631 n=10+10) 64Sizes/8192-12 8.35GB/s ± 4% 8.38GB/s ± 6% ~ (p=0.631 n=10+10) 128Sizes/32-12 2.27GB/s ± 2% 2.68GB/s ± 5% +18.00% (p=0.000 n=10+10) 128Sizes/64-12 3.55GB/s ± 2% 4.00GB/s ± 3% +12.47% (p=0.000 n=8+9) 128Sizes/128-12 5.09GB/s ± 1% 5.43GB/s ± 3% +6.65% (p=0.000 n=9+9) 128Sizes/256-12 6.33GB/s ± 3% 5.65GB/s ± 4% -10.79% (p=0.000 n=9+10) 128Sizes/512-12 6.78GB/s ± 3% 6.74GB/s ± 6% ~ (p=0.968 n=9+10) 128Sizes/1024-12 7.46GB/s ± 4% 7.56GB/s ± 4% ~ (p=0.222 n=9+9) 128Sizes/2048-12 7.99GB/s ± 4% 7.96GB/s ± 3% ~ (p=0.666 n=9+9) 128Sizes/4096-12 8.20GB/s ± 2% 8.25GB/s ± 4% ~ (p=0.631 n=10+10) 128Sizes/8192-12 8.24GB/s ± 2% 8.26GB/s ± 5% ~ (p=0.673 n=8+9) ``` 64/128 bit sums, amd64: ``` 64Sizes/32-12 2.34GB/s ± 5% 4.36GB/s ± 3% +85.86% (p=0.000 n=10+10) 64Sizes/64-12 3.62GB/s ± 5% 6.27GB/s ± 3% +73.37% (p=0.000 n=10+9) 64Sizes/128-12 5.12GB/s ± 3% 7.70GB/s ± 6% +50.27% (p=0.000 n=10+10) 64Sizes/256-12 6.35GB/s ± 2% 8.61GB/s ± 3% +35.50% (p=0.000 n=10+10) 64Sizes/512-12 6.58GB/s ± 7% 8.59GB/s ± 4% +30.48% (p=0.000 n=10+9) 64Sizes/1024-12 7.49GB/s ± 3% 8.81GB/s ± 2% +17.66% (p=0.000 n=10+10) 64Sizes/2048-12 8.06GB/s ± 2% 8.90GB/s ± 4% +10.49% (p=0.000 n=9+10) 64Sizes/4096-12 8.27GB/s ± 6% 8.90GB/s ± 4% +7.54% (p=0.000 n=10+10) 64Sizes/8192-12 8.35GB/s ± 4% 9.00GB/s ± 3% +7.80% (p=0.000 n=10+9) 128Sizes/32-12 2.27GB/s ± 2% 4.29GB/s ± 9% +88.75% (p=0.000 n=10+10) 128Sizes/64-12 3.55GB/s ± 2% 6.10GB/s ± 8% +71.78% (p=0.000 n=8+10) 128Sizes/128-12 5.09GB/s ± 1% 7.62GB/s ± 9% +49.63% (p=0.000 n=9+10) 128Sizes/256-12 6.33GB/s ± 3% 8.65GB/s ± 3% +36.71% (p=0.000 n=9+10) 128Sizes/512-12 6.78GB/s ± 3% 8.39GB/s ± 6% +23.77% (p=0.000 n=9+10) 128Sizes/1024-12 7.46GB/s ± 4% 8.70GB/s ± 4% +16.70% (p=0.000 n=9+10) 128Sizes/2048-12 7.99GB/s ± 4% 8.73GB/s ± 8% +9.26% (p=0.003 n=9+10) 128Sizes/4096-12 8.20GB/s ± 2% 8.86GB/s ± 6% +8.00% (p=0.000 n=10+10) 128Sizes/8192-12 8.24GB/s ± 2% 9.01GB/s ± 3% +9.30% (p=0.000 n=8+10) ``` murmur3-1.1.6/go.mod000066400000000000000000000000501410707517500142410ustar00rootroot00000000000000module github.com/twmb/murmur3 go 1.11 murmur3-1.1.6/murmur.go000066400000000000000000000032351410707517500150210ustar00rootroot00000000000000// Copyright 2013, Sébastien Paolacci. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Package murmur3 provides an amd64 native (Go generic fallback) // implementation of the murmur3 hash algorithm for strings and slices. // // Assembly is provided for amd64 go1.5+; pull requests are welcome for other // architectures. package murmur3 import ( "reflect" "unsafe" ) type bmixer interface { bmix(p []byte) (tail []byte) Size() (n int) reset() } type digest struct { clen int // Digested input cumulative length. tail []byte // 0 to Size()-1 bytes view of `buf'. buf [16]byte // Expected (but not required) to be Size() large. bmixer } func (d *digest) BlockSize() int { return 1 } func (d *digest) Write(p []byte) (n int, err error) { n = len(p) d.clen += n if len(d.tail) > 0 { // Stick back pending bytes. nfree := d.Size() - len(d.tail) // nfree ∈ [1, d.Size()-1]. if nfree < len(p) { // One full block can be formed. block := append(d.tail, p[:nfree]...) p = p[nfree:] _ = d.bmix(block) // No tail. } else { // Tail's buf is large enough to prevent reallocs. p = append(d.tail, p...) } } d.tail = d.bmix(p) // Keep own copy of the 0 to Size()-1 pending bytes. nn := copy(d.buf[:], d.tail) d.tail = d.buf[:nn] return n, nil } func (d *digest) Reset() { d.clen = 0 d.tail = nil d.bmixer.reset() } func strslice(slice []byte) string { var str string slicehdr := ((*reflect.SliceHeader)(unsafe.Pointer(&slice))) strhdr := (*reflect.StringHeader)(unsafe.Pointer(&str)) strhdr.Data = slicehdr.Data strhdr.Len = slicehdr.Len return str } murmur3-1.1.6/murmur128.go000066400000000000000000000071141410707517500152540ustar00rootroot00000000000000package murmur3 import ( "hash" "math/bits" ) const ( c1_128 = 0x87c37b91114253d5 c2_128 = 0x4cf5ad432745937f ) // Make sure interfaces are correctly implemented. var ( _ hash.Hash = new(digest128) _ Hash128 = new(digest128) _ bmixer = new(digest128) ) // Hash128 provides an interface for a streaming 128 bit hash. type Hash128 interface { hash.Hash Sum128() (uint64, uint64) } // digest128 represents a partial evaluation of a 128 bites hash. type digest128 struct { digest seed1 uint64 seed2 uint64 h1 uint64 // Unfinalized running hash part 1. h2 uint64 // Unfinalized running hash part 2. } // SeedNew128 returns a Hash128 for streaming 128 bit sums with its internal // digests initialized to seed1 and seed2. // // The canonical implementation allows one only uint32 seed; to imitate that // behavior, use the same, uint32-max seed for seed1 and seed2. func SeedNew128(seed1, seed2 uint64) Hash128 { d := &digest128{seed1: seed1, seed2: seed2} d.bmixer = d d.Reset() return d } // New128 returns a Hash128 for streaming 128 bit sums. func New128() Hash128 { return SeedNew128(0, 0) } func (d *digest128) Size() int { return 16 } func (d *digest128) reset() { d.h1, d.h2 = d.seed1, d.seed2 } func (d *digest128) Sum(b []byte) []byte { h1, h2 := d.Sum128() return append(b, byte(h1>>56), byte(h1>>48), byte(h1>>40), byte(h1>>32), byte(h1>>24), byte(h1>>16), byte(h1>>8), byte(h1), byte(h2>>56), byte(h2>>48), byte(h2>>40), byte(h2>>32), byte(h2>>24), byte(h2>>16), byte(h2>>8), byte(h2), ) } func (d *digest128) bmix(p []byte) (tail []byte) { h1, h2 := d.h1, d.h2 for len(p) >= 16 { k1 := uint64(p[0]) | uint64(p[1])<<8 | uint64(p[2])<<16 | uint64(p[3])<<24 | uint64(p[4])<<32 | uint64(p[5])<<40 | uint64(p[6])<<48 | uint64(p[7])<<56 k2 := uint64(p[8]) | uint64(p[9])<<8 | uint64(p[10])<<16 | uint64(p[11])<<24 | uint64(p[12])<<32 | uint64(p[13])<<40 | uint64(p[14])<<48 | uint64(p[15])<<56 p = p[16:] k1 *= c1_128 k1 = bits.RotateLeft64(k1, 31) k1 *= c2_128 h1 ^= k1 h1 = bits.RotateLeft64(h1, 27) h1 += h2 h1 = h1*5 + 0x52dce729 k2 *= c2_128 k2 = bits.RotateLeft64(k2, 33) k2 *= c1_128 h2 ^= k2 h2 = bits.RotateLeft64(h2, 31) h2 += h1 h2 = h2*5 + 0x38495ab5 } d.h1, d.h2 = h1, h2 return p } func (d *digest128) Sum128() (h1, h2 uint64) { h1, h2 = d.h1, d.h2 var k1, k2 uint64 switch len(d.tail) & 15 { case 15: k2 ^= uint64(d.tail[14]) << 48 fallthrough case 14: k2 ^= uint64(d.tail[13]) << 40 fallthrough case 13: k2 ^= uint64(d.tail[12]) << 32 fallthrough case 12: k2 ^= uint64(d.tail[11]) << 24 fallthrough case 11: k2 ^= uint64(d.tail[10]) << 16 fallthrough case 10: k2 ^= uint64(d.tail[9]) << 8 fallthrough case 9: k2 ^= uint64(d.tail[8]) << 0 k2 *= c2_128 k2 = bits.RotateLeft64(k2, 33) k2 *= c1_128 h2 ^= k2 fallthrough case 8: k1 ^= uint64(d.tail[7]) << 56 fallthrough case 7: k1 ^= uint64(d.tail[6]) << 48 fallthrough case 6: k1 ^= uint64(d.tail[5]) << 40 fallthrough case 5: k1 ^= uint64(d.tail[4]) << 32 fallthrough case 4: k1 ^= uint64(d.tail[3]) << 24 fallthrough case 3: k1 ^= uint64(d.tail[2]) << 16 fallthrough case 2: k1 ^= uint64(d.tail[1]) << 8 fallthrough case 1: k1 ^= uint64(d.tail[0]) << 0 k1 *= c1_128 k1 = bits.RotateLeft64(k1, 31) k1 *= c2_128 h1 ^= k1 } h1 ^= uint64(d.clen) h2 ^= uint64(d.clen) h1 += h2 h2 += h1 h1 = fmix64(h1) h2 = fmix64(h2) h1 += h2 h2 += h1 return h1, h2 } func fmix64(k uint64) uint64 { k ^= k >> 33 k *= 0xff51afd7ed558ccd k ^= k >> 33 k *= 0xc4ceb9fe1a85ec53 k ^= k >> 33 return k } murmur3-1.1.6/murmur128_amd64.s000066400000000000000000000067731410707517500161160ustar00rootroot00000000000000// +build go1.5,amd64 // SeedSum128(seed1, seed2 uint64, data []byte) (h1 uint64, h2 uint64) TEXT ·SeedSum128(SB), $0-56 MOVQ seed1+0(FP), R12 MOVQ seed2+8(FP), R13 MOVQ data_base+16(FP), SI MOVQ data_len+24(FP), R9 LEAQ h1+40(FP), BX JMP sum128internal<>(SB) // Sum128(data []byte) (h1 uint64, h2 uint64) TEXT ·Sum128(SB), $0-40 XORQ R12, R12 XORQ R13, R13 MOVQ data_base+0(FP), SI MOVQ data_len+8(FP), R9 LEAQ h1+24(FP), BX JMP sum128internal<>(SB) // SeedStringSum128(seed1, seed2 uint64, data string) (h1 uint64, h2 uint64) TEXT ·SeedStringSum128(SB), $0-48 MOVQ seed1+0(FP), R12 MOVQ seed2+8(FP), R13 MOVQ data_base+16(FP), SI MOVQ data_len+24(FP), R9 LEAQ h1+32(FP), BX JMP sum128internal<>(SB) // StringSum128(data string) (h1 uint64, h2 uint64) TEXT ·StringSum128(SB), $0-32 XORQ R12, R12 XORQ R13, R13 MOVQ data_base+0(FP), SI MOVQ data_len+8(FP), R9 LEAQ h1+16(FP), BX JMP sum128internal<>(SB) // Expects: // R12 == h1 uint64 seed // R13 == h2 uint64 seed // SI == &data // R9 == len(data) // BX == &[2]uint64 return TEXT sum128internal<>(SB), $0 MOVQ $0x87c37b91114253d5, R14 // c1 MOVQ $0x4cf5ad432745937f, R15 // c2 MOVQ R9, CX ANDQ $-16, CX // cx == data_len - (data_len % 16) // for r10 = 0; r10 < cx; r10 += 16 {... XORQ R10, R10 loop: CMPQ R10, CX JE tail MOVQ (SI)(R10*1), AX MOVQ 8(SI)(R10*1), DX ADDQ $16, R10 IMULQ R14, AX IMULQ R15, DX ROLQ $31, AX ROLQ $33, DX IMULQ R15, AX IMULQ R14, DX XORQ AX, R12 ROLQ $27, R12 ADDQ R13, R12 XORQ DX, R13 ROLQ $31, R13 LEAQ 0x52dce729(R12)(R12*4), R12 ADDQ R12, R13 LEAQ 0x38495ab5(R13)(R13*4), R13 JMP loop tail: MOVQ R9, CX ANDQ $0xf, CX JZ finalize // if len % 16 == 0 XORQ AX, AX // poor man's binary tree jump table SUBQ $8, CX JZ tail8 JG over8 ADDQ $4, CX JZ tail4 JG over4 ADDQ $2, CX JL tail1 JZ tail2 JMP tail3 over4: SUBQ $2, CX JL tail5 JZ tail6 JMP tail7 over8: SUBQ $4, CX JZ tail12 JG over12 ADDQ $2, CX JL tail9 JZ tail10 JMP tail11 over12: SUBQ $2, CX JL tail13 JZ tail14 tail15: MOVBQZX 14(SI)(R10*1), AX SALQ $16, AX tail14: MOVW 12(SI)(R10*1), AX SALQ $32, AX JMP tail12 tail13: MOVBQZX 12(SI)(R10*1), AX SALQ $32, AX tail12: MOVL 8(SI)(R10*1), DX ORQ DX, AX JMP fintailhigh tail11: MOVBQZX 10(SI)(R10*1), AX SALQ $16, AX tail10: MOVW 8(SI)(R10*1), AX JMP fintailhigh tail9: MOVB 8(SI)(R10*1), AL fintailhigh: IMULQ R15, AX ROLQ $33, AX IMULQ R14, AX XORQ AX, R13 tail8: MOVQ (SI)(R10*1), AX JMP fintaillow tail7: MOVBQZX 6(SI)(R10*1), AX SALQ $16, AX tail6: MOVW 4(SI)(R10*1), AX SALQ $32, AX JMP tail4 tail5: MOVBQZX 4(SI)(R10*1), AX SALQ $32, AX tail4: MOVL (SI)(R10*1), DX ORQ DX, AX JMP fintaillow tail3: MOVBQZX 2(SI)(R10*1), AX SALQ $16, AX tail2: MOVW (SI)(R10*1), AX JMP fintaillow tail1: MOVB (SI)(R10*1), AL fintaillow: IMULQ R14, AX ROLQ $31, AX IMULQ R15, AX XORQ AX, R12 finalize: XORQ R9, R12 XORQ R9, R13 ADDQ R13, R12 ADDQ R12, R13 // fmix128 (both interleaved) MOVQ R12, DX MOVQ R13, AX SHRQ $33, DX SHRQ $33, AX XORQ DX, R12 XORQ AX, R13 MOVQ $0xff51afd7ed558ccd, CX IMULQ CX, R12 IMULQ CX, R13 MOVQ R12, DX MOVQ R13, AX SHRQ $33, DX SHRQ $33, AX XORQ DX, R12 XORQ AX, R13 MOVQ $0xc4ceb9fe1a85ec53, CX IMULQ CX, R12 IMULQ CX, R13 MOVQ R12, DX MOVQ R13, AX SHRQ $33, DX SHRQ $33, AX XORQ DX, R12 XORQ AX, R13 ADDQ R13, R12 ADDQ R12, R13 MOVQ R12, (BX) MOVQ R13, 8(BX) RET murmur3-1.1.6/murmur128_decl.go000066400000000000000000000020511410707517500162360ustar00rootroot00000000000000// +build go1.5,amd64 package murmur3 //go:noescape // Sum128 returns the murmur3 sum of data. It is equivalent to the following // sequence (without the extra burden and the extra allocation): // hasher := New128() // hasher.Write(data) // return hasher.Sum128() func Sum128(data []byte) (h1 uint64, h2 uint64) //go:noescape // SeedSum128 returns the murmur3 sum of data with digests initialized to seed1 // and seed2. // // The canonical implementation allows only one uint32 seed; to imitate that // behavior, use the same, uint32-max seed for seed1 and seed2. // // This reads and processes the data in chunks of little endian uint64s; // thus, the returned hashes are portable across architectures. func SeedSum128(seed1, seed2 uint64, data []byte) (h1 uint64, h2 uint64) //go:noescape // StringSum128 is the string version of Sum128. func StringSum128(data string) (h1 uint64, h2 uint64) //go:noescape // SeedStringSum128 is the string version of SeedSum128. func SeedStringSum128(seed1, seed2 uint64, data string) (h1 uint64, h2 uint64) murmur3-1.1.6/murmur128_gen.go000066400000000000000000000060331410707517500161040ustar00rootroot00000000000000// +build !go1.5 !amd64 package murmur3 import "math/bits" // SeedSum128 returns the murmur3 sum of data with digests initialized to seed1 // and seed2. // // The canonical implementation allows only one uint32 seed; to imitate that // behavior, use the same, uint32-max seed for seed1 and seed2. // // This reads and processes the data in chunks of little endian uint64s; // thus, the returned hashes are portable across architectures. func SeedSum128(seed1, seed2 uint64, data []byte) (h1 uint64, h2 uint64) { return SeedStringSum128(seed1, seed2, strslice(data)) } // Sum128 returns the murmur3 sum of data. It is equivalent to the following // sequence (without the extra burden and the extra allocation): // hasher := New128() // hasher.Write(data) // return hasher.Sum128() func Sum128(data []byte) (h1 uint64, h2 uint64) { return SeedStringSum128(0, 0, strslice(data)) } // StringSum128 is the string version of Sum128. func StringSum128(data string) (h1 uint64, h2 uint64) { return SeedStringSum128(0, 0, data) } // SeedStringSum128 is the string version of SeedSum128. func SeedStringSum128(seed1, seed2 uint64, data string) (h1 uint64, h2 uint64) { h1, h2 = seed1, seed2 clen := len(data) for len(data) >= 16 { // yes, this is faster than using binary.LittleEndian.Uint64 k1 := uint64(data[0]) | uint64(data[1])<<8 | uint64(data[2])<<16 | uint64(data[3])<<24 | uint64(data[4])<<32 | uint64(data[5])<<40 | uint64(data[6])<<48 | uint64(data[7])<<56 k2 := uint64(data[8]) | uint64(data[9])<<8 | uint64(data[10])<<16 | uint64(data[11])<<24 | uint64(data[12])<<32 | uint64(data[13])<<40 | uint64(data[14])<<48 | uint64(data[15])<<56 data = data[16:] k1 *= c1_128 k1 = bits.RotateLeft64(k1, 31) k1 *= c2_128 h1 ^= k1 h1 = bits.RotateLeft64(h1, 27) h1 += h2 h1 = h1*5 + 0x52dce729 k2 *= c2_128 k2 = bits.RotateLeft64(k2, 33) k2 *= c1_128 h2 ^= k2 h2 = bits.RotateLeft64(h2, 31) h2 += h1 h2 = h2*5 + 0x38495ab5 } var k1, k2 uint64 switch len(data) { case 15: k2 ^= uint64(data[14]) << 48 fallthrough case 14: k2 ^= uint64(data[13]) << 40 fallthrough case 13: k2 ^= uint64(data[12]) << 32 fallthrough case 12: k2 ^= uint64(data[11]) << 24 fallthrough case 11: k2 ^= uint64(data[10]) << 16 fallthrough case 10: k2 ^= uint64(data[9]) << 8 fallthrough case 9: k2 ^= uint64(data[8]) << 0 k2 *= c2_128 k2 = bits.RotateLeft64(k2, 33) k2 *= c1_128 h2 ^= k2 fallthrough case 8: k1 ^= uint64(data[7]) << 56 fallthrough case 7: k1 ^= uint64(data[6]) << 48 fallthrough case 6: k1 ^= uint64(data[5]) << 40 fallthrough case 5: k1 ^= uint64(data[4]) << 32 fallthrough case 4: k1 ^= uint64(data[3]) << 24 fallthrough case 3: k1 ^= uint64(data[2]) << 16 fallthrough case 2: k1 ^= uint64(data[1]) << 8 fallthrough case 1: k1 ^= uint64(data[0]) << 0 k1 *= c1_128 k1 = bits.RotateLeft64(k1, 31) k1 *= c2_128 h1 ^= k1 } h1 ^= uint64(clen) h2 ^= uint64(clen) h1 += h2 h2 += h1 h1 = fmix64(h1) h2 = fmix64(h2) h1 += h2 h2 += h1 return h1, h2 } murmur3-1.1.6/murmur32.go000066400000000000000000000035211410707517500151640ustar00rootroot00000000000000package murmur3 import ( "hash" "math/bits" ) // Make sure interfaces are correctly implemented. var ( _ hash.Hash = new(digest32) _ hash.Hash32 = new(digest32) ) const ( c1_32 uint32 = 0xcc9e2d51 c2_32 uint32 = 0x1b873593 ) // digest32 represents a partial evaluation of a 32 bites hash. type digest32 struct { digest seed uint32 h1 uint32 // Unfinalized running hash. } // SeedNew32 returns a hash.Hash32 for streaming 32 bit sums with its internal // digest initialized to seed. // // This reads and processes the data in chunks of little endian uint32s; // thus, the returned hash is portable across architectures. func SeedNew32(seed uint32) hash.Hash32 { d := &digest32{seed: seed} d.bmixer = d d.Reset() return d } // New32 returns a hash.Hash32 for streaming 32 bit sums. func New32() hash.Hash32 { return SeedNew32(0) } func (d *digest32) Size() int { return 4 } func (d *digest32) reset() { d.h1 = d.seed } func (d *digest32) Sum(b []byte) []byte { h := d.Sum32() return append(b, byte(h>>24), byte(h>>16), byte(h>>8), byte(h)) } // Digest as many blocks as possible. func (d *digest32) bmix(p []byte) (tail []byte) { h1 := d.h1 for len(p) >= 4 { k1 := uint32(p[0]) | uint32(p[1])<<8 | uint32(p[2])<<16 | uint32(p[3])<<24 p = p[4:] k1 *= c1_32 k1 = bits.RotateLeft32(k1, 15) k1 *= c2_32 h1 ^= k1 h1 = bits.RotateLeft32(h1, 13) h1 = h1*5 + 0xe6546b64 } d.h1 = h1 return p } func (d *digest32) Sum32() (h1 uint32) { h1 = d.h1 var k1 uint32 switch len(d.tail) & 3 { case 3: k1 ^= uint32(d.tail[2]) << 16 fallthrough case 2: k1 ^= uint32(d.tail[1]) << 8 fallthrough case 1: k1 ^= uint32(d.tail[0]) k1 *= c1_32 k1 = bits.RotateLeft32(k1, 15) k1 *= c2_32 h1 ^= k1 } h1 ^= uint32(d.clen) h1 ^= h1 >> 16 h1 *= 0x85ebca6b h1 ^= h1 >> 13 h1 *= 0xc2b2ae35 h1 ^= h1 >> 16 return h1 } murmur3-1.1.6/murmur32_gen.go000066400000000000000000000030431410707517500160140ustar00rootroot00000000000000package murmur3 import "math/bits" // SeedSum32 returns the murmur3 sum of data with the digest initialized to // seed. // // This reads and processes the data in chunks of little endian uint32s; // thus, the returned hash is portable across architectures. func SeedSum32(seed uint32, data []byte) (h1 uint32) { return SeedStringSum32(seed, strslice(data)) } // Sum32 returns the murmur3 sum of data. It is equivalent to the following // sequence (without the extra burden and the extra allocation): // hasher := New32() // hasher.Write(data) // return hasher.Sum32() func Sum32(data []byte) uint32 { return SeedStringSum32(0, strslice(data)) } // StringSum32 is the string version of Sum32. func StringSum32(data string) uint32 { return SeedStringSum32(0, data) } // SeedStringSum32 is the string version of SeedSum32. func SeedStringSum32(seed uint32, data string) (h1 uint32) { h1 = seed clen := uint32(len(data)) for len(data) >= 4 { k1 := uint32(data[0]) | uint32(data[1])<<8 | uint32(data[2])<<16 | uint32(data[3])<<24 data = data[4:] k1 *= c1_32 k1 = bits.RotateLeft32(k1, 15) k1 *= c2_32 h1 ^= k1 h1 = bits.RotateLeft32(h1, 13) h1 = h1*5 + 0xe6546b64 } var k1 uint32 switch len(data) { case 3: k1 ^= uint32(data[2]) << 16 fallthrough case 2: k1 ^= uint32(data[1]) << 8 fallthrough case 1: k1 ^= uint32(data[0]) k1 *= c1_32 k1 = bits.RotateLeft32(k1, 15) k1 *= c2_32 h1 ^= k1 } h1 ^= uint32(clen) h1 ^= h1 >> 16 h1 *= 0x85ebca6b h1 ^= h1 >> 13 h1 *= 0xc2b2ae35 h1 ^= h1 >> 16 return h1 } murmur3-1.1.6/murmur64.go000066400000000000000000000033641410707517500151760ustar00rootroot00000000000000package murmur3 import ( "hash" ) // Make sure interfaces are correctly implemented. var ( _ hash.Hash = new(digest64) _ hash.Hash64 = new(digest64) _ bmixer = new(digest64) ) // digest64 is half a digest128. type digest64 digest128 // SeedNew64 returns a hash.Hash64 for streaming 64 bit sums. As the canonical // implementation does not support Sum64, this uses SeedNew128(seed, seed) func SeedNew64(seed uint64) hash.Hash64 { return (*digest64)(SeedNew128(seed, seed).(*digest128)) } // New64 returns a hash.Hash64 for streaming 64 bit sums. func New64() hash.Hash64 { return SeedNew64(0) } func (d *digest64) Sum(b []byte) []byte { h1 := d.Sum64() return append(b, byte(h1>>56), byte(h1>>48), byte(h1>>40), byte(h1>>32), byte(h1>>24), byte(h1>>16), byte(h1>>8), byte(h1)) } func (d *digest64) Sum64() uint64 { h1, _ := (*digest128)(d).Sum128() return h1 } // Sum64 returns the murmur3 sum of data. It is equivalent to the following // sequence (without the extra burden and the extra allocation): // hasher := New64() // hasher.Write(data) // return hasher.Sum64() func Sum64(data []byte) uint64 { h1, _ := Sum128(data) return h1 } // SeedSum64 returns the murmur3 sum of data with the digest initialized to // seed. // // Because the canonical implementation does not support SeedSum64, this uses // SeedSum128(seed, seed, data). func SeedSum64(seed uint64, data []byte) uint64 { h1, _ := SeedSum128(seed, seed, data) return h1 } // StringSum64 is the string version of Sum64. func StringSum64(data string) uint64 { h1, _ := StringSum128(data) return h1 } // SeedStringSum64 is the string version of SeedSum64. func SeedStringSum64(seed uint64, data string) uint64 { h1, _ := SeedStringSum128(seed, seed, data) return h1 } murmur3-1.1.6/murmur_test.go000066400000000000000000000263601410707517500160640ustar00rootroot00000000000000package murmur3 import ( "crypto/rand" "encoding/binary" "fmt" "hash" "io" "strconv" "testing" "testing/quick" "unsafe" "github.com/twmb/murmur3/testdata" ) var isLittleEndian = func() bool { i := uint16(1) return (*(*[2]byte)(unsafe.Pointer(&i)))[0] == 1 }() var data = []struct { h32 uint32 h64_1 uint64 h64_2 uint64 s string }{ {0x00000000, 0x0000000000000000, 0x0000000000000000, ""}, {0x248bfa47, 0xcbd8a7b341bd9b02, 0x5b1e906a48ae1d19, "hello"}, {0x149bbb7f, 0x342fac623a5ebc8e, 0x4cdcbc079642414d, "hello, world"}, {0xe31e8a70, 0xb89e5988b737affc, 0x664fc2950231b2cb, "19 Jan 2038 at 3:14:07 AM"}, {0xd5c48bfc, 0xcd99481f9ee902c9, 0x695da1a38987b6e7, "The quick brown fox jumps over the lazy dog."}, } func TestRef(t *testing.T) { for _, elem := range data { var h32 hash.Hash32 = New32() h32.Write([]byte(elem.s)) if v := h32.Sum32(); v != elem.h32 { t.Errorf("'%s': 0x%x (want 0x%x)", elem.s, v, elem.h32) } var h32_byte hash.Hash32 = New32() h32_byte.Write([]byte(elem.s)) target := fmt.Sprintf("%08x", elem.h32) if p := fmt.Sprintf("%x", h32_byte.Sum(nil)); p != target { t.Errorf("'%s': %s (want %s)", elem.s, p, target) } if v := Sum32([]byte(elem.s)); v != elem.h32 { t.Errorf("'%s': 0x%x (want 0x%x)", elem.s, v, elem.h32) } var h64 hash.Hash64 = New64() h64.Write([]byte(elem.s)) if v := h64.Sum64(); v != elem.h64_1 { t.Errorf("'%s': 0x%x (want 0x%x)", elem.s, v, elem.h64_1) } var h64_byte hash.Hash64 = New64() h64_byte.Write([]byte(elem.s)) target = fmt.Sprintf("%016x", elem.h64_1) if p := fmt.Sprintf("%x", h64_byte.Sum(nil)); p != target { t.Errorf("Sum64: '%s': %s (want %s)", elem.s, p, target) } if v := Sum64([]byte(elem.s)); v != elem.h64_1 { t.Errorf("Sum64: '%s': 0x%x (want 0x%x)", elem.s, v, elem.h64_1) } var h128 Hash128 = New128() h128.Write([]byte(elem.s)) if v1, v2 := h128.Sum128(); v1 != elem.h64_1 || v2 != elem.h64_2 { t.Errorf("New128: '%s': 0x%x-0x%x (want 0x%x-0x%x)", elem.s, v1, v2, elem.h64_1, elem.h64_2) } var h128_byte Hash128 = New128() h128_byte.Write([]byte(elem.s)) target = fmt.Sprintf("%016x%016x", elem.h64_1, elem.h64_2) if p := fmt.Sprintf("%x", h128_byte.Sum(nil)); p != target { t.Errorf("New128: '%s': %s (want %s)", elem.s, p, target) } if v1, v2 := Sum128([]byte(elem.s)); v1 != elem.h64_1 || v2 != elem.h64_2 { t.Errorf("Sum128: '%s': 0x%x-0x%x (want 0x%x-0x%x)", elem.s, v1, v2, elem.h64_1, elem.h64_2) } } } func TestQuickSum32(t *testing.T) { f := func(data []byte) bool { goh1 := Sum32(data) goh2 := StringSum32(string(data)) cpph1 := goh1 if isLittleEndian { cpph1 = testdata.SeedSum32(0, data) } return goh1 == goh2 && goh1 == cpph1 } if err := quick.Check(f, nil); err != nil { t.Error(err) } } func TestQuickSeedSum32(t *testing.T) { f := func(seed uint32, data []byte) bool { goh1 := SeedSum32(seed, data) goh2 := SeedStringSum32(seed, string(data)) goh3 := func() uint32 { h := SeedNew32(seed); h.Write(data); return binary.BigEndian.Uint32(h.Sum(nil)) }() cpph1 := goh1 if isLittleEndian { cpph1 = testdata.SeedSum32(seed, data) } return goh1 == goh2 && goh1 == goh3 && goh1 == cpph1 } if err := quick.Check(f, nil); err != nil { t.Error(err) } } func TestQuickSum64(t *testing.T) { f := func(data []byte) bool { goh1 := Sum64(data) goh2 := StringSum64(string(data)) cpph1 := goh1 if isLittleEndian { cpph1 = testdata.SeedSum64(0, data) } return goh1 == goh2 && goh1 == cpph1 } if err := quick.Check(f, nil); err != nil { t.Error(err) } } func TestQuickSeedSum64(t *testing.T) { f := func(seed uint32, data []byte) bool { goh1 := SeedSum64(uint64(seed), data) goh2 := SeedStringSum64(uint64(seed), string(data)) goh3 := func() uint64 { h := SeedNew64(uint64(seed)); h.Write(data); return binary.BigEndian.Uint64(h.Sum(nil)) }() cpph1 := goh1 if isLittleEndian { cpph1 = testdata.SeedSum64(seed, data) } return goh1 == goh2 && goh1 == goh3 && goh1 == cpph1 } if err := quick.Check(f, nil); err != nil { t.Error(err) } } func TestQuickSum128(t *testing.T) { f := func(data []byte) bool { goh1, goh2 := Sum128(data) goh3, goh4 := StringSum128(string(data)) cpph1, cpph2 := goh1, goh2 if isLittleEndian { cpph1, cpph2 = testdata.SeedSum128(0, data) } return goh1 == goh3 && goh2 == goh4 && goh1 == cpph1 && goh2 == cpph2 } if err := quick.Check(f, nil); err != nil { t.Error(err) } } func TestQuickSeedSum128(t *testing.T) { f := func(seed uint32, data []byte) bool { goh1, goh2 := SeedSum128(uint64(seed), uint64(seed), data) goh3, goh4 := SeedStringSum128(uint64(seed), uint64(seed), string(data)) goh5, goh6 := func() (uint64, uint64) { h := SeedNew128(uint64(seed), uint64(seed)) h.Write(data) sum := h.Sum(nil) return binary.BigEndian.Uint64(sum), binary.BigEndian.Uint64(sum[8:]) }() cpph1, cpph2 := goh1, goh2 if isLittleEndian { testdata.SeedSum128(seed, data) } return goh1 == goh3 && goh2 == goh4 && goh1 == goh5 && goh2 == goh6 && goh1 == cpph1 && goh2 == cpph2 } if err := quick.Check(f, nil); err != nil { t.Error(err) } } // go1.14 showed that doing *(*uint32)(unsafe.Pointer(&data[i*4])) was unsafe // due to alignment issues; this test ensures that we will always catch that. func TestUnaligned(t *testing.T) { in1 := []byte("abcdefghijklmnopqrstuvwxyz") in2 := []byte("_abcdefghijklmnopqrstuvwxyz") { sum1 := Sum32(in1) sum2 := Sum32(in2[1:]) if sum1 != sum2 { t.Errorf("%s: got sum1 %v sum2 %v unexpectedly not equal", "Sum32", sum1, sum2) } } { sum1 := Sum64(in1) sum2 := Sum64(in2[1:]) if sum1 != sum2 { t.Errorf("%s: got sum1 %v sum2 %v unexpectedly not equal", "Sum64", sum1, sum2) } } { sum1l, sum1r := Sum128(in1) sum2l, sum2r := Sum128(in2[1:]) if sum1l != sum2l { t.Errorf("%s: got sum1l %v sum2l %v unexpectedly not equal", "Sum128 left", sum1l, sum2l) } if sum1r != sum2r { t.Errorf("%s: got sum1r %v sum2r %v unexpectedly not equal", "Sum128 right", sum1r, sum2r) } } { sum1 := func() uint32 { n := New32(); n.Write(in1); return n.Sum32() }() sum2 := func() uint32 { n := New32(); n.Write(in2[1:]); return n.Sum32() }() if sum1 != sum2 { t.Errorf("%s: got sum1 %v sum2 %v unexpectedly not equal", "New32", sum1, sum2) } } { sum1 := func() uint64 { n := New64(); n.Write(in1); return n.Sum64() }() sum2 := func() uint64 { n := New64(); n.Write(in2[1:]); return n.Sum64() }() if sum1 != sum2 { t.Errorf("%s: got sum1 %v sum2 %v unexpectedly not equal", "New64", sum1, sum2) } } } // TestBoundaries forces every block/tail path to be exercised for Sum32 and Sum128. func TestBoundaries(t *testing.T) { const maxCheck = 17 var data [maxCheck]byte for i := 0; !t.Failed() && i < 20; i++ { // Check all zeros the first iteration. for size := 0; size <= maxCheck; size++ { test := data[:size] g32h1 := Sum32(test) g32h1s := SeedSum32(0, test) c32h1 := g32h1 if isLittleEndian { c32h1 = testdata.SeedSum32(0, test) } if g32h1 != c32h1 { t.Errorf("size #%d: in: %x, g32h1 (%d) != c32h1 (%d); attempt #%d", size, test, g32h1, c32h1, i) } if g32h1s != c32h1 { t.Errorf("size #%d: in: %x, gh32h1s (%d) != c32h1 (%d); attempt #%d", size, test, g32h1s, c32h1, i) } g64h1 := Sum64(test) g64h1s := SeedSum64(0, test) c64h1 := g64h1 if isLittleEndian { c64h1 = testdata.SeedSum64(0, test) } if g64h1 != c64h1 { t.Errorf("size #%d: in: %x, g64h1 (%d) != c64h1 (%d); attempt #%d", size, test, g64h1, c64h1, i) } if g64h1s != c64h1 { t.Errorf("size #%d: in: %x, g64h1s (%d) != c64h1 (%d); attempt #%d", size, test, g64h1s, c64h1, i) } g128h1, g128h2 := Sum128(test) g128h1s, g128h2s := SeedSum128(0, 0, test) c128h1, c128h2 := g128h1, g128h2 if isLittleEndian { c128h1, c128h2 = testdata.SeedSum128(0, test) } if g128h1 != c128h1 { t.Errorf("size #%d: in: %x, g128h1 (%d) != c128h1 (%d); attempt #%d", size, test, g128h1, c128h1, i) } if g128h2 != c128h2 { t.Errorf("size #%d: in: %x, g128h2 (%d) != c128h2 (%d); attempt #%d", size, test, g128h2, c128h2, i) } if g128h1s != c128h1 { t.Errorf("size #%d: in: %x, g128h1s (%d) != c128h1 (%d); attempt #%d", size, test, g128h1s, c128h1, i) } if g128h2s != c128h2 { t.Errorf("size #%d: in: %x, g128h2s (%d) != c128h2 (%d); attempt #%d", size, test, g128h2s, c128h2, i) } } // Randomize the data for all subsequent tests. io.ReadFull(rand.Reader, data[:]) } } func TestIncremental(t *testing.T) { for _, elem := range data { h32 := New32() h128 := New128() for i, j, k := 0, 0, len(elem.s); i < k; i = j { j = 2*i + 3 if j > k { j = k } s := elem.s[i:j] print(s + "|") h32.Write([]byte(s)) h128.Write([]byte(s)) } println() if v := h32.Sum32(); v != elem.h32 { t.Errorf("'%s': 0x%x (want 0x%x)", elem.s, v, elem.h32) } if v1, v2 := h128.Sum128(); v1 != elem.h64_1 || v2 != elem.h64_2 { t.Errorf("'%s': 0x%x-0x%x (want 0x%x-0x%x)", elem.s, v1, v2, elem.h64_1, elem.h64_2) } } } // Our lengths force 1) the function base itself (no loop/tail), 2) remainders // and 3) the loop itself. func Benchmark32Branches(b *testing.B) { for length := 0; length <= 4; length++ { b.Run(strconv.Itoa(length), func(b *testing.B) { buf := make([]byte, length) b.SetBytes(int64(length)) b.ResetTimer() for length := 0; length < b.N; length++ { Sum32(buf) } }) } } func BenchmarkPartial32Branches(b *testing.B) { for length := 0; length <= 4; length++ { b.Run(strconv.Itoa(length), func(b *testing.B) { buf := make([]byte, length) b.SetBytes(int64(length)) b.ResetTimer() for i := 0; i < b.N; i++ { hasher := New32() hasher.Write(buf) hasher.Sum32() } }) } } func Benchmark128Branches(b *testing.B) { for length := 0; length <= 16; length++ { b.Run(strconv.Itoa(length), func(b *testing.B) { buf := make([]byte, length) b.SetBytes(int64(length)) b.ResetTimer() for i := 0; i < b.N; i++ { Sum128(buf) } }) } } // Sizes below pick up where branches left off to demonstrate speed at larger // slice sizes. func Benchmark32Sizes(b *testing.B) { buf := make([]byte, 8192) for length := 32; length <= cap(buf); length *= 2 { b.Run(strconv.Itoa(length), func(b *testing.B) { buf = buf[:length] b.SetBytes(int64(length)) b.ResetTimer() for i := 0; i < b.N; i++ { Sum32(buf) } }) } } func Benchmark64Sizes(b *testing.B) { buf := make([]byte, 8192) for length := 32; length <= cap(buf); length *= 2 { b.Run(strconv.Itoa(length), func(b *testing.B) { buf = buf[:length] b.SetBytes(int64(length)) b.ResetTimer() for i := 0; i < b.N; i++ { Sum64(buf) } }) } } func Benchmark128Sizes(b *testing.B) { buf := make([]byte, 8192) for length := 32; length <= cap(buf); length *= 2 { b.Run(strconv.Itoa(length), func(b *testing.B) { buf = buf[:length] b.SetBytes(int64(length)) b.ResetTimer() for i := 0; i < b.N; i++ { Sum128(buf) } }) } } func BenchmarkNoescape32(b *testing.B) { for i := 0; i < b.N; i++ { var buf [8192]byte Sum32(buf[:]) } } func BenchmarkNoescape128(b *testing.B) { for i := 0; i < b.N; i++ { var buf [8192]byte Sum128(buf[:]) } } func BenchmarkStrslice(b *testing.B) { var s []byte for i := 0; i < b.N; i++ { strslice(s) } } murmur3-1.1.6/testdata/000077500000000000000000000000001410707517500147515ustar00rootroot00000000000000murmur3-1.1.6/testdata/MurmurHash3.cpp000066400000000000000000000173351410707517500176440ustar00rootroot00000000000000//----------------------------------------------------------------------------- // MurmurHash3 was written by Austin Appleby, and is placed in the public // domain. The author hereby disclaims copyright to this source code. // Note - The x86 and x64 versions do _not_ produce the same results, as the // algorithms are optimized for their respective platforms. You can still // compile and run any of them on any platform, but your performance with the // non-native version will be less than optimal. #include "MurmurHash3.h" //----------------------------------------------------------------------------- // Platform-specific functions and macros // Microsoft Visual Studio #if defined(_MSC_VER) #define FORCE_INLINE __forceinline #include #define ROTL32(x,y) _rotl(x,y) #define ROTL64(x,y) _rotl64(x,y) #define BIG_CONSTANT(x) (x) // Other compilers #else // defined(_MSC_VER) #define FORCE_INLINE inline __attribute__((always_inline)) inline uint32_t rotl32 ( uint32_t x, int8_t r ) { return (x << r) | (x >> (32 - r)); } inline uint64_t rotl64 ( uint64_t x, int8_t r ) { return (x << r) | (x >> (64 - r)); } #define ROTL32(x,y) rotl32(x,y) #define ROTL64(x,y) rotl64(x,y) #define BIG_CONSTANT(x) (x##LLU) #endif // !defined(_MSC_VER) //----------------------------------------------------------------------------- // Block read - if your platform needs to do endian-swapping or can only // handle aligned reads, do the conversion here FORCE_INLINE uint32_t getblock32 ( const uint32_t * p, int i ) { return p[i]; } FORCE_INLINE uint64_t getblock64 ( const uint64_t * p, int i ) { return p[i]; } //----------------------------------------------------------------------------- // Finalization mix - force all bits of a hash block to avalanche FORCE_INLINE uint32_t fmix32 ( uint32_t h ) { h ^= h >> 16; h *= 0x85ebca6b; h ^= h >> 13; h *= 0xc2b2ae35; h ^= h >> 16; return h; } //---------- FORCE_INLINE uint64_t fmix64 ( uint64_t k ) { k ^= k >> 33; k *= BIG_CONSTANT(0xff51afd7ed558ccd); k ^= k >> 33; k *= BIG_CONSTANT(0xc4ceb9fe1a85ec53); k ^= k >> 33; return k; } //----------------------------------------------------------------------------- void MurmurHash3_x86_32 ( const void * key, int len, uint32_t seed, void * out ) { const uint8_t * data = (const uint8_t*)key; const int nblocks = len / 4; uint32_t h1 = seed; const uint32_t c1 = 0xcc9e2d51; const uint32_t c2 = 0x1b873593; //---------- // body const uint32_t * blocks = (const uint32_t *)(data + nblocks*4); for(int i = -nblocks; i; i++) { uint32_t k1 = getblock32(blocks,i); k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1; h1 = ROTL32(h1,13); h1 = h1*5+0xe6546b64; } //---------- // tail const uint8_t * tail = (const uint8_t*)(data + nblocks*4); uint32_t k1 = 0; switch(len & 3) { case 3: k1 ^= tail[2] << 16; case 2: k1 ^= tail[1] << 8; case 1: k1 ^= tail[0]; k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1; }; //---------- // finalization h1 ^= len; h1 = fmix32(h1); *(uint32_t*)out = h1; } //----------------------------------------------------------------------------- void MurmurHash3_x86_128 ( const void * key, const int len, uint32_t seed, void * out ) { const uint8_t * data = (const uint8_t*)key; const int nblocks = len / 16; uint32_t h1 = seed; uint32_t h2 = seed; uint32_t h3 = seed; uint32_t h4 = seed; const uint32_t c1 = 0x239b961b; const uint32_t c2 = 0xab0e9789; const uint32_t c3 = 0x38b34ae5; const uint32_t c4 = 0xa1e38b93; //---------- // body const uint32_t * blocks = (const uint32_t *)(data + nblocks*16); for(int i = -nblocks; i; i++) { uint32_t k1 = getblock32(blocks,i*4+0); uint32_t k2 = getblock32(blocks,i*4+1); uint32_t k3 = getblock32(blocks,i*4+2); uint32_t k4 = getblock32(blocks,i*4+3); k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1; h1 = ROTL32(h1,19); h1 += h2; h1 = h1*5+0x561ccd1b; k2 *= c2; k2 = ROTL32(k2,16); k2 *= c3; h2 ^= k2; h2 = ROTL32(h2,17); h2 += h3; h2 = h2*5+0x0bcaa747; k3 *= c3; k3 = ROTL32(k3,17); k3 *= c4; h3 ^= k3; h3 = ROTL32(h3,15); h3 += h4; h3 = h3*5+0x96cd1c35; k4 *= c4; k4 = ROTL32(k4,18); k4 *= c1; h4 ^= k4; h4 = ROTL32(h4,13); h4 += h1; h4 = h4*5+0x32ac3b17; } //---------- // tail const uint8_t * tail = (const uint8_t*)(data + nblocks*16); uint32_t k1 = 0; uint32_t k2 = 0; uint32_t k3 = 0; uint32_t k4 = 0; switch(len & 15) { case 15: k4 ^= tail[14] << 16; case 14: k4 ^= tail[13] << 8; case 13: k4 ^= tail[12] << 0; k4 *= c4; k4 = ROTL32(k4,18); k4 *= c1; h4 ^= k4; case 12: k3 ^= tail[11] << 24; case 11: k3 ^= tail[10] << 16; case 10: k3 ^= tail[ 9] << 8; case 9: k3 ^= tail[ 8] << 0; k3 *= c3; k3 = ROTL32(k3,17); k3 *= c4; h3 ^= k3; case 8: k2 ^= tail[ 7] << 24; case 7: k2 ^= tail[ 6] << 16; case 6: k2 ^= tail[ 5] << 8; case 5: k2 ^= tail[ 4] << 0; k2 *= c2; k2 = ROTL32(k2,16); k2 *= c3; h2 ^= k2; case 4: k1 ^= tail[ 3] << 24; case 3: k1 ^= tail[ 2] << 16; case 2: k1 ^= tail[ 1] << 8; case 1: k1 ^= tail[ 0] << 0; k1 *= c1; k1 = ROTL32(k1,15); k1 *= c2; h1 ^= k1; }; //---------- // finalization h1 ^= len; h2 ^= len; h3 ^= len; h4 ^= len; h1 += h2; h1 += h3; h1 += h4; h2 += h1; h3 += h1; h4 += h1; h1 = fmix32(h1); h2 = fmix32(h2); h3 = fmix32(h3); h4 = fmix32(h4); h1 += h2; h1 += h3; h1 += h4; h2 += h1; h3 += h1; h4 += h1; ((uint32_t*)out)[0] = h1; ((uint32_t*)out)[1] = h2; ((uint32_t*)out)[2] = h3; ((uint32_t*)out)[3] = h4; } //----------------------------------------------------------------------------- void MurmurHash3_x64_128 ( const void * key, const int len, const uint32_t seed, void * out ) { const uint8_t * data = (const uint8_t*)key; const int nblocks = len / 16; uint64_t h1 = seed; uint64_t h2 = seed; const uint64_t c1 = BIG_CONSTANT(0x87c37b91114253d5); const uint64_t c2 = BIG_CONSTANT(0x4cf5ad432745937f); //---------- // body const uint64_t * blocks = (const uint64_t *)(data); for(int i = 0; i < nblocks; i++) { uint64_t k1 = getblock64(blocks,i*2+0); uint64_t k2 = getblock64(blocks,i*2+1); k1 *= c1; k1 = ROTL64(k1,31); k1 *= c2; h1 ^= k1; h1 = ROTL64(h1,27); h1 += h2; h1 = h1*5+0x52dce729; k2 *= c2; k2 = ROTL64(k2,33); k2 *= c1; h2 ^= k2; h2 = ROTL64(h2,31); h2 += h1; h2 = h2*5+0x38495ab5; } //---------- // tail const uint8_t * tail = (const uint8_t*)(data + nblocks*16); uint64_t k1 = 0; uint64_t k2 = 0; switch(len & 15) { case 15: k2 ^= ((uint64_t)tail[14]) << 48; case 14: k2 ^= ((uint64_t)tail[13]) << 40; case 13: k2 ^= ((uint64_t)tail[12]) << 32; case 12: k2 ^= ((uint64_t)tail[11]) << 24; case 11: k2 ^= ((uint64_t)tail[10]) << 16; case 10: k2 ^= ((uint64_t)tail[ 9]) << 8; case 9: k2 ^= ((uint64_t)tail[ 8]) << 0; k2 *= c2; k2 = ROTL64(k2,33); k2 *= c1; h2 ^= k2; case 8: k1 ^= ((uint64_t)tail[ 7]) << 56; case 7: k1 ^= ((uint64_t)tail[ 6]) << 48; case 6: k1 ^= ((uint64_t)tail[ 5]) << 40; case 5: k1 ^= ((uint64_t)tail[ 4]) << 32; case 4: k1 ^= ((uint64_t)tail[ 3]) << 24; case 3: k1 ^= ((uint64_t)tail[ 2]) << 16; case 2: k1 ^= ((uint64_t)tail[ 1]) << 8; case 1: k1 ^= ((uint64_t)tail[ 0]) << 0; k1 *= c1; k1 = ROTL64(k1,31); k1 *= c2; h1 ^= k1; }; //---------- // finalization h1 ^= len; h2 ^= len; h1 += h2; h2 += h1; h1 = fmix64(h1); h2 = fmix64(h2); h1 += h2; h2 += h1; ((uint64_t*)out)[0] = h1; ((uint64_t*)out)[1] = h2; } //----------------------------------------------------------------------------- murmur3-1.1.6/testdata/MurmurHash3.h000066400000000000000000000021221410707517500172750ustar00rootroot00000000000000//----------------------------------------------------------------------------- // MurmurHash3 was written by Austin Appleby, and is placed in the public // domain. The author hereby disclaims copyright to this source code. #ifndef _MURMURHASH3_H_ #define _MURMURHASH3_H_ //----------------------------------------------------------------------------- // Platform-specific functions and macros // Microsoft Visual Studio #if defined(_MSC_VER) && (_MSC_VER < 1600) typedef unsigned char uint8_t; typedef unsigned int uint32_t; typedef unsigned __int64 uint64_t; // Other compilers #else // defined(_MSC_VER) #include #endif // !defined(_MSC_VER) //----------------------------------------------------------------------------- void MurmurHash3_x86_32 ( const void * key, int len, uint32_t seed, void * out ); void MurmurHash3_x86_128 ( const void * key, int len, uint32_t seed, void * out ); void MurmurHash3_x64_128 ( const void * key, int len, uint32_t seed, void * out ); //----------------------------------------------------------------------------- #endif // _MURMURHASH3_H_ murmur3-1.1.6/testdata/test.go000066400000000000000000000016761410707517500162710ustar00rootroot00000000000000package testdata // #cgo CFLAGS: -std=gnu99 // #include // #include "MurmurHash3.cpp" // #include "MurmurHash3.h" import "C" import "unsafe" func SeedSum32(seed uint32, data []byte) uint32 { var p unsafe.Pointer if len(data) > 0 { p = unsafe.Pointer(&data[0]) } var out uint32 C.MurmurHash3_x86_32(p, C.int(len(data)), C.uint32_t(seed), unsafe.Pointer(&out)) return out } func SeedSum64(seed uint32, data []byte) uint64 { var p unsafe.Pointer if len(data) > 0 { p = unsafe.Pointer(&data[0]) } var out struct { h1 uint64 h2 uint64 } C.MurmurHash3_x64_128(p, C.int(len(data)), C.uint32_t(seed), unsafe.Pointer(&out)) return out.h1 } func SeedSum128(seed uint32, data []byte) (h1, h2 uint64) { var p unsafe.Pointer if len(data) > 0 { p = unsafe.Pointer(&data[0]) } var out struct { h1 uint64 h2 uint64 } C.MurmurHash3_x64_128(p, C.int(len(data)), C.uint32_t(seed), unsafe.Pointer(&out)) return out.h1, out.h2 }