pax_global_header00006660000000000000000000000064127040607120014511gustar00rootroot0000000000000052 comment=d72a1d25b3c88ff0ace49f15a9b2e2bf3f453666 golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/000077500000000000000000000000001270406071200230505ustar00rootroot00000000000000golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/LICENSE000066400000000000000000000027141270406071200240610ustar00rootroot00000000000000Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of the author nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/README.md000066400000000000000000000006421270406071200243310ustar00rootroot00000000000000# go Collection of Open-Source Go libraries and tools. ## Codec [Codec](https://github.com/ugorji/go/tree/master/codec#readme) is a High Performance and Feature-Rich Idiomatic encode/decode and rpc library for [msgpack](http://msgpack.org) and [Binc](https://github.com/ugorji/binc). Online documentation is at [http://godoc.org/github.com/ugorji/go/codec]. Install using: go get github.com/ugorji/go/codec golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/000077500000000000000000000000001270406071200241255ustar00rootroot00000000000000golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/0doc.go000066400000000000000000000115371270406071200253100ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. /* High Performance, Feature-Rich Idiomatic Go encoding library for msgpack and binc . Supported Serialization formats are: - msgpack: [https://github.com/msgpack/msgpack] - binc: [http://github.com/ugorji/binc] To install: go get github.com/ugorji/go/codec The idiomatic Go support is as seen in other encoding packages in the standard library (ie json, xml, gob, etc). Rich Feature Set includes: - Simple but extremely powerful and feature-rich API - Very High Performance. Our extensive benchmarks show us outperforming Gob, Json and Bson by 2-4X. This was achieved by taking extreme care on: - managing allocation - function frame size (important due to Go's use of split stacks), - reflection use (and by-passing reflection for common types) - recursion implications - zero-copy mode (encoding/decoding to byte slice without using temp buffers) - Correct. Care was taken to precisely handle corner cases like: overflows, nil maps and slices, nil value in stream, etc. - Efficient zero-copying into temporary byte buffers when encoding into or decoding from a byte slice. - Standard field renaming via tags - Encoding from any value (struct, slice, map, primitives, pointers, interface{}, etc) - Decoding into pointer to any non-nil typed value (struct, slice, map, int, float32, bool, string, reflect.Value, etc) - Supports extension functions to handle the encode/decode of custom types - Support Go 1.2 encoding.BinaryMarshaler/BinaryUnmarshaler - Schema-less decoding (decode into a pointer to a nil interface{} as opposed to a typed non-nil value). Includes Options to configure what specific map or slice type to use when decoding an encoded list or map into a nil interface{} - Provides a RPC Server and Client Codec for net/rpc communication protocol. - Msgpack Specific: - Provides extension functions to handle spec-defined extensions (binary, timestamp) - Options to resolve ambiguities in handling raw bytes (as string or []byte) during schema-less decoding (decoding into a nil interface{}) - RPC Server/Client Codec for msgpack-rpc protocol defined at: https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md - Fast Paths for some container types: For some container types, we circumvent reflection and its associated overhead and allocation costs, and encode/decode directly. These types are: []interface{} []int []string map[interface{}]interface{} map[int]interface{} map[string]interface{} Extension Support Users can register a function to handle the encoding or decoding of their custom types. There are no restrictions on what the custom type can be. Some examples: type BisSet []int type BitSet64 uint64 type UUID string type MyStructWithUnexportedFields struct { a int; b bool; c []int; } type GifImage struct { ... } As an illustration, MyStructWithUnexportedFields would normally be encoded as an empty map because it has no exported fields, while UUID would be encoded as a string. However, with extension support, you can encode any of these however you like. RPC RPC Client and Server Codecs are implemented, so the codecs can be used with the standard net/rpc package. Usage Typical usage model: // create and configure Handle var ( bh codec.BincHandle mh codec.MsgpackHandle ) mh.MapType = reflect.TypeOf(map[string]interface{}(nil)) // configure extensions // e.g. for msgpack, define functions and enable Time support for tag 1 // mh.AddExt(reflect.TypeOf(time.Time{}), 1, myMsgpackTimeEncodeExtFn, myMsgpackTimeDecodeExtFn) // create and use decoder/encoder var ( r io.Reader w io.Writer b []byte h = &bh // or mh to use msgpack ) dec = codec.NewDecoder(r, h) dec = codec.NewDecoderBytes(b, h) err = dec.Decode(&v) enc = codec.NewEncoder(w, h) enc = codec.NewEncoderBytes(&b, h) err = enc.Encode(v) //RPC Server go func() { for { conn, err := listener.Accept() rpcCodec := codec.GoRpc.ServerCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h) rpc.ServeCodec(rpcCodec) } }() //RPC Communication (client side) conn, err = net.Dial("tcp", "localhost:5555") rpcCodec := codec.GoRpc.ClientCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h) client := rpc.NewClientWithCodec(rpcCodec) Representative Benchmark Results Run the benchmark suite using: go test -bi -bench=. -benchmem To run full benchmark suite (including against vmsgpack and bson), see notes in ext_dep_test.go */ package codec golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/README.md000066400000000000000000000154661270406071200254200ustar00rootroot00000000000000# Codec High Performance and Feature-Rich Idiomatic Go Library providing encode/decode support for different serialization formats. Supported Serialization formats are: - msgpack: [https://github.com/msgpack/msgpack] - binc: [http://github.com/ugorji/binc] To install: go get github.com/ugorji/go/codec Online documentation: [http://godoc.org/github.com/ugorji/go/codec] The idiomatic Go support is as seen in other encoding packages in the standard library (ie json, xml, gob, etc). Rich Feature Set includes: - Simple but extremely powerful and feature-rich API - Very High Performance. Our extensive benchmarks show us outperforming Gob, Json and Bson by 2-4X. This was achieved by taking extreme care on: - managing allocation - function frame size (important due to Go's use of split stacks), - reflection use (and by-passing reflection for common types) - recursion implications - zero-copy mode (encoding/decoding to byte slice without using temp buffers) - Correct. Care was taken to precisely handle corner cases like: overflows, nil maps and slices, nil value in stream, etc. - Efficient zero-copying into temporary byte buffers when encoding into or decoding from a byte slice. - Standard field renaming via tags - Encoding from any value (struct, slice, map, primitives, pointers, interface{}, etc) - Decoding into pointer to any non-nil typed value (struct, slice, map, int, float32, bool, string, reflect.Value, etc) - Supports extension functions to handle the encode/decode of custom types - Support Go 1.2 encoding.BinaryMarshaler/BinaryUnmarshaler - Schema-less decoding (decode into a pointer to a nil interface{} as opposed to a typed non-nil value). Includes Options to configure what specific map or slice type to use when decoding an encoded list or map into a nil interface{} - Provides a RPC Server and Client Codec for net/rpc communication protocol. - Msgpack Specific: - Provides extension functions to handle spec-defined extensions (binary, timestamp) - Options to resolve ambiguities in handling raw bytes (as string or []byte) during schema-less decoding (decoding into a nil interface{}) - RPC Server/Client Codec for msgpack-rpc protocol defined at: https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md - Fast Paths for some container types: For some container types, we circumvent reflection and its associated overhead and allocation costs, and encode/decode directly. These types are: []interface{} []int []string map[interface{}]interface{} map[int]interface{} map[string]interface{} ## Extension Support Users can register a function to handle the encoding or decoding of their custom types. There are no restrictions on what the custom type can be. Some examples: type BisSet []int type BitSet64 uint64 type UUID string type MyStructWithUnexportedFields struct { a int; b bool; c []int; } type GifImage struct { ... } As an illustration, MyStructWithUnexportedFields would normally be encoded as an empty map because it has no exported fields, while UUID would be encoded as a string. However, with extension support, you can encode any of these however you like. ## RPC RPC Client and Server Codecs are implemented, so the codecs can be used with the standard net/rpc package. ## Usage Typical usage model: // create and configure Handle var ( bh codec.BincHandle mh codec.MsgpackHandle ) mh.MapType = reflect.TypeOf(map[string]interface{}(nil)) // configure extensions // e.g. for msgpack, define functions and enable Time support for tag 1 // mh.AddExt(reflect.TypeOf(time.Time{}), 1, myMsgpackTimeEncodeExtFn, myMsgpackTimeDecodeExtFn) // create and use decoder/encoder var ( r io.Reader w io.Writer b []byte h = &bh // or mh to use msgpack ) dec = codec.NewDecoder(r, h) dec = codec.NewDecoderBytes(b, h) err = dec.Decode(&v) enc = codec.NewEncoder(w, h) enc = codec.NewEncoderBytes(&b, h) err = enc.Encode(v) //RPC Server go func() { for { conn, err := listener.Accept() rpcCodec := codec.GoRpc.ServerCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h) rpc.ServeCodec(rpcCodec) } }() //RPC Communication (client side) conn, err = net.Dial("tcp", "localhost:5555") rpcCodec := codec.GoRpc.ClientCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h) client := rpc.NewClientWithCodec(rpcCodec) ## Representative Benchmark Results A sample run of benchmark using "go test -bi -bench=. -benchmem": /proc/cpuinfo: Intel(R) Core(TM) i7-2630QM CPU @ 2.00GHz (HT) .............................................. BENCHMARK INIT: 2013-10-16 11:02:50.345970786 -0400 EDT To run full benchmark comparing encodings (MsgPack, Binc, JSON, GOB, etc), use: "go test -bench=." Benchmark: Struct recursive Depth: 1 ApproxDeepSize Of benchmark Struct: 4694 bytes Benchmark One-Pass Run: v-msgpack: len: 1600 bytes bson: len: 3025 bytes msgpack: len: 1560 bytes binc: len: 1187 bytes gob: len: 1972 bytes json: len: 2538 bytes .............................................. PASS Benchmark__Msgpack____Encode 50000 54359 ns/op 14953 B/op 83 allocs/op Benchmark__Msgpack____Decode 10000 106531 ns/op 14990 B/op 410 allocs/op Benchmark__Binc_NoSym_Encode 50000 53956 ns/op 14966 B/op 83 allocs/op Benchmark__Binc_NoSym_Decode 10000 103751 ns/op 14529 B/op 386 allocs/op Benchmark__Binc_Sym___Encode 50000 65961 ns/op 17130 B/op 88 allocs/op Benchmark__Binc_Sym___Decode 10000 106310 ns/op 15857 B/op 287 allocs/op Benchmark__Gob________Encode 10000 135944 ns/op 21189 B/op 237 allocs/op Benchmark__Gob________Decode 5000 405390 ns/op 83460 B/op 1841 allocs/op Benchmark__Json_______Encode 20000 79412 ns/op 13874 B/op 102 allocs/op Benchmark__Json_______Decode 10000 247979 ns/op 14202 B/op 493 allocs/op Benchmark__Bson_______Encode 10000 121762 ns/op 27814 B/op 514 allocs/op Benchmark__Bson_______Decode 10000 162126 ns/op 16514 B/op 789 allocs/op Benchmark__VMsgpack___Encode 50000 69155 ns/op 12370 B/op 344 allocs/op Benchmark__VMsgpack___Decode 10000 151609 ns/op 20307 B/op 571 allocs/op ok ugorji.net/codec 30.827s To run full benchmark suite (including against vmsgpack and bson), see notes in ext\_dep\_test.go golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/bench_test.go000066400000000000000000000211371270406071200265760ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec import ( "bytes" "encoding/gob" "encoding/json" "flag" "fmt" "reflect" "runtime" "testing" "time" ) // Sample way to run: // go test -bi -bv -bd=1 -benchmem -bench=. var ( _ = fmt.Printf benchTs *TestStruc approxSize int benchDoInitBench bool benchVerify bool benchUnscientificRes bool = false //depth of 0 maps to ~400bytes json-encoded string, 1 maps to ~1400 bytes, etc //For depth>1, we likely trigger stack growth for encoders, making benchmarking unreliable. benchDepth int benchInitDebug bool benchCheckers []benchChecker ) type benchEncFn func(interface{}) ([]byte, error) type benchDecFn func([]byte, interface{}) error type benchIntfFn func() interface{} type benchChecker struct { name string encodefn benchEncFn decodefn benchDecFn } func benchInitFlags() { flag.BoolVar(&benchInitDebug, "bg", false, "Bench Debug") flag.IntVar(&benchDepth, "bd", 1, "Bench Depth: If >1, potential unreliable results due to stack growth") flag.BoolVar(&benchDoInitBench, "bi", false, "Run Bench Init") flag.BoolVar(&benchVerify, "bv", false, "Verify Decoded Value during Benchmark") flag.BoolVar(&benchUnscientificRes, "bu", false, "Show Unscientific Results during Benchmark") } func benchInit() { benchTs = newTestStruc(benchDepth, true) approxSize = approxDataSize(reflect.ValueOf(benchTs)) bytesLen := 1024 * 4 * (benchDepth + 1) * (benchDepth + 1) if bytesLen < approxSize { bytesLen = approxSize } benchCheckers = append(benchCheckers, benchChecker{"msgpack", fnMsgpackEncodeFn, fnMsgpackDecodeFn}, benchChecker{"binc-nosym", fnBincNoSymEncodeFn, fnBincNoSymDecodeFn}, benchChecker{"binc-sym", fnBincSymEncodeFn, fnBincSymDecodeFn}, benchChecker{"simple", fnSimpleEncodeFn, fnSimpleDecodeFn}, benchChecker{"gob", fnGobEncodeFn, fnGobDecodeFn}, benchChecker{"json", fnJsonEncodeFn, fnJsonDecodeFn}, ) if benchDoInitBench { runBenchInit() } } func runBenchInit() { logT(nil, "..............................................") logT(nil, "BENCHMARK INIT: %v", time.Now()) logT(nil, "To run full benchmark comparing encodings (MsgPack, Binc, Simple, JSON, GOB, etc), "+ "use: \"go test -bench=.\"") logT(nil, "Benchmark: ") logT(nil, "\tStruct recursive Depth: %d", benchDepth) if approxSize > 0 { logT(nil, "\tApproxDeepSize Of benchmark Struct: %d bytes", approxSize) } if benchUnscientificRes { logT(nil, "Benchmark One-Pass Run (with Unscientific Encode/Decode times): ") } else { logT(nil, "Benchmark One-Pass Run:") } for _, bc := range benchCheckers { doBenchCheck(bc.name, bc.encodefn, bc.decodefn) } logT(nil, "..............................................") if benchInitDebug { logT(nil, "<<<<====>>>> depth: %v, ts: %#v\n", benchDepth, benchTs) } } func fnBenchNewTs() interface{} { return new(TestStruc) } func doBenchCheck(name string, encfn benchEncFn, decfn benchDecFn) { runtime.GC() tnow := time.Now() buf, err := encfn(benchTs) if err != nil { logT(nil, "\t%10s: **** Error encoding benchTs: %v", name, err) } encDur := time.Now().Sub(tnow) encLen := len(buf) runtime.GC() if !benchUnscientificRes { logT(nil, "\t%10s: len: %d bytes\n", name, encLen) return } tnow = time.Now() if err = decfn(buf, new(TestStruc)); err != nil { logT(nil, "\t%10s: **** Error decoding into new TestStruc: %v", name, err) } decDur := time.Now().Sub(tnow) logT(nil, "\t%10s: len: %d bytes, encode: %v, decode: %v\n", name, encLen, encDur, decDur) } func fnBenchmarkEncode(b *testing.B, encName string, ts interface{}, encfn benchEncFn) { runtime.GC() b.ResetTimer() for i := 0; i < b.N; i++ { _, err := encfn(ts) if err != nil { logT(b, "Error encoding benchTs: %s: %v", encName, err) b.FailNow() } } } func fnBenchmarkDecode(b *testing.B, encName string, ts interface{}, encfn benchEncFn, decfn benchDecFn, newfn benchIntfFn, ) { buf, err := encfn(ts) if err != nil { logT(b, "Error encoding benchTs: %s: %v", encName, err) b.FailNow() } runtime.GC() b.ResetTimer() for i := 0; i < b.N; i++ { ts = newfn() if err = decfn(buf, ts); err != nil { logT(b, "Error decoding into new TestStruc: %s: %v", encName, err) b.FailNow() } if benchVerify { if vts, vok := ts.(*TestStruc); vok { verifyTsTree(b, vts) } } } } func verifyTsTree(b *testing.B, ts *TestStruc) { var ts0, ts1m, ts2m, ts1s, ts2s *TestStruc ts0 = ts if benchDepth > 0 { ts1m, ts1s = verifyCheckAndGet(b, ts0) } if benchDepth > 1 { ts2m, ts2s = verifyCheckAndGet(b, ts1m) } for _, tsx := range []*TestStruc{ts0, ts1m, ts2m, ts1s, ts2s} { if tsx != nil { verifyOneOne(b, tsx) } } } func verifyCheckAndGet(b *testing.B, ts0 *TestStruc) (ts1m *TestStruc, ts1s *TestStruc) { // if len(ts1m.Ms) <= 2 { // logT(b, "Error: ts1m.Ms len should be > 2. Got: %v", len(ts1m.Ms)) // b.FailNow() // } if len(ts0.Its) == 0 { logT(b, "Error: ts0.Islice len should be > 0. Got: %v", len(ts0.Its)) b.FailNow() } ts1m = ts0.Mtsptr["0"] ts1s = ts0.Its[0] if ts1m == nil || ts1s == nil { logT(b, "Error: At benchDepth 1, No *TestStruc found") b.FailNow() } return } func verifyOneOne(b *testing.B, ts *TestStruc) { if ts.I64slice[2] != int64(3) { logT(b, "Error: Decode failed by checking values") b.FailNow() } } func fnMsgpackEncodeFn(ts interface{}) (bs []byte, err error) { err = NewEncoderBytes(&bs, testMsgpackH).Encode(ts) return } func fnMsgpackDecodeFn(buf []byte, ts interface{}) error { return NewDecoderBytes(buf, testMsgpackH).Decode(ts) } func fnBincEncodeFn(ts interface{}, sym AsSymbolFlag) (bs []byte, err error) { tSym := testBincH.AsSymbols testBincH.AsSymbols = sym err = NewEncoderBytes(&bs, testBincH).Encode(ts) testBincH.AsSymbols = tSym return } func fnBincDecodeFn(buf []byte, ts interface{}, sym AsSymbolFlag) (err error) { tSym := testBincH.AsSymbols testBincH.AsSymbols = sym err = NewDecoderBytes(buf, testBincH).Decode(ts) testBincH.AsSymbols = tSym return } func fnBincNoSymEncodeFn(ts interface{}) (bs []byte, err error) { return fnBincEncodeFn(ts, AsSymbolNone) } func fnBincNoSymDecodeFn(buf []byte, ts interface{}) error { return fnBincDecodeFn(buf, ts, AsSymbolNone) } func fnBincSymEncodeFn(ts interface{}) (bs []byte, err error) { return fnBincEncodeFn(ts, AsSymbolAll) } func fnBincSymDecodeFn(buf []byte, ts interface{}) error { return fnBincDecodeFn(buf, ts, AsSymbolAll) } func fnSimpleEncodeFn(ts interface{}) (bs []byte, err error) { err = NewEncoderBytes(&bs, testSimpleH).Encode(ts) return } func fnSimpleDecodeFn(buf []byte, ts interface{}) error { return NewDecoderBytes(buf, testSimpleH).Decode(ts) } func fnGobEncodeFn(ts interface{}) ([]byte, error) { bbuf := new(bytes.Buffer) err := gob.NewEncoder(bbuf).Encode(ts) return bbuf.Bytes(), err } func fnGobDecodeFn(buf []byte, ts interface{}) error { return gob.NewDecoder(bytes.NewBuffer(buf)).Decode(ts) } func fnJsonEncodeFn(ts interface{}) ([]byte, error) { return json.Marshal(ts) } func fnJsonDecodeFn(buf []byte, ts interface{}) error { return json.Unmarshal(buf, ts) } func Benchmark__Msgpack____Encode(b *testing.B) { fnBenchmarkEncode(b, "msgpack", benchTs, fnMsgpackEncodeFn) } func Benchmark__Msgpack____Decode(b *testing.B) { fnBenchmarkDecode(b, "msgpack", benchTs, fnMsgpackEncodeFn, fnMsgpackDecodeFn, fnBenchNewTs) } func Benchmark__Binc_NoSym_Encode(b *testing.B) { fnBenchmarkEncode(b, "binc", benchTs, fnBincNoSymEncodeFn) } func Benchmark__Binc_NoSym_Decode(b *testing.B) { fnBenchmarkDecode(b, "binc", benchTs, fnBincNoSymEncodeFn, fnBincNoSymDecodeFn, fnBenchNewTs) } func Benchmark__Binc_Sym___Encode(b *testing.B) { fnBenchmarkEncode(b, "binc", benchTs, fnBincSymEncodeFn) } func Benchmark__Binc_Sym___Decode(b *testing.B) { fnBenchmarkDecode(b, "binc", benchTs, fnBincSymEncodeFn, fnBincSymDecodeFn, fnBenchNewTs) } func Benchmark__Simple____Encode(b *testing.B) { fnBenchmarkEncode(b, "simple", benchTs, fnSimpleEncodeFn) } func Benchmark__Simple____Decode(b *testing.B) { fnBenchmarkDecode(b, "simple", benchTs, fnSimpleEncodeFn, fnSimpleDecodeFn, fnBenchNewTs) } func Benchmark__Gob________Encode(b *testing.B) { fnBenchmarkEncode(b, "gob", benchTs, fnGobEncodeFn) } func Benchmark__Gob________Decode(b *testing.B) { fnBenchmarkDecode(b, "gob", benchTs, fnGobEncodeFn, fnGobDecodeFn, fnBenchNewTs) } func Benchmark__Json_______Encode(b *testing.B) { fnBenchmarkEncode(b, "json", benchTs, fnJsonEncodeFn) } func Benchmark__Json_______Decode(b *testing.B) { fnBenchmarkDecode(b, "json", benchTs, fnJsonEncodeFn, fnJsonDecodeFn, fnBenchNewTs) } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/binc.go000066400000000000000000000417431270406071200254000ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec import ( "math" // "reflect" // "sync/atomic" "time" //"fmt" ) const bincDoPrune = true // No longer needed. Needed before as C lib did not support pruning. //var _ = fmt.Printf // vd as low 4 bits (there are 16 slots) const ( bincVdSpecial byte = iota bincVdPosInt bincVdNegInt bincVdFloat bincVdString bincVdByteArray bincVdArray bincVdMap bincVdTimestamp bincVdSmallInt bincVdUnicodeOther bincVdSymbol bincVdDecimal _ // open slot _ // open slot bincVdCustomExt = 0x0f ) const ( bincSpNil byte = iota bincSpFalse bincSpTrue bincSpNan bincSpPosInf bincSpNegInf bincSpZeroFloat bincSpZero bincSpNegOne ) const ( bincFlBin16 byte = iota bincFlBin32 _ // bincFlBin32e bincFlBin64 _ // bincFlBin64e // others not currently supported ) type bincEncDriver struct { w encWriter m map[string]uint16 // symbols s uint32 // symbols sequencer b [8]byte } func (e *bincEncDriver) isBuiltinType(rt uintptr) bool { return rt == timeTypId } func (e *bincEncDriver) encodeBuiltin(rt uintptr, v interface{}) { switch rt { case timeTypId: bs := encodeTime(v.(time.Time)) e.w.writen1(bincVdTimestamp<<4 | uint8(len(bs))) e.w.writeb(bs) } } func (e *bincEncDriver) encodeNil() { e.w.writen1(bincVdSpecial<<4 | bincSpNil) } func (e *bincEncDriver) encodeBool(b bool) { if b { e.w.writen1(bincVdSpecial<<4 | bincSpTrue) } else { e.w.writen1(bincVdSpecial<<4 | bincSpFalse) } } func (e *bincEncDriver) encodeFloat32(f float32) { if f == 0 { e.w.writen1(bincVdSpecial<<4 | bincSpZeroFloat) return } e.w.writen1(bincVdFloat<<4 | bincFlBin32) e.w.writeUint32(math.Float32bits(f)) } func (e *bincEncDriver) encodeFloat64(f float64) { if f == 0 { e.w.writen1(bincVdSpecial<<4 | bincSpZeroFloat) return } bigen.PutUint64(e.b[:], math.Float64bits(f)) if bincDoPrune { i := 7 for ; i >= 0 && (e.b[i] == 0); i-- { } i++ if i <= 6 { e.w.writen1(bincVdFloat<<4 | 0x8 | bincFlBin64) e.w.writen1(byte(i)) e.w.writeb(e.b[:i]) return } } e.w.writen1(bincVdFloat<<4 | bincFlBin64) e.w.writeb(e.b[:]) } func (e *bincEncDriver) encIntegerPrune(bd byte, pos bool, v uint64, lim uint8) { if lim == 4 { bigen.PutUint32(e.b[:lim], uint32(v)) } else { bigen.PutUint64(e.b[:lim], v) } if bincDoPrune { i := pruneSignExt(e.b[:lim], pos) e.w.writen1(bd | lim - 1 - byte(i)) e.w.writeb(e.b[i:lim]) } else { e.w.writen1(bd | lim - 1) e.w.writeb(e.b[:lim]) } } func (e *bincEncDriver) encodeInt(v int64) { const nbd byte = bincVdNegInt << 4 switch { case v >= 0: e.encUint(bincVdPosInt<<4, true, uint64(v)) case v == -1: e.w.writen1(bincVdSpecial<<4 | bincSpNegOne) default: e.encUint(bincVdNegInt<<4, false, uint64(-v)) } } func (e *bincEncDriver) encodeUint(v uint64) { e.encUint(bincVdPosInt<<4, true, v) } func (e *bincEncDriver) encUint(bd byte, pos bool, v uint64) { switch { case v == 0: e.w.writen1(bincVdSpecial<<4 | bincSpZero) case pos && v >= 1 && v <= 16: e.w.writen1(bincVdSmallInt<<4 | byte(v-1)) case v <= math.MaxUint8: e.w.writen2(bd|0x0, byte(v)) case v <= math.MaxUint16: e.w.writen1(bd | 0x01) e.w.writeUint16(uint16(v)) case v <= math.MaxUint32: e.encIntegerPrune(bd, pos, v, 4) default: e.encIntegerPrune(bd, pos, v, 8) } } func (e *bincEncDriver) encodeExtPreamble(xtag byte, length int) { e.encLen(bincVdCustomExt<<4, uint64(length)) e.w.writen1(xtag) } func (e *bincEncDriver) encodeArrayPreamble(length int) { e.encLen(bincVdArray<<4, uint64(length)) } func (e *bincEncDriver) encodeMapPreamble(length int) { e.encLen(bincVdMap<<4, uint64(length)) } func (e *bincEncDriver) encodeString(c charEncoding, v string) { l := uint64(len(v)) e.encBytesLen(c, l) if l > 0 { e.w.writestr(v) } } func (e *bincEncDriver) encodeSymbol(v string) { // if WriteSymbolsNoRefs { // e.encodeString(c_UTF8, v) // return // } //symbols only offer benefit when string length > 1. //This is because strings with length 1 take only 2 bytes to store //(bd with embedded length, and single byte for string val). l := len(v) switch l { case 0: e.encBytesLen(c_UTF8, 0) return case 1: e.encBytesLen(c_UTF8, 1) e.w.writen1(v[0]) return } if e.m == nil { e.m = make(map[string]uint16, 16) } ui, ok := e.m[v] if ok { if ui <= math.MaxUint8 { e.w.writen2(bincVdSymbol<<4, byte(ui)) } else { e.w.writen1(bincVdSymbol<<4 | 0x8) e.w.writeUint16(ui) } } else { e.s++ ui = uint16(e.s) //ui = uint16(atomic.AddUint32(&e.s, 1)) e.m[v] = ui var lenprec uint8 switch { case l <= math.MaxUint8: // lenprec = 0 case l <= math.MaxUint16: lenprec = 1 case int64(l) <= math.MaxUint32: lenprec = 2 default: lenprec = 3 } if ui <= math.MaxUint8 { e.w.writen2(bincVdSymbol<<4|0x0|0x4|lenprec, byte(ui)) } else { e.w.writen1(bincVdSymbol<<4 | 0x8 | 0x4 | lenprec) e.w.writeUint16(ui) } switch lenprec { case 0: e.w.writen1(byte(l)) case 1: e.w.writeUint16(uint16(l)) case 2: e.w.writeUint32(uint32(l)) default: e.w.writeUint64(uint64(l)) } e.w.writestr(v) } } func (e *bincEncDriver) encodeStringBytes(c charEncoding, v []byte) { l := uint64(len(v)) e.encBytesLen(c, l) if l > 0 { e.w.writeb(v) } } func (e *bincEncDriver) encBytesLen(c charEncoding, length uint64) { //TODO: support bincUnicodeOther (for now, just use string or bytearray) if c == c_RAW { e.encLen(bincVdByteArray<<4, length) } else { e.encLen(bincVdString<<4, length) } } func (e *bincEncDriver) encLen(bd byte, l uint64) { if l < 12 { e.w.writen1(bd | uint8(l+4)) } else { e.encLenNumber(bd, l) } } func (e *bincEncDriver) encLenNumber(bd byte, v uint64) { switch { case v <= math.MaxUint8: e.w.writen2(bd, byte(v)) case v <= math.MaxUint16: e.w.writen1(bd | 0x01) e.w.writeUint16(uint16(v)) case v <= math.MaxUint32: e.w.writen1(bd | 0x02) e.w.writeUint32(uint32(v)) default: e.w.writen1(bd | 0x03) e.w.writeUint64(uint64(v)) } } //------------------------------------ type bincDecDriver struct { r decReader bdRead bool bdType valueType bd byte vd byte vs byte b [8]byte m map[uint32]string // symbols (use uint32 as key, as map optimizes for it) } func (d *bincDecDriver) initReadNext() { if d.bdRead { return } d.bd = d.r.readn1() d.vd = d.bd >> 4 d.vs = d.bd & 0x0f d.bdRead = true d.bdType = valueTypeUnset } func (d *bincDecDriver) currentEncodedType() valueType { if d.bdType == valueTypeUnset { switch d.vd { case bincVdSpecial: switch d.vs { case bincSpNil: d.bdType = valueTypeNil case bincSpFalse, bincSpTrue: d.bdType = valueTypeBool case bincSpNan, bincSpNegInf, bincSpPosInf, bincSpZeroFloat: d.bdType = valueTypeFloat case bincSpZero: d.bdType = valueTypeUint case bincSpNegOne: d.bdType = valueTypeInt default: decErr("currentEncodedType: Unrecognized special value 0x%x", d.vs) } case bincVdSmallInt: d.bdType = valueTypeUint case bincVdPosInt: d.bdType = valueTypeUint case bincVdNegInt: d.bdType = valueTypeInt case bincVdFloat: d.bdType = valueTypeFloat case bincVdString: d.bdType = valueTypeString case bincVdSymbol: d.bdType = valueTypeSymbol case bincVdByteArray: d.bdType = valueTypeBytes case bincVdTimestamp: d.bdType = valueTypeTimestamp case bincVdCustomExt: d.bdType = valueTypeExt case bincVdArray: d.bdType = valueTypeArray case bincVdMap: d.bdType = valueTypeMap default: decErr("currentEncodedType: Unrecognized d.vd: 0x%x", d.vd) } } return d.bdType } func (d *bincDecDriver) tryDecodeAsNil() bool { if d.bd == bincVdSpecial<<4|bincSpNil { d.bdRead = false return true } return false } func (d *bincDecDriver) isBuiltinType(rt uintptr) bool { return rt == timeTypId } func (d *bincDecDriver) decodeBuiltin(rt uintptr, v interface{}) { switch rt { case timeTypId: if d.vd != bincVdTimestamp { decErr("Invalid d.vd. Expecting 0x%x. Received: 0x%x", bincVdTimestamp, d.vd) } tt, err := decodeTime(d.r.readn(int(d.vs))) if err != nil { panic(err) } var vt *time.Time = v.(*time.Time) *vt = tt d.bdRead = false } } func (d *bincDecDriver) decFloatPre(vs, defaultLen byte) { if vs&0x8 == 0 { d.r.readb(d.b[0:defaultLen]) } else { l := d.r.readn1() if l > 8 { decErr("At most 8 bytes used to represent float. Received: %v bytes", l) } for i := l; i < 8; i++ { d.b[i] = 0 } d.r.readb(d.b[0:l]) } } func (d *bincDecDriver) decFloat() (f float64) { //if true { f = math.Float64frombits(d.r.readUint64()); break; } switch vs := d.vs; vs & 0x7 { case bincFlBin32: d.decFloatPre(vs, 4) f = float64(math.Float32frombits(bigen.Uint32(d.b[0:4]))) case bincFlBin64: d.decFloatPre(vs, 8) f = math.Float64frombits(bigen.Uint64(d.b[0:8])) default: decErr("only float32 and float64 are supported. d.vd: 0x%x, d.vs: 0x%x", d.vd, d.vs) } return } func (d *bincDecDriver) decUint() (v uint64) { // need to inline the code (interface conversion and type assertion expensive) switch d.vs { case 0: v = uint64(d.r.readn1()) case 1: d.r.readb(d.b[6:]) v = uint64(bigen.Uint16(d.b[6:])) case 2: d.b[4] = 0 d.r.readb(d.b[5:]) v = uint64(bigen.Uint32(d.b[4:])) case 3: d.r.readb(d.b[4:]) v = uint64(bigen.Uint32(d.b[4:])) case 4, 5, 6: lim := int(7 - d.vs) d.r.readb(d.b[lim:]) for i := 0; i < lim; i++ { d.b[i] = 0 } v = uint64(bigen.Uint64(d.b[:])) case 7: d.r.readb(d.b[:]) v = uint64(bigen.Uint64(d.b[:])) default: decErr("unsigned integers with greater than 64 bits of precision not supported") } return } func (d *bincDecDriver) decIntAny() (ui uint64, i int64, neg bool) { switch d.vd { case bincVdPosInt: ui = d.decUint() i = int64(ui) case bincVdNegInt: ui = d.decUint() i = -(int64(ui)) neg = true case bincVdSmallInt: i = int64(d.vs) + 1 ui = uint64(d.vs) + 1 case bincVdSpecial: switch d.vs { case bincSpZero: //i = 0 case bincSpNegOne: neg = true ui = 1 i = -1 default: decErr("numeric decode fails for special value: d.vs: 0x%x", d.vs) } default: decErr("number can only be decoded from uint or int values. d.bd: 0x%x, d.vd: 0x%x", d.bd, d.vd) } return } func (d *bincDecDriver) decodeInt(bitsize uint8) (i int64) { _, i, _ = d.decIntAny() checkOverflow(0, i, bitsize) d.bdRead = false return } func (d *bincDecDriver) decodeUint(bitsize uint8) (ui uint64) { ui, i, neg := d.decIntAny() if neg { decErr("Assigning negative signed value: %v, to unsigned type", i) } checkOverflow(ui, 0, bitsize) d.bdRead = false return } func (d *bincDecDriver) decodeFloat(chkOverflow32 bool) (f float64) { switch d.vd { case bincVdSpecial: d.bdRead = false switch d.vs { case bincSpNan: return math.NaN() case bincSpPosInf: return math.Inf(1) case bincSpZeroFloat, bincSpZero: return case bincSpNegInf: return math.Inf(-1) default: decErr("Invalid d.vs decoding float where d.vd=bincVdSpecial: %v", d.vs) } case bincVdFloat: f = d.decFloat() default: _, i, _ := d.decIntAny() f = float64(i) } checkOverflowFloat32(f, chkOverflow32) d.bdRead = false return } // bool can be decoded from bool only (single byte). func (d *bincDecDriver) decodeBool() (b bool) { switch d.bd { case (bincVdSpecial | bincSpFalse): // b = false case (bincVdSpecial | bincSpTrue): b = true default: decErr("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd) } d.bdRead = false return } func (d *bincDecDriver) readMapLen() (length int) { if d.vd != bincVdMap { decErr("Invalid d.vd for map. Expecting 0x%x. Got: 0x%x", bincVdMap, d.vd) } length = d.decLen() d.bdRead = false return } func (d *bincDecDriver) readArrayLen() (length int) { if d.vd != bincVdArray { decErr("Invalid d.vd for array. Expecting 0x%x. Got: 0x%x", bincVdArray, d.vd) } length = d.decLen() d.bdRead = false return } func (d *bincDecDriver) decLen() int { if d.vs <= 3 { return int(d.decUint()) } return int(d.vs - 4) } func (d *bincDecDriver) decodeString() (s string) { switch d.vd { case bincVdString, bincVdByteArray: if length := d.decLen(); length > 0 { s = string(d.r.readn(length)) } case bincVdSymbol: //from vs: extract numSymbolBytes, containsStringVal, strLenPrecision, //extract symbol //if containsStringVal, read it and put in map //else look in map for string value var symbol uint32 vs := d.vs //fmt.Printf(">>>> d.vs: 0b%b, & 0x8: %v, & 0x4: %v\n", d.vs, vs & 0x8, vs & 0x4) if vs&0x8 == 0 { symbol = uint32(d.r.readn1()) } else { symbol = uint32(d.r.readUint16()) } if d.m == nil { d.m = make(map[uint32]string, 16) } if vs&0x4 == 0 { s = d.m[symbol] } else { var slen int switch vs & 0x3 { case 0: slen = int(d.r.readn1()) case 1: slen = int(d.r.readUint16()) case 2: slen = int(d.r.readUint32()) case 3: slen = int(d.r.readUint64()) } s = string(d.r.readn(slen)) d.m[symbol] = s } default: decErr("Invalid d.vd for string. Expecting string:0x%x, bytearray:0x%x or symbol: 0x%x. Got: 0x%x", bincVdString, bincVdByteArray, bincVdSymbol, d.vd) } d.bdRead = false return } func (d *bincDecDriver) decodeBytes(bs []byte) (bsOut []byte, changed bool) { var clen int switch d.vd { case bincVdString, bincVdByteArray: clen = d.decLen() default: decErr("Invalid d.vd for bytes. Expecting string:0x%x or bytearray:0x%x. Got: 0x%x", bincVdString, bincVdByteArray, d.vd) } if clen > 0 { // if no contents in stream, don't update the passed byteslice if len(bs) != clen { if len(bs) > clen { bs = bs[:clen] } else { bs = make([]byte, clen) } bsOut = bs changed = true } d.r.readb(bs) } d.bdRead = false return } func (d *bincDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte) { switch d.vd { case bincVdCustomExt: l := d.decLen() xtag = d.r.readn1() if verifyTag && xtag != tag { decErr("Wrong extension tag. Got %b. Expecting: %v", xtag, tag) } xbs = d.r.readn(l) case bincVdByteArray: xbs, _ = d.decodeBytes(nil) default: decErr("Invalid d.vd for extensions (Expecting extensions or byte array). Got: 0x%x", d.vd) } d.bdRead = false return } func (d *bincDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurther bool) { d.initReadNext() switch d.vd { case bincVdSpecial: switch d.vs { case bincSpNil: vt = valueTypeNil case bincSpFalse: vt = valueTypeBool v = false case bincSpTrue: vt = valueTypeBool v = true case bincSpNan: vt = valueTypeFloat v = math.NaN() case bincSpPosInf: vt = valueTypeFloat v = math.Inf(1) case bincSpNegInf: vt = valueTypeFloat v = math.Inf(-1) case bincSpZeroFloat: vt = valueTypeFloat v = float64(0) case bincSpZero: vt = valueTypeUint v = int64(0) // int8(0) case bincSpNegOne: vt = valueTypeInt v = int64(-1) // int8(-1) default: decErr("decodeNaked: Unrecognized special value 0x%x", d.vs) } case bincVdSmallInt: vt = valueTypeUint v = uint64(int8(d.vs)) + 1 // int8(d.vs) + 1 case bincVdPosInt: vt = valueTypeUint v = d.decUint() case bincVdNegInt: vt = valueTypeInt v = -(int64(d.decUint())) case bincVdFloat: vt = valueTypeFloat v = d.decFloat() case bincVdSymbol: vt = valueTypeSymbol v = d.decodeString() case bincVdString: vt = valueTypeString v = d.decodeString() case bincVdByteArray: vt = valueTypeBytes v, _ = d.decodeBytes(nil) case bincVdTimestamp: vt = valueTypeTimestamp tt, err := decodeTime(d.r.readn(int(d.vs))) if err != nil { panic(err) } v = tt case bincVdCustomExt: vt = valueTypeExt l := d.decLen() var re RawExt re.Tag = d.r.readn1() re.Data = d.r.readn(l) v = &re vt = valueTypeExt case bincVdArray: vt = valueTypeArray decodeFurther = true case bincVdMap: vt = valueTypeMap decodeFurther = true default: decErr("decodeNaked: Unrecognized d.vd: 0x%x", d.vd) } if !decodeFurther { d.bdRead = false } return } //------------------------------------ //BincHandle is a Handle for the Binc Schema-Free Encoding Format //defined at https://github.com/ugorji/binc . // //BincHandle currently supports all Binc features with the following EXCEPTIONS: // - only integers up to 64 bits of precision are supported. // big integers are unsupported. // - Only IEEE 754 binary32 and binary64 floats are supported (ie Go float32 and float64 types). // extended precision and decimal IEEE 754 floats are unsupported. // - Only UTF-8 strings supported. // Unicode_Other Binc types (UTF16, UTF32) are currently unsupported. //Note that these EXCEPTIONS are temporary and full support is possible and may happen soon. type BincHandle struct { BasicHandle } func (h *BincHandle) newEncDriver(w encWriter) encDriver { return &bincEncDriver{w: w} } func (h *BincHandle) newDecDriver(r decReader) decDriver { return &bincDecDriver{r: r} } func (_ *BincHandle) writeExt() bool { return true } func (h *BincHandle) getBasicHandle() *BasicHandle { return &h.BasicHandle } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/codecs_test.go000066400000000000000000000665351270406071200267720ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec // Test works by using a slice of interfaces. // It can test for encoding/decoding into/from a nil interface{} // or passing the object to encode/decode into. // // There are basically 2 main tests here. // First test internally encodes and decodes things and verifies that // the artifact was as expected. // Second test will use python msgpack to create a bunch of golden files, // read those files, and compare them to what it should be. It then // writes those files back out and compares the byte streams. // // Taken together, the tests are pretty extensive. import ( "bytes" "encoding/gob" "flag" "fmt" "io/ioutil" "math" "net" "net/rpc" "os" "os/exec" "path/filepath" "reflect" "runtime" "strconv" "sync/atomic" "testing" "time" ) type testVerifyArg int const ( testVerifyMapTypeSame testVerifyArg = iota testVerifyMapTypeStrIntf testVerifyMapTypeIntfIntf // testVerifySliceIntf testVerifyForPython ) var ( testInitDebug bool testUseIoEncDec bool testStructToArray bool testWriteNoSymbols bool _ = fmt.Printf skipVerifyVal interface{} = &(struct{}{}) // For Go Time, do not use a descriptive timezone. // It's unnecessary, and makes it harder to do a reflect.DeepEqual. // The Offset already tells what the offset should be, if not on UTC and unknown zone name. timeLoc = time.FixedZone("", -8*60*60) // UTC-08:00 //time.UTC-8 timeToCompare1 = time.Date(2012, 2, 2, 2, 2, 2, 2000, timeLoc) timeToCompare2 = time.Date(1900, 2, 2, 2, 2, 2, 2000, timeLoc) timeToCompare3 = time.Unix(0, 0).UTC() timeToCompare4 = time.Time{}.UTC() table []interface{} // main items we encode tableVerify []interface{} // we verify encoded things against this after decode tableTestNilVerify []interface{} // for nil interface, use this to verify (rules are different) tablePythonVerify []interface{} // for verifying for python, since Python sometimes // will encode a float32 as float64, or large int as uint testRpcInt = new(TestRpcInt) testMsgpackH = &MsgpackHandle{} testBincH = &BincHandle{} testSimpleH = &SimpleHandle{} ) func testInitFlags() { // delete(testDecOpts.ExtFuncs, timeTyp) flag.BoolVar(&testInitDebug, "tg", false, "Test Debug") flag.BoolVar(&testUseIoEncDec, "ti", false, "Use IO Reader/Writer for Marshal/Unmarshal") flag.BoolVar(&testStructToArray, "ts", false, "Set StructToArray option") flag.BoolVar(&testWriteNoSymbols, "tn", false, "Set NoSymbols option") } type AnonInTestStruc struct { AS string AI64 int64 AI16 int16 AUi64 uint64 ASslice []string AI64slice []int64 } type TestStruc struct { S string I64 int64 I16 int16 Ui64 uint64 Ui8 uint8 B bool By byte Sslice []string I64slice []int64 I16slice []int16 Ui64slice []uint64 Ui8slice []uint8 Bslice []bool Byslice []byte Islice []interface{} Iptrslice []*int64 AnonInTestStruc //M map[interface{}]interface{} `json:"-",bson:"-"` Ms map[string]interface{} Msi64 map[string]int64 Nintf interface{} //don't set this, so we can test for nil T time.Time Nmap map[string]bool //don't set this, so we can test for nil Nslice []byte //don't set this, so we can test for nil Nint64 *int64 //don't set this, so we can test for nil Mtsptr map[string]*TestStruc Mts map[string]TestStruc Its []*TestStruc Nteststruc *TestStruc } type TestABC struct { A, B, C string } type TestRpcInt struct { i int } func (r *TestRpcInt) Update(n int, res *int) error { r.i = n; *res = r.i; return nil } func (r *TestRpcInt) Square(ignore int, res *int) error { *res = r.i * r.i; return nil } func (r *TestRpcInt) Mult(n int, res *int) error { *res = r.i * n; return nil } func (r *TestRpcInt) EchoStruct(arg TestABC, res *string) error { *res = fmt.Sprintf("%#v", arg) return nil } func (r *TestRpcInt) Echo123(args []string, res *string) error { *res = fmt.Sprintf("%#v", args) return nil } func testVerifyVal(v interface{}, arg testVerifyArg) (v2 interface{}) { //for python msgpack, // - all positive integers are unsigned 64-bit ints // - all floats are float64 switch iv := v.(type) { case int8: if iv > 0 { v2 = uint64(iv) } else { v2 = int64(iv) } case int16: if iv > 0 { v2 = uint64(iv) } else { v2 = int64(iv) } case int32: if iv > 0 { v2 = uint64(iv) } else { v2 = int64(iv) } case int64: if iv > 0 { v2 = uint64(iv) } else { v2 = int64(iv) } case uint8: v2 = uint64(iv) case uint16: v2 = uint64(iv) case uint32: v2 = uint64(iv) case uint64: v2 = uint64(iv) case float32: v2 = float64(iv) case float64: v2 = float64(iv) case []interface{}: m2 := make([]interface{}, len(iv)) for j, vj := range iv { m2[j] = testVerifyVal(vj, arg) } v2 = m2 case map[string]bool: switch arg { case testVerifyMapTypeSame: m2 := make(map[string]bool) for kj, kv := range iv { m2[kj] = kv } v2 = m2 case testVerifyMapTypeStrIntf, testVerifyForPython: m2 := make(map[string]interface{}) for kj, kv := range iv { m2[kj] = kv } v2 = m2 case testVerifyMapTypeIntfIntf: m2 := make(map[interface{}]interface{}) for kj, kv := range iv { m2[kj] = kv } v2 = m2 } case map[string]interface{}: switch arg { case testVerifyMapTypeSame: m2 := make(map[string]interface{}) for kj, kv := range iv { m2[kj] = testVerifyVal(kv, arg) } v2 = m2 case testVerifyMapTypeStrIntf, testVerifyForPython: m2 := make(map[string]interface{}) for kj, kv := range iv { m2[kj] = testVerifyVal(kv, arg) } v2 = m2 case testVerifyMapTypeIntfIntf: m2 := make(map[interface{}]interface{}) for kj, kv := range iv { m2[kj] = testVerifyVal(kv, arg) } v2 = m2 } case map[interface{}]interface{}: m2 := make(map[interface{}]interface{}) for kj, kv := range iv { m2[testVerifyVal(kj, arg)] = testVerifyVal(kv, arg) } v2 = m2 case time.Time: switch arg { case testVerifyForPython: if iv2 := iv.UnixNano(); iv2 > 0 { v2 = uint64(iv2) } else { v2 = int64(iv2) } default: v2 = v } default: v2 = v } return } func testInit() { gob.Register(new(TestStruc)) if testInitDebug { ts0 := newTestStruc(2, false) fmt.Printf("====> depth: %v, ts: %#v\n", 2, ts0) } testBincH.StructToArray = testStructToArray if testWriteNoSymbols { testBincH.AsSymbols = AsSymbolNone } else { testBincH.AsSymbols = AsSymbolAll } testMsgpackH.StructToArray = testStructToArray testMsgpackH.RawToString = true // testMsgpackH.AddExt(byteSliceTyp, 0, testMsgpackH.BinaryEncodeExt, testMsgpackH.BinaryDecodeExt) // testMsgpackH.AddExt(timeTyp, 1, testMsgpackH.TimeEncodeExt, testMsgpackH.TimeDecodeExt) timeEncExt := func(rv reflect.Value) ([]byte, error) { return encodeTime(rv.Interface().(time.Time)), nil } timeDecExt := func(rv reflect.Value, bs []byte) error { tt, err := decodeTime(bs) if err == nil { rv.Set(reflect.ValueOf(tt)) } return err } // add extensions for msgpack, simple for time.Time, so we can encode/decode same way. testMsgpackH.AddExt(timeTyp, 1, timeEncExt, timeDecExt) testSimpleH.AddExt(timeTyp, 1, timeEncExt, timeDecExt) primitives := []interface{}{ int8(-8), int16(-1616), int32(-32323232), int64(-6464646464646464), uint8(192), uint16(1616), uint32(32323232), uint64(6464646464646464), byte(192), float32(-3232.0), float64(-6464646464.0), float32(3232.0), float64(6464646464.0), false, true, nil, "someday", "", "bytestring", timeToCompare1, timeToCompare2, timeToCompare3, timeToCompare4, } mapsAndStrucs := []interface{}{ map[string]bool{ "true": true, "false": false, }, map[string]interface{}{ "true": "True", "false": false, "uint16(1616)": uint16(1616), }, //add a complex combo map in here. (map has list which has map) //note that after the first thing, everything else should be generic. map[string]interface{}{ "list": []interface{}{ int16(1616), int32(32323232), true, float32(-3232.0), map[string]interface{}{ "TRUE": true, "FALSE": false, }, []interface{}{true, false}, }, "int32": int32(32323232), "bool": true, "LONG STRING": "123456789012345678901234567890123456789012345678901234567890", "SHORT STRING": "1234567890", }, map[interface{}]interface{}{ true: "true", uint8(138): false, "false": uint8(200), }, newTestStruc(0, false), } table = []interface{}{} table = append(table, primitives...) //0-19 are primitives table = append(table, primitives) //20 is a list of primitives table = append(table, mapsAndStrucs...) //21-24 are maps. 25 is a *struct tableVerify = make([]interface{}, len(table)) tableTestNilVerify = make([]interface{}, len(table)) tablePythonVerify = make([]interface{}, len(table)) lp := len(primitives) av := tableVerify for i, v := range table { if i == lp+3 { av[i] = skipVerifyVal continue } //av[i] = testVerifyVal(v, testVerifyMapTypeSame) switch v.(type) { case []interface{}: av[i] = testVerifyVal(v, testVerifyMapTypeSame) case map[string]interface{}: av[i] = testVerifyVal(v, testVerifyMapTypeSame) case map[interface{}]interface{}: av[i] = testVerifyVal(v, testVerifyMapTypeSame) default: av[i] = v } } av = tableTestNilVerify for i, v := range table { if i > lp+3 { av[i] = skipVerifyVal continue } av[i] = testVerifyVal(v, testVerifyMapTypeStrIntf) } av = tablePythonVerify for i, v := range table { if i > lp+3 { av[i] = skipVerifyVal continue } av[i] = testVerifyVal(v, testVerifyForPython) } tablePythonVerify = tablePythonVerify[:24] } func testUnmarshal(v interface{}, data []byte, h Handle) error { if testUseIoEncDec { return NewDecoder(bytes.NewBuffer(data), h).Decode(v) } return NewDecoderBytes(data, h).Decode(v) } func testMarshal(v interface{}, h Handle) (bs []byte, err error) { if testUseIoEncDec { var buf bytes.Buffer err = NewEncoder(&buf, h).Encode(v) bs = buf.Bytes() return } err = NewEncoderBytes(&bs, h).Encode(v) return } func testMarshalErr(v interface{}, h Handle, t *testing.T, name string) (bs []byte, err error) { if bs, err = testMarshal(v, h); err != nil { logT(t, "Error encoding %s: %v, Err: %v", name, v, err) t.FailNow() } return } func testUnmarshalErr(v interface{}, data []byte, h Handle, t *testing.T, name string) (err error) { if err = testUnmarshal(v, data, h); err != nil { logT(t, "Error Decoding into %s: %v, Err: %v", name, v, err) t.FailNow() } return } func newTestStruc(depth int, bench bool) (ts *TestStruc) { var i64a, i64b, i64c, i64d int64 = 64, 6464, 646464, 64646464 ts = &TestStruc{ S: "some string", I64: math.MaxInt64 * 2 / 3, // 64, I16: 16, Ui64: uint64(int64(math.MaxInt64 * 2 / 3)), // 64, //don't use MaxUint64, as bson can't write it Ui8: 160, B: true, By: 5, Sslice: []string{"one", "two", "three"}, I64slice: []int64{1, 2, 3}, I16slice: []int16{4, 5, 6}, Ui64slice: []uint64{137, 138, 139}, Ui8slice: []uint8{210, 211, 212}, Bslice: []bool{true, false, true, false}, Byslice: []byte{13, 14, 15}, Islice: []interface{}{"true", true, "no", false, uint64(288), float64(0.4)}, Ms: map[string]interface{}{ "true": "true", "int64(9)": false, }, Msi64: map[string]int64{ "one": 1, "two": 2, }, T: timeToCompare1, AnonInTestStruc: AnonInTestStruc{ AS: "A-String", AI64: 64, AI16: 16, AUi64: 64, ASslice: []string{"Aone", "Atwo", "Athree"}, AI64slice: []int64{1, 2, 3}, }, } //For benchmarks, some things will not work. if !bench { //json and bson require string keys in maps //ts.M = map[interface{}]interface{}{ // true: "true", // int8(9): false, //} //gob cannot encode nil in element in array (encodeArray: nil element) ts.Iptrslice = []*int64{nil, &i64a, nil, &i64b, nil, &i64c, nil, &i64d, nil} // ts.Iptrslice = nil } if depth > 0 { depth-- if ts.Mtsptr == nil { ts.Mtsptr = make(map[string]*TestStruc) } if ts.Mts == nil { ts.Mts = make(map[string]TestStruc) } ts.Mtsptr["0"] = newTestStruc(depth, bench) ts.Mts["0"] = *(ts.Mtsptr["0"]) ts.Its = append(ts.Its, ts.Mtsptr["0"]) } return } // doTestCodecTableOne allows us test for different variations based on arguments passed. func doTestCodecTableOne(t *testing.T, testNil bool, h Handle, vs []interface{}, vsVerify []interface{}) { //if testNil, then just test for when a pointer to a nil interface{} is passed. It should work. //Current setup allows us test (at least manually) the nil interface or typed interface. logT(t, "================ TestNil: %v ================\n", testNil) for i, v0 := range vs { logT(t, "..............................................") logT(t, " Testing: #%d:, %T, %#v\n", i, v0, v0) b0, err := testMarshalErr(v0, h, t, "v0") if err != nil { continue } logT(t, " Encoded bytes: len: %v, %v\n", len(b0), b0) var v1 interface{} if testNil { err = testUnmarshal(&v1, b0, h) } else { if v0 != nil { v0rt := reflect.TypeOf(v0) // ptr rv1 := reflect.New(v0rt) err = testUnmarshal(rv1.Interface(), b0, h) v1 = rv1.Elem().Interface() // v1 = reflect.Indirect(reflect.ValueOf(v1)).Interface() } } logT(t, " v1 returned: %T, %#v", v1, v1) // if v1 != nil { // logT(t, " v1 returned: %T, %#v", v1, v1) // //we always indirect, because ptr to typed value may be passed (if not testNil) // v1 = reflect.Indirect(reflect.ValueOf(v1)).Interface() // } if err != nil { logT(t, "-------- Error: %v. Partial return: %v", err, v1) failT(t) continue } v0check := vsVerify[i] if v0check == skipVerifyVal { logT(t, " Nil Check skipped: Decoded: %T, %#v\n", v1, v1) continue } if err = deepEqual(v0check, v1); err == nil { logT(t, "++++++++ Before and After marshal matched\n") } else { logT(t, "-------- Before and After marshal do not match: Error: %v"+ " ====> GOLDEN: (%T) %#v, DECODED: (%T) %#v\n", err, v0check, v0check, v1, v1) failT(t) } } } func testCodecTableOne(t *testing.T, h Handle) { // func TestMsgpackAllExperimental(t *testing.T) { // dopts := testDecOpts(nil, nil, false, true, true), switch v := h.(type) { case *MsgpackHandle: var oldWriteExt, oldRawToString bool oldWriteExt, v.WriteExt = v.WriteExt, true oldRawToString, v.RawToString = v.RawToString, true doTestCodecTableOne(t, false, h, table, tableVerify) v.WriteExt, v.RawToString = oldWriteExt, oldRawToString default: doTestCodecTableOne(t, false, h, table, tableVerify) } // func TestMsgpackAll(t *testing.T) { idxTime, numPrim, numMap := 19, 23, 4 //skip []interface{} containing time.Time doTestCodecTableOne(t, false, h, table[:numPrim], tableVerify[:numPrim]) doTestCodecTableOne(t, false, h, table[numPrim+1:], tableVerify[numPrim+1:]) // func TestMsgpackNilStringMap(t *testing.T) { var oldMapType reflect.Type v := h.getBasicHandle() oldMapType, v.MapType = v.MapType, mapStrIntfTyp //skip time.Time, []interface{} containing time.Time, last map, and newStruc doTestCodecTableOne(t, true, h, table[:idxTime], tableTestNilVerify[:idxTime]) doTestCodecTableOne(t, true, h, table[numPrim+1:numPrim+numMap], tableTestNilVerify[numPrim+1:numPrim+numMap]) v.MapType = oldMapType // func TestMsgpackNilIntf(t *testing.T) { //do newTestStruc and last element of map doTestCodecTableOne(t, true, h, table[numPrim+numMap:], tableTestNilVerify[numPrim+numMap:]) //TODO? What is this one? //doTestCodecTableOne(t, true, h, table[17:18], tableTestNilVerify[17:18]) } func testCodecMiscOne(t *testing.T, h Handle) { b, err := testMarshalErr(32, h, t, "32") // Cannot do this nil one, because faster type assertion decoding will panic // var i *int32 // if err = testUnmarshal(b, i, nil); err == nil { // logT(t, "------- Expecting error because we cannot unmarshal to int32 nil ptr") // t.FailNow() // } var i2 int32 = 0 err = testUnmarshalErr(&i2, b, h, t, "int32-ptr") if i2 != int32(32) { logT(t, "------- didn't unmarshal to 32: Received: %d", i2) t.FailNow() } // func TestMsgpackDecodePtr(t *testing.T) { ts := newTestStruc(0, false) b, err = testMarshalErr(ts, h, t, "pointer-to-struct") if len(b) < 40 { logT(t, "------- Size must be > 40. Size: %d", len(b)) t.FailNow() } logT(t, "------- b: %v", b) ts2 := new(TestStruc) err = testUnmarshalErr(ts2, b, h, t, "pointer-to-struct") if ts2.I64 != math.MaxInt64*2/3 { logT(t, "------- Unmarshal wrong. Expect I64 = 64. Got: %v", ts2.I64) t.FailNow() } // func TestMsgpackIntfDecode(t *testing.T) { m := map[string]int{"A": 2, "B": 3} p := []interface{}{m} bs, err := testMarshalErr(p, h, t, "p") m2 := map[string]int{} p2 := []interface{}{m2} err = testUnmarshalErr(&p2, bs, h, t, "&p2") if m2["A"] != 2 || m2["B"] != 3 { logT(t, "m2 not as expected: expecting: %v, got: %v", m, m2) t.FailNow() } // log("m: %v, m2: %v, p: %v, p2: %v", m, m2, p, p2) checkEqualT(t, p, p2, "p=p2") checkEqualT(t, m, m2, "m=m2") if err = deepEqual(p, p2); err == nil { logT(t, "p and p2 match") } else { logT(t, "Not Equal: %v. p: %v, p2: %v", err, p, p2) t.FailNow() } if err = deepEqual(m, m2); err == nil { logT(t, "m and m2 match") } else { logT(t, "Not Equal: %v. m: %v, m2: %v", err, m, m2) t.FailNow() } // func TestMsgpackDecodeStructSubset(t *testing.T) { // test that we can decode a subset of the stream mm := map[string]interface{}{"A": 5, "B": 99, "C": 333} bs, err = testMarshalErr(mm, h, t, "mm") type ttt struct { A uint8 C int32 } var t2 ttt testUnmarshalErr(&t2, bs, h, t, "t2") t3 := ttt{5, 333} checkEqualT(t, t2, t3, "t2=t3") // println(">>>>>") // test simple arrays, non-addressable arrays, slices type tarr struct { A int64 B [3]int64 C []byte D [3]byte } var tarr0 = tarr{1, [3]int64{2, 3, 4}, []byte{4, 5, 6}, [3]byte{7, 8, 9}} // test both pointer and non-pointer (value) for _, tarr1 := range []interface{}{tarr0, &tarr0} { bs, err = testMarshalErr(tarr1, h, t, "tarr1") var tarr2 tarr testUnmarshalErr(&tarr2, bs, h, t, "tarr2") checkEqualT(t, tarr0, tarr2, "tarr0=tarr2") // fmt.Printf(">>>> err: %v. tarr1: %v, tarr2: %v\n", err, tarr0, tarr2) } // test byte array, even if empty (msgpack only) if h == testMsgpackH { type ystruct struct { Anarray []byte } var ya = ystruct{} testUnmarshalErr(&ya, []byte{0x91, 0x90}, h, t, "ya") } } func testCodecEmbeddedPointer(t *testing.T, h Handle) { type Z int type A struct { AnInt int } type B struct { *Z *A MoreInt int } var z Z = 4 x1 := &B{&z, &A{5}, 6} bs, err := testMarshalErr(x1, h, t, "x1") // fmt.Printf("buf: len(%v): %x\n", buf.Len(), buf.Bytes()) var x2 = new(B) err = testUnmarshalErr(x2, bs, h, t, "x2") err = checkEqualT(t, x1, x2, "x1=x2") _ = err } func doTestRpcOne(t *testing.T, rr Rpc, h Handle, doRequest bool, exitSleepMs time.Duration, ) (port int) { // rpc needs EOF, which is sent via a panic, and so must be recovered. if !recoverPanicToErr { logT(t, "EXPECTED. set recoverPanicToErr=true, since rpc needs EOF") t.FailNow() } srv := rpc.NewServer() srv.Register(testRpcInt) ln, err := net.Listen("tcp", "127.0.0.1:0") // log("listener: %v", ln.Addr()) checkErrT(t, err) port = (ln.Addr().(*net.TCPAddr)).Port // var opts *DecoderOptions // opts := testDecOpts // opts.MapType = mapStrIntfTyp // opts.RawToString = false serverExitChan := make(chan bool, 1) var serverExitFlag uint64 = 0 serverFn := func() { for { conn1, err1 := ln.Accept() // if err1 != nil { // //fmt.Printf("accept err1: %v\n", err1) // continue // } if atomic.LoadUint64(&serverExitFlag) == 1 { serverExitChan <- true conn1.Close() return // exit serverFn goroutine } if err1 == nil { var sc rpc.ServerCodec = rr.ServerCodec(conn1, h) srv.ServeCodec(sc) } } } clientFn := func(cc rpc.ClientCodec) { cl := rpc.NewClientWithCodec(cc) defer cl.Close() var up, sq, mult int var rstr string // log("Calling client") checkErrT(t, cl.Call("TestRpcInt.Update", 5, &up)) // log("Called TestRpcInt.Update") checkEqualT(t, testRpcInt.i, 5, "testRpcInt.i=5") checkEqualT(t, up, 5, "up=5") checkErrT(t, cl.Call("TestRpcInt.Square", 1, &sq)) checkEqualT(t, sq, 25, "sq=25") checkErrT(t, cl.Call("TestRpcInt.Mult", 20, &mult)) checkEqualT(t, mult, 100, "mult=100") checkErrT(t, cl.Call("TestRpcInt.EchoStruct", TestABC{"Aa", "Bb", "Cc"}, &rstr)) checkEqualT(t, rstr, fmt.Sprintf("%#v", TestABC{"Aa", "Bb", "Cc"}), "rstr=") checkErrT(t, cl.Call("TestRpcInt.Echo123", []string{"A1", "B2", "C3"}, &rstr)) checkEqualT(t, rstr, fmt.Sprintf("%#v", []string{"A1", "B2", "C3"}), "rstr=") } connFn := func() (bs net.Conn) { // log("calling f1") bs, err2 := net.Dial(ln.Addr().Network(), ln.Addr().String()) //fmt.Printf("f1. bs: %v, err2: %v\n", bs, err2) checkErrT(t, err2) return } exitFn := func() { atomic.StoreUint64(&serverExitFlag, 1) bs := connFn() <-serverExitChan bs.Close() // serverExitChan <- true } go serverFn() runtime.Gosched() //time.Sleep(100 * time.Millisecond) if exitSleepMs == 0 { defer ln.Close() defer exitFn() } if doRequest { bs := connFn() cc := rr.ClientCodec(bs, h) clientFn(cc) } if exitSleepMs != 0 { go func() { defer ln.Close() time.Sleep(exitSleepMs) exitFn() }() } return } // Comprehensive testing that generates data encoded from python msgpack, // and validates that our code can read and write it out accordingly. // We keep this unexported here, and put actual test in ext_dep_test.go. // This way, it can be excluded by excluding file completely. func doTestMsgpackPythonGenStreams(t *testing.T) { logT(t, "TestPythonGenStreams") tmpdir, err := ioutil.TempDir("", "golang-msgpack-test") if err != nil { logT(t, "-------- Unable to create temp directory\n") t.FailNow() } defer os.RemoveAll(tmpdir) logT(t, "tmpdir: %v", tmpdir) cmd := exec.Command("python", "msgpack_test.py", "testdata", tmpdir) //cmd.Stdin = strings.NewReader("some input") //cmd.Stdout = &out var cmdout []byte if cmdout, err = cmd.CombinedOutput(); err != nil { logT(t, "-------- Error running msgpack_test.py testdata. Err: %v", err) logT(t, " %v", string(cmdout)) t.FailNow() } oldMapType := testMsgpackH.MapType for i, v := range tablePythonVerify { testMsgpackH.MapType = oldMapType //load up the golden file based on number //decode it //compare to in-mem object //encode it again //compare to output stream logT(t, "..............................................") logT(t, " Testing: #%d: %T, %#v\n", i, v, v) var bss []byte bss, err = ioutil.ReadFile(filepath.Join(tmpdir, strconv.Itoa(i)+".golden")) if err != nil { logT(t, "-------- Error reading golden file: %d. Err: %v", i, err) failT(t) continue } testMsgpackH.MapType = mapStrIntfTyp var v1 interface{} if err = testUnmarshal(&v1, bss, testMsgpackH); err != nil { logT(t, "-------- Error decoding stream: %d: Err: %v", i, err) failT(t) continue } if v == skipVerifyVal { continue } //no need to indirect, because we pass a nil ptr, so we already have the value //if v1 != nil { v1 = reflect.Indirect(reflect.ValueOf(v1)).Interface() } if err = deepEqual(v, v1); err == nil { logT(t, "++++++++ Objects match") } else { logT(t, "-------- Objects do not match: %v. Source: %T. Decoded: %T", err, v, v1) logT(t, "-------- AGAINST: %#v", v) logT(t, "-------- DECODED: %#v <====> %#v", v1, reflect.Indirect(reflect.ValueOf(v1)).Interface()) failT(t) } bsb, err := testMarshal(v1, testMsgpackH) if err != nil { logT(t, "Error encoding to stream: %d: Err: %v", i, err) failT(t) continue } if err = deepEqual(bsb, bss); err == nil { logT(t, "++++++++ Bytes match") } else { logT(t, "???????? Bytes do not match. %v.", err) xs := "--------" if reflect.ValueOf(v).Kind() == reflect.Map { xs = " " logT(t, "%s It's a map. Ok that they don't match (dependent on ordering).", xs) } else { logT(t, "%s It's not a map. They should match.", xs) failT(t) } logT(t, "%s FROM_FILE: %4d] %v", xs, len(bss), bss) logT(t, "%s ENCODED: %4d] %v", xs, len(bsb), bsb) } } testMsgpackH.MapType = oldMapType } // To test MsgpackSpecRpc, we test 3 scenarios: // - Go Client to Go RPC Service (contained within TestMsgpackRpcSpec) // - Go client to Python RPC Service (contained within doTestMsgpackRpcSpecGoClientToPythonSvc) // - Python Client to Go RPC Service (contained within doTestMsgpackRpcSpecPythonClientToGoSvc) // // This allows us test the different calling conventions // - Go Service requires only one argument // - Python Service allows multiple arguments func doTestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) { openPort := "6789" cmd := exec.Command("python", "msgpack_test.py", "rpc-server", openPort, "2") checkErrT(t, cmd.Start()) time.Sleep(100 * time.Millisecond) // time for python rpc server to start bs, err2 := net.Dial("tcp", ":"+openPort) checkErrT(t, err2) cc := MsgpackSpecRpc.ClientCodec(bs, testMsgpackH) cl := rpc.NewClientWithCodec(cc) defer cl.Close() var rstr string checkErrT(t, cl.Call("EchoStruct", TestABC{"Aa", "Bb", "Cc"}, &rstr)) //checkEqualT(t, rstr, "{'A': 'Aa', 'B': 'Bb', 'C': 'Cc'}") var mArgs MsgpackSpecRpcMultiArgs = []interface{}{"A1", "B2", "C3"} checkErrT(t, cl.Call("Echo123", mArgs, &rstr)) checkEqualT(t, rstr, "1:A1 2:B2 3:C3", "rstr=") } func doTestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) { port := doTestRpcOne(t, MsgpackSpecRpc, testMsgpackH, false, 1*time.Second) //time.Sleep(1000 * time.Millisecond) cmd := exec.Command("python", "msgpack_test.py", "rpc-client-go-service", strconv.Itoa(port)) var cmdout []byte var err error if cmdout, err = cmd.CombinedOutput(); err != nil { logT(t, "-------- Error running msgpack_test.py rpc-client-go-service. Err: %v", err) logT(t, " %v", string(cmdout)) t.FailNow() } checkEqualT(t, string(cmdout), fmt.Sprintf("%#v\n%#v\n", []string{"A1", "B2", "C3"}, TestABC{"Aa", "Bb", "Cc"}), "cmdout=") } func TestBincCodecsTable(t *testing.T) { testCodecTableOne(t, testBincH) } func TestBincCodecsMisc(t *testing.T) { testCodecMiscOne(t, testBincH) } func TestBincCodecsEmbeddedPointer(t *testing.T) { testCodecEmbeddedPointer(t, testBincH) } func TestSimpleCodecsTable(t *testing.T) { testCodecTableOne(t, testSimpleH) } func TestSimpleCodecsMisc(t *testing.T) { testCodecMiscOne(t, testSimpleH) } func TestSimpleCodecsEmbeddedPointer(t *testing.T) { testCodecEmbeddedPointer(t, testSimpleH) } func TestMsgpackCodecsTable(t *testing.T) { testCodecTableOne(t, testMsgpackH) } func TestMsgpackCodecsMisc(t *testing.T) { testCodecMiscOne(t, testMsgpackH) } func TestMsgpackCodecsEmbeddedPointer(t *testing.T) { testCodecEmbeddedPointer(t, testMsgpackH) } func TestBincRpcGo(t *testing.T) { doTestRpcOne(t, GoRpc, testBincH, true, 0) } func _TestSimpleRpcGo(t *testing.T) { doTestRpcOne(t, GoRpc, testSimpleH, true, 0) } func TestMsgpackRpcGo(t *testing.T) { doTestRpcOne(t, GoRpc, testMsgpackH, true, 0) } func TestMsgpackRpcSpec(t *testing.T) { doTestRpcOne(t, MsgpackSpecRpc, testMsgpackH, true, 0) } // TODO: // Add Tests for: // - decoding empty list/map in stream into a nil slice/map // - binary(M|Unm)arsher support for time.Time golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/decode.go000066400000000000000000000650621270406071200257100ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec import ( "io" "reflect" // "runtime/debug" ) // Some tagging information for error messages. const ( msgTagDec = "codec.decoder" msgBadDesc = "Unrecognized descriptor byte" msgDecCannotExpandArr = "cannot expand go array from %v to stream length: %v" ) // decReader abstracts the reading source, allowing implementations that can // read from an io.Reader or directly off a byte slice with zero-copying. type decReader interface { readn(n int) []byte readb([]byte) readn1() uint8 readUint16() uint16 readUint32() uint32 readUint64() uint64 } type decDriver interface { initReadNext() tryDecodeAsNil() bool currentEncodedType() valueType isBuiltinType(rt uintptr) bool decodeBuiltin(rt uintptr, v interface{}) //decodeNaked: Numbers are decoded as int64, uint64, float64 only (no smaller sized number types). decodeNaked() (v interface{}, vt valueType, decodeFurther bool) decodeInt(bitsize uint8) (i int64) decodeUint(bitsize uint8) (ui uint64) decodeFloat(chkOverflow32 bool) (f float64) decodeBool() (b bool) // decodeString can also decode symbols decodeString() (s string) decodeBytes(bs []byte) (bsOut []byte, changed bool) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte) readMapLen() int readArrayLen() int } type DecodeOptions struct { // An instance of MapType is used during schema-less decoding of a map in the stream. // If nil, we use map[interface{}]interface{} MapType reflect.Type // An instance of SliceType is used during schema-less decoding of an array in the stream. // If nil, we use []interface{} SliceType reflect.Type // ErrorIfNoField controls whether an error is returned when decoding a map // from a codec stream into a struct, and no matching struct field is found. ErrorIfNoField bool } // ------------------------------------ // ioDecReader is a decReader that reads off an io.Reader type ioDecReader struct { r io.Reader br io.ByteReader x [8]byte //temp byte array re-used internally for efficiency } func (z *ioDecReader) readn(n int) (bs []byte) { if n <= 0 { return } bs = make([]byte, n) if _, err := io.ReadAtLeast(z.r, bs, n); err != nil { panic(err) } return } func (z *ioDecReader) readb(bs []byte) { if _, err := io.ReadAtLeast(z.r, bs, len(bs)); err != nil { panic(err) } } func (z *ioDecReader) readn1() uint8 { if z.br != nil { b, err := z.br.ReadByte() if err != nil { panic(err) } return b } z.readb(z.x[:1]) return z.x[0] } func (z *ioDecReader) readUint16() uint16 { z.readb(z.x[:2]) return bigen.Uint16(z.x[:2]) } func (z *ioDecReader) readUint32() uint32 { z.readb(z.x[:4]) return bigen.Uint32(z.x[:4]) } func (z *ioDecReader) readUint64() uint64 { z.readb(z.x[:8]) return bigen.Uint64(z.x[:8]) } // ------------------------------------ // bytesDecReader is a decReader that reads off a byte slice with zero copying type bytesDecReader struct { b []byte // data c int // cursor a int // available } func (z *bytesDecReader) consume(n int) (oldcursor int) { if z.a == 0 { panic(io.EOF) } if n > z.a { decErr("Trying to read %v bytes. Only %v available", n, z.a) } // z.checkAvailable(n) oldcursor = z.c z.c = oldcursor + n z.a = z.a - n return } func (z *bytesDecReader) readn(n int) (bs []byte) { if n <= 0 { return } c0 := z.consume(n) bs = z.b[c0:z.c] return } func (z *bytesDecReader) readb(bs []byte) { copy(bs, z.readn(len(bs))) } func (z *bytesDecReader) readn1() uint8 { c0 := z.consume(1) return z.b[c0] } // Use binaryEncoding helper for 4 and 8 bits, but inline it for 2 bits // creating temp slice variable and copying it to helper function is expensive // for just 2 bits. func (z *bytesDecReader) readUint16() uint16 { c0 := z.consume(2) return uint16(z.b[c0+1]) | uint16(z.b[c0])<<8 } func (z *bytesDecReader) readUint32() uint32 { c0 := z.consume(4) return bigen.Uint32(z.b[c0:z.c]) } func (z *bytesDecReader) readUint64() uint64 { c0 := z.consume(8) return bigen.Uint64(z.b[c0:z.c]) } // ------------------------------------ // decFnInfo has methods for registering handling decoding of a specific type // based on some characteristics (builtin, extension, reflect Kind, etc) type decFnInfo struct { ti *typeInfo d *Decoder dd decDriver xfFn func(reflect.Value, []byte) error xfTag byte array bool } func (f *decFnInfo) builtin(rv reflect.Value) { f.dd.decodeBuiltin(f.ti.rtid, rv.Addr().Interface()) } func (f *decFnInfo) rawExt(rv reflect.Value) { xtag, xbs := f.dd.decodeExt(false, 0) rv.Field(0).SetUint(uint64(xtag)) rv.Field(1).SetBytes(xbs) } func (f *decFnInfo) ext(rv reflect.Value) { _, xbs := f.dd.decodeExt(true, f.xfTag) if fnerr := f.xfFn(rv, xbs); fnerr != nil { panic(fnerr) } } func (f *decFnInfo) binaryMarshal(rv reflect.Value) { var bm binaryUnmarshaler if f.ti.unmIndir == -1 { bm = rv.Addr().Interface().(binaryUnmarshaler) } else if f.ti.unmIndir == 0 { bm = rv.Interface().(binaryUnmarshaler) } else { for j, k := int8(0), f.ti.unmIndir; j < k; j++ { if rv.IsNil() { rv.Set(reflect.New(rv.Type().Elem())) } rv = rv.Elem() } bm = rv.Interface().(binaryUnmarshaler) } xbs, _ := f.dd.decodeBytes(nil) if fnerr := bm.UnmarshalBinary(xbs); fnerr != nil { panic(fnerr) } } func (f *decFnInfo) kErr(rv reflect.Value) { decErr("Unhandled value for kind: %v: %s", rv.Kind(), msgBadDesc) } func (f *decFnInfo) kString(rv reflect.Value) { rv.SetString(f.dd.decodeString()) } func (f *decFnInfo) kBool(rv reflect.Value) { rv.SetBool(f.dd.decodeBool()) } func (f *decFnInfo) kInt(rv reflect.Value) { rv.SetInt(f.dd.decodeInt(intBitsize)) } func (f *decFnInfo) kInt64(rv reflect.Value) { rv.SetInt(f.dd.decodeInt(64)) } func (f *decFnInfo) kInt32(rv reflect.Value) { rv.SetInt(f.dd.decodeInt(32)) } func (f *decFnInfo) kInt8(rv reflect.Value) { rv.SetInt(f.dd.decodeInt(8)) } func (f *decFnInfo) kInt16(rv reflect.Value) { rv.SetInt(f.dd.decodeInt(16)) } func (f *decFnInfo) kFloat32(rv reflect.Value) { rv.SetFloat(f.dd.decodeFloat(true)) } func (f *decFnInfo) kFloat64(rv reflect.Value) { rv.SetFloat(f.dd.decodeFloat(false)) } func (f *decFnInfo) kUint8(rv reflect.Value) { rv.SetUint(f.dd.decodeUint(8)) } func (f *decFnInfo) kUint64(rv reflect.Value) { rv.SetUint(f.dd.decodeUint(64)) } func (f *decFnInfo) kUint(rv reflect.Value) { rv.SetUint(f.dd.decodeUint(uintBitsize)) } func (f *decFnInfo) kUint32(rv reflect.Value) { rv.SetUint(f.dd.decodeUint(32)) } func (f *decFnInfo) kUint16(rv reflect.Value) { rv.SetUint(f.dd.decodeUint(16)) } // func (f *decFnInfo) kPtr(rv reflect.Value) { // debugf(">>>>>>> ??? decode kPtr called - shouldn't get called") // if rv.IsNil() { // rv.Set(reflect.New(rv.Type().Elem())) // } // f.d.decodeValue(rv.Elem()) // } func (f *decFnInfo) kInterface(rv reflect.Value) { // debugf("\t===> kInterface") if !rv.IsNil() { f.d.decodeValue(rv.Elem()) return } // nil interface: // use some hieristics to set the nil interface to an // appropriate value based on the first byte read (byte descriptor bd) v, vt, decodeFurther := f.dd.decodeNaked() if vt == valueTypeNil { return } // Cannot decode into nil interface with methods (e.g. error, io.Reader, etc) // if non-nil value in stream. if num := f.ti.rt.NumMethod(); num > 0 { decErr("decodeValue: Cannot decode non-nil codec value into nil %v (%v methods)", f.ti.rt, num) } var rvn reflect.Value var useRvn bool switch vt { case valueTypeMap: if f.d.h.MapType == nil { var m2 map[interface{}]interface{} v = &m2 } else { rvn = reflect.New(f.d.h.MapType).Elem() useRvn = true } case valueTypeArray: if f.d.h.SliceType == nil { var m2 []interface{} v = &m2 } else { rvn = reflect.New(f.d.h.SliceType).Elem() useRvn = true } case valueTypeExt: re := v.(*RawExt) var bfn func(reflect.Value, []byte) error rvn, bfn = f.d.h.getDecodeExtForTag(re.Tag) if bfn == nil { rvn = reflect.ValueOf(*re) } else if fnerr := bfn(rvn, re.Data); fnerr != nil { panic(fnerr) } rv.Set(rvn) return } if decodeFurther { if useRvn { f.d.decodeValue(rvn) } else if v != nil { // this v is a pointer, so we need to dereference it when done f.d.decode(v) rvn = reflect.ValueOf(v).Elem() useRvn = true } } if useRvn { rv.Set(rvn) } else if v != nil { rv.Set(reflect.ValueOf(v)) } } func (f *decFnInfo) kStruct(rv reflect.Value) { fti := f.ti if currEncodedType := f.dd.currentEncodedType(); currEncodedType == valueTypeMap { containerLen := f.dd.readMapLen() if containerLen == 0 { return } tisfi := fti.sfi for j := 0; j < containerLen; j++ { // var rvkencname string // ddecode(&rvkencname) f.dd.initReadNext() rvkencname := f.dd.decodeString() // rvksi := ti.getForEncName(rvkencname) if k := fti.indexForEncName(rvkencname); k > -1 { sfik := tisfi[k] if sfik.i != -1 { f.d.decodeValue(rv.Field(int(sfik.i))) } else { f.d.decEmbeddedField(rv, sfik.is) } // f.d.decodeValue(ti.field(k, rv)) } else { if f.d.h.ErrorIfNoField { decErr("No matching struct field found when decoding stream map with key: %v", rvkencname) } else { var nilintf0 interface{} f.d.decodeValue(reflect.ValueOf(&nilintf0).Elem()) } } } } else if currEncodedType == valueTypeArray { containerLen := f.dd.readArrayLen() if containerLen == 0 { return } for j, si := range fti.sfip { if j == containerLen { break } if si.i != -1 { f.d.decodeValue(rv.Field(int(si.i))) } else { f.d.decEmbeddedField(rv, si.is) } } if containerLen > len(fti.sfip) { // read remaining values and throw away for j := len(fti.sfip); j < containerLen; j++ { var nilintf0 interface{} f.d.decodeValue(reflect.ValueOf(&nilintf0).Elem()) } } } else { decErr("Only encoded map or array can be decoded into a struct. (valueType: %x)", currEncodedType) } } func (f *decFnInfo) kSlice(rv reflect.Value) { // A slice can be set from a map or array in stream. currEncodedType := f.dd.currentEncodedType() switch currEncodedType { case valueTypeBytes, valueTypeString: if f.ti.rtid == uint8SliceTypId || f.ti.rt.Elem().Kind() == reflect.Uint8 { if bs2, changed2 := f.dd.decodeBytes(rv.Bytes()); changed2 { rv.SetBytes(bs2) } return } } if shortCircuitReflectToFastPath && rv.CanAddr() { switch f.ti.rtid { case intfSliceTypId: f.d.decSliceIntf(rv.Addr().Interface().(*[]interface{}), currEncodedType, f.array) return case uint64SliceTypId: f.d.decSliceUint64(rv.Addr().Interface().(*[]uint64), currEncodedType, f.array) return case int64SliceTypId: f.d.decSliceInt64(rv.Addr().Interface().(*[]int64), currEncodedType, f.array) return case strSliceTypId: f.d.decSliceStr(rv.Addr().Interface().(*[]string), currEncodedType, f.array) return } } containerLen, containerLenS := decContLens(f.dd, currEncodedType) // an array can never return a nil slice. so no need to check f.array here. if rv.IsNil() { rv.Set(reflect.MakeSlice(f.ti.rt, containerLenS, containerLenS)) } if containerLen == 0 { return } if rvcap, rvlen := rv.Len(), rv.Cap(); containerLenS > rvcap { if f.array { // !rv.CanSet() decErr(msgDecCannotExpandArr, rvcap, containerLenS) } rvn := reflect.MakeSlice(f.ti.rt, containerLenS, containerLenS) if rvlen > 0 { reflect.Copy(rvn, rv) } rv.Set(rvn) } else if containerLenS > rvlen { rv.SetLen(containerLenS) } for j := 0; j < containerLenS; j++ { f.d.decodeValue(rv.Index(j)) } } func (f *decFnInfo) kArray(rv reflect.Value) { // f.d.decodeValue(rv.Slice(0, rv.Len())) f.kSlice(rv.Slice(0, rv.Len())) } func (f *decFnInfo) kMap(rv reflect.Value) { if shortCircuitReflectToFastPath && rv.CanAddr() { switch f.ti.rtid { case mapStrIntfTypId: f.d.decMapStrIntf(rv.Addr().Interface().(*map[string]interface{})) return case mapIntfIntfTypId: f.d.decMapIntfIntf(rv.Addr().Interface().(*map[interface{}]interface{})) return case mapInt64IntfTypId: f.d.decMapInt64Intf(rv.Addr().Interface().(*map[int64]interface{})) return case mapUint64IntfTypId: f.d.decMapUint64Intf(rv.Addr().Interface().(*map[uint64]interface{})) return } } containerLen := f.dd.readMapLen() if rv.IsNil() { rv.Set(reflect.MakeMap(f.ti.rt)) } if containerLen == 0 { return } ktype, vtype := f.ti.rt.Key(), f.ti.rt.Elem() ktypeId := reflect.ValueOf(ktype).Pointer() for j := 0; j < containerLen; j++ { rvk := reflect.New(ktype).Elem() f.d.decodeValue(rvk) // special case if a byte array. // if ktype == intfTyp { if ktypeId == intfTypId { rvk = rvk.Elem() if rvk.Type() == uint8SliceTyp { rvk = reflect.ValueOf(string(rvk.Bytes())) } } rvv := rv.MapIndex(rvk) if !rvv.IsValid() { rvv = reflect.New(vtype).Elem() } f.d.decodeValue(rvv) rv.SetMapIndex(rvk, rvv) } } // ---------------------------------------- type decFn struct { i *decFnInfo f func(*decFnInfo, reflect.Value) } // A Decoder reads and decodes an object from an input stream in the codec format. type Decoder struct { r decReader d decDriver h *BasicHandle f map[uintptr]decFn x []uintptr s []decFn } // NewDecoder returns a Decoder for decoding a stream of bytes from an io.Reader. // // For efficiency, Users are encouraged to pass in a memory buffered writer // (eg bufio.Reader, bytes.Buffer). func NewDecoder(r io.Reader, h Handle) *Decoder { z := ioDecReader{ r: r, } z.br, _ = r.(io.ByteReader) return &Decoder{r: &z, d: h.newDecDriver(&z), h: h.getBasicHandle()} } // NewDecoderBytes returns a Decoder which efficiently decodes directly // from a byte slice with zero copying. func NewDecoderBytes(in []byte, h Handle) *Decoder { z := bytesDecReader{ b: in, a: len(in), } return &Decoder{r: &z, d: h.newDecDriver(&z), h: h.getBasicHandle()} } // Decode decodes the stream from reader and stores the result in the // value pointed to by v. v cannot be a nil pointer. v can also be // a reflect.Value of a pointer. // // Note that a pointer to a nil interface is not a nil pointer. // If you do not know what type of stream it is, pass in a pointer to a nil interface. // We will decode and store a value in that nil interface. // // Sample usages: // // Decoding into a non-nil typed value // var f float32 // err = codec.NewDecoder(r, handle).Decode(&f) // // // Decoding into nil interface // var v interface{} // dec := codec.NewDecoder(r, handle) // err = dec.Decode(&v) // // When decoding into a nil interface{}, we will decode into an appropriate value based // on the contents of the stream: // - Numbers are decoded as float64, int64 or uint64. // - Other values are decoded appropriately depending on the type: // bool, string, []byte, time.Time, etc // - Extensions are decoded as RawExt (if no ext function registered for the tag) // Configurations exist on the Handle to override defaults // (e.g. for MapType, SliceType and how to decode raw bytes). // // When decoding into a non-nil interface{} value, the mode of encoding is based on the // type of the value. When a value is seen: // - If an extension is registered for it, call that extension function // - If it implements BinaryUnmarshaler, call its UnmarshalBinary(data []byte) error // - Else decode it based on its reflect.Kind // // There are some special rules when decoding into containers (slice/array/map/struct). // Decode will typically use the stream contents to UPDATE the container. // - A map can be decoded from a stream map, by updating matching keys. // - A slice can be decoded from a stream array, // by updating the first n elements, where n is length of the stream. // - A slice can be decoded from a stream map, by decoding as if // it contains a sequence of key-value pairs. // - A struct can be decoded from a stream map, by updating matching fields. // - A struct can be decoded from a stream array, // by updating fields as they occur in the struct (by index). // // When decoding a stream map or array with length of 0 into a nil map or slice, // we reset the destination map or slice to a zero-length value. // // However, when decoding a stream nil, we reset the destination container // to its "zero" value (e.g. nil for slice/map, etc). // func (d *Decoder) Decode(v interface{}) (err error) { defer panicToErr(&err) d.decode(v) return } func (d *Decoder) decode(iv interface{}) { d.d.initReadNext() switch v := iv.(type) { case nil: decErr("Cannot decode into nil.") case reflect.Value: d.chkPtrValue(v) d.decodeValue(v.Elem()) case *string: *v = d.d.decodeString() case *bool: *v = d.d.decodeBool() case *int: *v = int(d.d.decodeInt(intBitsize)) case *int8: *v = int8(d.d.decodeInt(8)) case *int16: *v = int16(d.d.decodeInt(16)) case *int32: *v = int32(d.d.decodeInt(32)) case *int64: *v = d.d.decodeInt(64) case *uint: *v = uint(d.d.decodeUint(uintBitsize)) case *uint8: *v = uint8(d.d.decodeUint(8)) case *uint16: *v = uint16(d.d.decodeUint(16)) case *uint32: *v = uint32(d.d.decodeUint(32)) case *uint64: *v = d.d.decodeUint(64) case *float32: *v = float32(d.d.decodeFloat(true)) case *float64: *v = d.d.decodeFloat(false) case *[]byte: *v, _ = d.d.decodeBytes(*v) case *[]interface{}: d.decSliceIntf(v, valueTypeInvalid, false) case *[]uint64: d.decSliceUint64(v, valueTypeInvalid, false) case *[]int64: d.decSliceInt64(v, valueTypeInvalid, false) case *[]string: d.decSliceStr(v, valueTypeInvalid, false) case *map[string]interface{}: d.decMapStrIntf(v) case *map[interface{}]interface{}: d.decMapIntfIntf(v) case *map[uint64]interface{}: d.decMapUint64Intf(v) case *map[int64]interface{}: d.decMapInt64Intf(v) case *interface{}: d.decodeValue(reflect.ValueOf(iv).Elem()) default: rv := reflect.ValueOf(iv) d.chkPtrValue(rv) d.decodeValue(rv.Elem()) } } func (d *Decoder) decodeValue(rv reflect.Value) { d.d.initReadNext() if d.d.tryDecodeAsNil() { // If value in stream is nil, set the dereferenced value to its "zero" value (if settable) if rv.Kind() == reflect.Ptr { if !rv.IsNil() { rv.Set(reflect.Zero(rv.Type())) } return } // for rv.Kind() == reflect.Ptr { // rv = rv.Elem() // } if rv.IsValid() { // rv.CanSet() // always settable, except it's invalid rv.Set(reflect.Zero(rv.Type())) } return } // If stream is not containing a nil value, then we can deref to the base // non-pointer value, and decode into that. for rv.Kind() == reflect.Ptr { if rv.IsNil() { rv.Set(reflect.New(rv.Type().Elem())) } rv = rv.Elem() } rt := rv.Type() rtid := reflect.ValueOf(rt).Pointer() // retrieve or register a focus'ed function for this type // to eliminate need to do the retrieval multiple times // if d.f == nil && d.s == nil { debugf("---->Creating new dec f map for type: %v\n", rt) } var fn decFn var ok bool if useMapForCodecCache { fn, ok = d.f[rtid] } else { for i, v := range d.x { if v == rtid { fn, ok = d.s[i], true break } } } if !ok { // debugf("\tCreating new dec fn for type: %v\n", rt) fi := decFnInfo{ti: getTypeInfo(rtid, rt), d: d, dd: d.d} fn.i = &fi // An extension can be registered for any type, regardless of the Kind // (e.g. type BitSet int64, type MyStruct { / * unexported fields * / }, type X []int, etc. // // We can't check if it's an extension byte here first, because the user may have // registered a pointer or non-pointer type, meaning we may have to recurse first // before matching a mapped type, even though the extension byte is already detected. // // NOTE: if decoding into a nil interface{}, we return a non-nil // value except even if the container registers a length of 0. if rtid == rawExtTypId { fn.f = (*decFnInfo).rawExt } else if d.d.isBuiltinType(rtid) { fn.f = (*decFnInfo).builtin } else if xfTag, xfFn := d.h.getDecodeExt(rtid); xfFn != nil { fi.xfTag, fi.xfFn = xfTag, xfFn fn.f = (*decFnInfo).ext } else if supportBinaryMarshal && fi.ti.unm { fn.f = (*decFnInfo).binaryMarshal } else { switch rk := rt.Kind(); rk { case reflect.String: fn.f = (*decFnInfo).kString case reflect.Bool: fn.f = (*decFnInfo).kBool case reflect.Int: fn.f = (*decFnInfo).kInt case reflect.Int64: fn.f = (*decFnInfo).kInt64 case reflect.Int32: fn.f = (*decFnInfo).kInt32 case reflect.Int8: fn.f = (*decFnInfo).kInt8 case reflect.Int16: fn.f = (*decFnInfo).kInt16 case reflect.Float32: fn.f = (*decFnInfo).kFloat32 case reflect.Float64: fn.f = (*decFnInfo).kFloat64 case reflect.Uint8: fn.f = (*decFnInfo).kUint8 case reflect.Uint64: fn.f = (*decFnInfo).kUint64 case reflect.Uint: fn.f = (*decFnInfo).kUint case reflect.Uint32: fn.f = (*decFnInfo).kUint32 case reflect.Uint16: fn.f = (*decFnInfo).kUint16 // case reflect.Ptr: // fn.f = (*decFnInfo).kPtr case reflect.Interface: fn.f = (*decFnInfo).kInterface case reflect.Struct: fn.f = (*decFnInfo).kStruct case reflect.Slice: fn.f = (*decFnInfo).kSlice case reflect.Array: fi.array = true fn.f = (*decFnInfo).kArray case reflect.Map: fn.f = (*decFnInfo).kMap default: fn.f = (*decFnInfo).kErr } } if useMapForCodecCache { if d.f == nil { d.f = make(map[uintptr]decFn, 16) } d.f[rtid] = fn } else { d.s = append(d.s, fn) d.x = append(d.x, rtid) } } fn.f(fn.i, rv) return } func (d *Decoder) chkPtrValue(rv reflect.Value) { // We can only decode into a non-nil pointer if rv.Kind() == reflect.Ptr && !rv.IsNil() { return } if !rv.IsValid() { decErr("Cannot decode into a zero (ie invalid) reflect.Value") } if !rv.CanInterface() { decErr("Cannot decode into a value without an interface: %v", rv) } rvi := rv.Interface() decErr("Cannot decode into non-pointer or nil pointer. Got: %v, %T, %v", rv.Kind(), rvi, rvi) } func (d *Decoder) decEmbeddedField(rv reflect.Value, index []int) { // d.decodeValue(rv.FieldByIndex(index)) // nil pointers may be here; so reproduce FieldByIndex logic + enhancements for _, j := range index { if rv.Kind() == reflect.Ptr { if rv.IsNil() { rv.Set(reflect.New(rv.Type().Elem())) } // If a pointer, it must be a pointer to struct (based on typeInfo contract) rv = rv.Elem() } rv = rv.Field(j) } d.decodeValue(rv) } // -------------------------------------------------- // short circuit functions for common maps and slices func (d *Decoder) decSliceIntf(v *[]interface{}, currEncodedType valueType, doNotReset bool) { _, containerLenS := decContLens(d.d, currEncodedType) s := *v if s == nil { s = make([]interface{}, containerLenS, containerLenS) } else if containerLenS > cap(s) { if doNotReset { decErr(msgDecCannotExpandArr, cap(s), containerLenS) } s = make([]interface{}, containerLenS, containerLenS) copy(s, *v) } else if containerLenS > len(s) { s = s[:containerLenS] } for j := 0; j < containerLenS; j++ { d.decode(&s[j]) } *v = s } func (d *Decoder) decSliceInt64(v *[]int64, currEncodedType valueType, doNotReset bool) { _, containerLenS := decContLens(d.d, currEncodedType) s := *v if s == nil { s = make([]int64, containerLenS, containerLenS) } else if containerLenS > cap(s) { if doNotReset { decErr(msgDecCannotExpandArr, cap(s), containerLenS) } s = make([]int64, containerLenS, containerLenS) copy(s, *v) } else if containerLenS > len(s) { s = s[:containerLenS] } for j := 0; j < containerLenS; j++ { // d.decode(&s[j]) d.d.initReadNext() s[j] = d.d.decodeInt(intBitsize) } *v = s } func (d *Decoder) decSliceUint64(v *[]uint64, currEncodedType valueType, doNotReset bool) { _, containerLenS := decContLens(d.d, currEncodedType) s := *v if s == nil { s = make([]uint64, containerLenS, containerLenS) } else if containerLenS > cap(s) { if doNotReset { decErr(msgDecCannotExpandArr, cap(s), containerLenS) } s = make([]uint64, containerLenS, containerLenS) copy(s, *v) } else if containerLenS > len(s) { s = s[:containerLenS] } for j := 0; j < containerLenS; j++ { // d.decode(&s[j]) d.d.initReadNext() s[j] = d.d.decodeUint(intBitsize) } *v = s } func (d *Decoder) decSliceStr(v *[]string, currEncodedType valueType, doNotReset bool) { _, containerLenS := decContLens(d.d, currEncodedType) s := *v if s == nil { s = make([]string, containerLenS, containerLenS) } else if containerLenS > cap(s) { if doNotReset { decErr(msgDecCannotExpandArr, cap(s), containerLenS) } s = make([]string, containerLenS, containerLenS) copy(s, *v) } else if containerLenS > len(s) { s = s[:containerLenS] } for j := 0; j < containerLenS; j++ { // d.decode(&s[j]) d.d.initReadNext() s[j] = d.d.decodeString() } *v = s } func (d *Decoder) decMapIntfIntf(v *map[interface{}]interface{}) { containerLen := d.d.readMapLen() m := *v if m == nil { m = make(map[interface{}]interface{}, containerLen) *v = m } for j := 0; j < containerLen; j++ { var mk interface{} d.decode(&mk) // special case if a byte array. if bv, bok := mk.([]byte); bok { mk = string(bv) } mv := m[mk] d.decode(&mv) m[mk] = mv } } func (d *Decoder) decMapInt64Intf(v *map[int64]interface{}) { containerLen := d.d.readMapLen() m := *v if m == nil { m = make(map[int64]interface{}, containerLen) *v = m } for j := 0; j < containerLen; j++ { d.d.initReadNext() mk := d.d.decodeInt(intBitsize) mv := m[mk] d.decode(&mv) m[mk] = mv } } func (d *Decoder) decMapUint64Intf(v *map[uint64]interface{}) { containerLen := d.d.readMapLen() m := *v if m == nil { m = make(map[uint64]interface{}, containerLen) *v = m } for j := 0; j < containerLen; j++ { d.d.initReadNext() mk := d.d.decodeUint(intBitsize) mv := m[mk] d.decode(&mv) m[mk] = mv } } func (d *Decoder) decMapStrIntf(v *map[string]interface{}) { containerLen := d.d.readMapLen() m := *v if m == nil { m = make(map[string]interface{}, containerLen) *v = m } for j := 0; j < containerLen; j++ { d.d.initReadNext() mk := d.d.decodeString() mv := m[mk] d.decode(&mv) m[mk] = mv } } // ---------------------------------------- func decContLens(dd decDriver, currEncodedType valueType) (containerLen, containerLenS int) { if currEncodedType == valueTypeInvalid { currEncodedType = dd.currentEncodedType() } switch currEncodedType { case valueTypeArray: containerLen = dd.readArrayLen() containerLenS = containerLen case valueTypeMap: containerLen = dd.readMapLen() containerLenS = containerLen * 2 default: decErr("Only encoded map or array can be decoded into a slice. (valueType: %0x)", currEncodedType) } return } func decErr(format string, params ...interface{}) { doPanic(msgTagDec, format, params...) } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/encode.go000066400000000000000000000570761270406071200257300ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec import ( "io" "reflect" ) const ( // Some tagging information for error messages. msgTagEnc = "codec.encoder" defEncByteBufSize = 1 << 6 // 4:16, 6:64, 8:256, 10:1024 // maxTimeSecs32 = math.MaxInt32 / 60 / 24 / 366 ) // AsSymbolFlag defines what should be encoded as symbols. type AsSymbolFlag uint8 const ( // AsSymbolDefault is default. // Currently, this means only encode struct field names as symbols. // The default is subject to change. AsSymbolDefault AsSymbolFlag = iota // AsSymbolAll means encode anything which could be a symbol as a symbol. AsSymbolAll = 0xfe // AsSymbolNone means do not encode anything as a symbol. AsSymbolNone = 1 << iota // AsSymbolMapStringKeys means encode keys in map[string]XXX as symbols. AsSymbolMapStringKeysFlag // AsSymbolStructFieldName means encode struct field names as symbols. AsSymbolStructFieldNameFlag ) // encWriter abstracting writing to a byte array or to an io.Writer. type encWriter interface { writeUint16(uint16) writeUint32(uint32) writeUint64(uint64) writeb([]byte) writestr(string) writen1(byte) writen2(byte, byte) atEndOfEncode() } // encDriver abstracts the actual codec (binc vs msgpack, etc) type encDriver interface { isBuiltinType(rt uintptr) bool encodeBuiltin(rt uintptr, v interface{}) encodeNil() encodeInt(i int64) encodeUint(i uint64) encodeBool(b bool) encodeFloat32(f float32) encodeFloat64(f float64) encodeExtPreamble(xtag byte, length int) encodeArrayPreamble(length int) encodeMapPreamble(length int) encodeString(c charEncoding, v string) encodeSymbol(v string) encodeStringBytes(c charEncoding, v []byte) //TODO //encBignum(f *big.Int) //encStringRunes(c charEncoding, v []rune) } type ioEncWriterWriter interface { WriteByte(c byte) error WriteString(s string) (n int, err error) Write(p []byte) (n int, err error) } type ioEncStringWriter interface { WriteString(s string) (n int, err error) } type EncodeOptions struct { // Encode a struct as an array, and not as a map. StructToArray bool // AsSymbols defines what should be encoded as symbols. // // Encoding as symbols can reduce the encoded size significantly. // // However, during decoding, each string to be encoded as a symbol must // be checked to see if it has been seen before. Consequently, encoding time // will increase if using symbols, because string comparisons has a clear cost. // // Sample values: // AsSymbolNone // AsSymbolAll // AsSymbolMapStringKeys // AsSymbolMapStringKeysFlag | AsSymbolStructFieldNameFlag AsSymbols AsSymbolFlag } // --------------------------------------------- type simpleIoEncWriterWriter struct { w io.Writer bw io.ByteWriter sw ioEncStringWriter } func (o *simpleIoEncWriterWriter) WriteByte(c byte) (err error) { if o.bw != nil { return o.bw.WriteByte(c) } _, err = o.w.Write([]byte{c}) return } func (o *simpleIoEncWriterWriter) WriteString(s string) (n int, err error) { if o.sw != nil { return o.sw.WriteString(s) } return o.w.Write([]byte(s)) } func (o *simpleIoEncWriterWriter) Write(p []byte) (n int, err error) { return o.w.Write(p) } // ---------------------------------------- // ioEncWriter implements encWriter and can write to an io.Writer implementation type ioEncWriter struct { w ioEncWriterWriter x [8]byte // temp byte array re-used internally for efficiency } func (z *ioEncWriter) writeUint16(v uint16) { bigen.PutUint16(z.x[:2], v) z.writeb(z.x[:2]) } func (z *ioEncWriter) writeUint32(v uint32) { bigen.PutUint32(z.x[:4], v) z.writeb(z.x[:4]) } func (z *ioEncWriter) writeUint64(v uint64) { bigen.PutUint64(z.x[:8], v) z.writeb(z.x[:8]) } func (z *ioEncWriter) writeb(bs []byte) { if len(bs) == 0 { return } n, err := z.w.Write(bs) if err != nil { panic(err) } if n != len(bs) { encErr("write: Incorrect num bytes written. Expecting: %v, Wrote: %v", len(bs), n) } } func (z *ioEncWriter) writestr(s string) { n, err := z.w.WriteString(s) if err != nil { panic(err) } if n != len(s) { encErr("write: Incorrect num bytes written. Expecting: %v, Wrote: %v", len(s), n) } } func (z *ioEncWriter) writen1(b byte) { if err := z.w.WriteByte(b); err != nil { panic(err) } } func (z *ioEncWriter) writen2(b1 byte, b2 byte) { z.writen1(b1) z.writen1(b2) } func (z *ioEncWriter) atEndOfEncode() {} // ---------------------------------------- // bytesEncWriter implements encWriter and can write to an byte slice. // It is used by Marshal function. type bytesEncWriter struct { b []byte c int // cursor out *[]byte // write out on atEndOfEncode } func (z *bytesEncWriter) writeUint16(v uint16) { c := z.grow(2) z.b[c] = byte(v >> 8) z.b[c+1] = byte(v) } func (z *bytesEncWriter) writeUint32(v uint32) { c := z.grow(4) z.b[c] = byte(v >> 24) z.b[c+1] = byte(v >> 16) z.b[c+2] = byte(v >> 8) z.b[c+3] = byte(v) } func (z *bytesEncWriter) writeUint64(v uint64) { c := z.grow(8) z.b[c] = byte(v >> 56) z.b[c+1] = byte(v >> 48) z.b[c+2] = byte(v >> 40) z.b[c+3] = byte(v >> 32) z.b[c+4] = byte(v >> 24) z.b[c+5] = byte(v >> 16) z.b[c+6] = byte(v >> 8) z.b[c+7] = byte(v) } func (z *bytesEncWriter) writeb(s []byte) { if len(s) == 0 { return } c := z.grow(len(s)) copy(z.b[c:], s) } func (z *bytesEncWriter) writestr(s string) { c := z.grow(len(s)) copy(z.b[c:], s) } func (z *bytesEncWriter) writen1(b1 byte) { c := z.grow(1) z.b[c] = b1 } func (z *bytesEncWriter) writen2(b1 byte, b2 byte) { c := z.grow(2) z.b[c] = b1 z.b[c+1] = b2 } func (z *bytesEncWriter) atEndOfEncode() { *(z.out) = z.b[:z.c] } func (z *bytesEncWriter) grow(n int) (oldcursor int) { oldcursor = z.c z.c = oldcursor + n if z.c > cap(z.b) { // Tried using appendslice logic: (if cap < 1024, *2, else *1.25). // However, it was too expensive, causing too many iterations of copy. // Using bytes.Buffer model was much better (2*cap + n) bs := make([]byte, 2*cap(z.b)+n) copy(bs, z.b[:oldcursor]) z.b = bs } else if z.c > len(z.b) { z.b = z.b[:cap(z.b)] } return } // --------------------------------------------- type encFnInfo struct { ti *typeInfo e *Encoder ee encDriver xfFn func(reflect.Value) ([]byte, error) xfTag byte } func (f *encFnInfo) builtin(rv reflect.Value) { f.ee.encodeBuiltin(f.ti.rtid, rv.Interface()) } func (f *encFnInfo) rawExt(rv reflect.Value) { f.e.encRawExt(rv.Interface().(RawExt)) } func (f *encFnInfo) ext(rv reflect.Value) { bs, fnerr := f.xfFn(rv) if fnerr != nil { panic(fnerr) } if bs == nil { f.ee.encodeNil() return } if f.e.hh.writeExt() { f.ee.encodeExtPreamble(f.xfTag, len(bs)) f.e.w.writeb(bs) } else { f.ee.encodeStringBytes(c_RAW, bs) } } func (f *encFnInfo) binaryMarshal(rv reflect.Value) { var bm binaryMarshaler if f.ti.mIndir == 0 { bm = rv.Interface().(binaryMarshaler) } else if f.ti.mIndir == -1 { bm = rv.Addr().Interface().(binaryMarshaler) } else { for j, k := int8(0), f.ti.mIndir; j < k; j++ { if rv.IsNil() { f.ee.encodeNil() return } rv = rv.Elem() } bm = rv.Interface().(binaryMarshaler) } // debugf(">>>> binaryMarshaler: %T", rv.Interface()) bs, fnerr := bm.MarshalBinary() if fnerr != nil { panic(fnerr) } if bs == nil { f.ee.encodeNil() } else { f.ee.encodeStringBytes(c_RAW, bs) } } func (f *encFnInfo) kBool(rv reflect.Value) { f.ee.encodeBool(rv.Bool()) } func (f *encFnInfo) kString(rv reflect.Value) { f.ee.encodeString(c_UTF8, rv.String()) } func (f *encFnInfo) kFloat64(rv reflect.Value) { f.ee.encodeFloat64(rv.Float()) } func (f *encFnInfo) kFloat32(rv reflect.Value) { f.ee.encodeFloat32(float32(rv.Float())) } func (f *encFnInfo) kInt(rv reflect.Value) { f.ee.encodeInt(rv.Int()) } func (f *encFnInfo) kUint(rv reflect.Value) { f.ee.encodeUint(rv.Uint()) } func (f *encFnInfo) kInvalid(rv reflect.Value) { f.ee.encodeNil() } func (f *encFnInfo) kErr(rv reflect.Value) { encErr("Unsupported kind: %s, for: %#v", rv.Kind(), rv) } func (f *encFnInfo) kSlice(rv reflect.Value) { if rv.IsNil() { f.ee.encodeNil() return } if shortCircuitReflectToFastPath { switch f.ti.rtid { case intfSliceTypId: f.e.encSliceIntf(rv.Interface().([]interface{})) return case strSliceTypId: f.e.encSliceStr(rv.Interface().([]string)) return case uint64SliceTypId: f.e.encSliceUint64(rv.Interface().([]uint64)) return case int64SliceTypId: f.e.encSliceInt64(rv.Interface().([]int64)) return } } // If in this method, then there was no extension function defined. // So it's okay to treat as []byte. if f.ti.rtid == uint8SliceTypId || f.ti.rt.Elem().Kind() == reflect.Uint8 { f.ee.encodeStringBytes(c_RAW, rv.Bytes()) return } l := rv.Len() if f.ti.mbs { if l%2 == 1 { encErr("mapBySlice: invalid length (must be divisible by 2): %v", l) } f.ee.encodeMapPreamble(l / 2) } else { f.ee.encodeArrayPreamble(l) } if l == 0 { return } for j := 0; j < l; j++ { // TODO: Consider perf implication of encoding odd index values as symbols if type is string f.e.encodeValue(rv.Index(j)) } } func (f *encFnInfo) kArray(rv reflect.Value) { // We cannot share kSlice method, because the array may be non-addressable. // E.g. type struct S{B [2]byte}; Encode(S{}) will bomb on "panic: slice of unaddressable array". // So we have to duplicate the functionality here. // f.e.encodeValue(rv.Slice(0, rv.Len())) // f.kSlice(rv.Slice(0, rv.Len())) l := rv.Len() // Handle an array of bytes specially (in line with what is done for slices) if f.ti.rt.Elem().Kind() == reflect.Uint8 { if l == 0 { f.ee.encodeStringBytes(c_RAW, nil) return } var bs []byte if rv.CanAddr() { bs = rv.Slice(0, l).Bytes() } else { bs = make([]byte, l) for i := 0; i < l; i++ { bs[i] = byte(rv.Index(i).Uint()) } } f.ee.encodeStringBytes(c_RAW, bs) return } if f.ti.mbs { if l%2 == 1 { encErr("mapBySlice: invalid length (must be divisible by 2): %v", l) } f.ee.encodeMapPreamble(l / 2) } else { f.ee.encodeArrayPreamble(l) } if l == 0 { return } for j := 0; j < l; j++ { // TODO: Consider perf implication of encoding odd index values as symbols if type is string f.e.encodeValue(rv.Index(j)) } } func (f *encFnInfo) kStruct(rv reflect.Value) { fti := f.ti newlen := len(fti.sfi) rvals := make([]reflect.Value, newlen) var encnames []string e := f.e tisfi := fti.sfip toMap := !(fti.toArray || e.h.StructToArray) // if toMap, use the sorted array. If toArray, use unsorted array (to match sequence in struct) if toMap { tisfi = fti.sfi encnames = make([]string, newlen) } newlen = 0 for _, si := range tisfi { if si.i != -1 { rvals[newlen] = rv.Field(int(si.i)) } else { rvals[newlen] = rv.FieldByIndex(si.is) } if toMap { if si.omitEmpty && isEmptyValue(rvals[newlen]) { continue } encnames[newlen] = si.encName } else { if si.omitEmpty && isEmptyValue(rvals[newlen]) { rvals[newlen] = reflect.Value{} //encode as nil } } newlen++ } // debugf(">>>> kStruct: newlen: %v", newlen) if toMap { ee := f.ee //don't dereference everytime ee.encodeMapPreamble(newlen) // asSymbols := e.h.AsSymbols&AsSymbolStructFieldNameFlag != 0 asSymbols := e.h.AsSymbols == AsSymbolDefault || e.h.AsSymbols&AsSymbolStructFieldNameFlag != 0 for j := 0; j < newlen; j++ { if asSymbols { ee.encodeSymbol(encnames[j]) } else { ee.encodeString(c_UTF8, encnames[j]) } e.encodeValue(rvals[j]) } } else { f.ee.encodeArrayPreamble(newlen) for j := 0; j < newlen; j++ { e.encodeValue(rvals[j]) } } } // func (f *encFnInfo) kPtr(rv reflect.Value) { // debugf(">>>>>>> ??? encode kPtr called - shouldn't get called") // if rv.IsNil() { // f.ee.encodeNil() // return // } // f.e.encodeValue(rv.Elem()) // } func (f *encFnInfo) kInterface(rv reflect.Value) { if rv.IsNil() { f.ee.encodeNil() return } f.e.encodeValue(rv.Elem()) } func (f *encFnInfo) kMap(rv reflect.Value) { if rv.IsNil() { f.ee.encodeNil() return } if shortCircuitReflectToFastPath { switch f.ti.rtid { case mapIntfIntfTypId: f.e.encMapIntfIntf(rv.Interface().(map[interface{}]interface{})) return case mapStrIntfTypId: f.e.encMapStrIntf(rv.Interface().(map[string]interface{})) return case mapStrStrTypId: f.e.encMapStrStr(rv.Interface().(map[string]string)) return case mapInt64IntfTypId: f.e.encMapInt64Intf(rv.Interface().(map[int64]interface{})) return case mapUint64IntfTypId: f.e.encMapUint64Intf(rv.Interface().(map[uint64]interface{})) return } } l := rv.Len() f.ee.encodeMapPreamble(l) if l == 0 { return } // keyTypeIsString := f.ti.rt.Key().Kind() == reflect.String keyTypeIsString := f.ti.rt.Key() == stringTyp var asSymbols bool if keyTypeIsString { asSymbols = f.e.h.AsSymbols&AsSymbolMapStringKeysFlag != 0 } mks := rv.MapKeys() // for j, lmks := 0, len(mks); j < lmks; j++ { for j := range mks { if keyTypeIsString { if asSymbols { f.ee.encodeSymbol(mks[j].String()) } else { f.ee.encodeString(c_UTF8, mks[j].String()) } } else { f.e.encodeValue(mks[j]) } f.e.encodeValue(rv.MapIndex(mks[j])) } } // -------------------------------------------------- // encFn encapsulates the captured variables and the encode function. // This way, we only do some calculations one times, and pass to the // code block that should be called (encapsulated in a function) // instead of executing the checks every time. type encFn struct { i *encFnInfo f func(*encFnInfo, reflect.Value) } // -------------------------------------------------- // An Encoder writes an object to an output stream in the codec format. type Encoder struct { w encWriter e encDriver h *BasicHandle hh Handle f map[uintptr]encFn x []uintptr s []encFn } // NewEncoder returns an Encoder for encoding into an io.Writer. // // For efficiency, Users are encouraged to pass in a memory buffered writer // (eg bufio.Writer, bytes.Buffer). func NewEncoder(w io.Writer, h Handle) *Encoder { ww, ok := w.(ioEncWriterWriter) if !ok { sww := simpleIoEncWriterWriter{w: w} sww.bw, _ = w.(io.ByteWriter) sww.sw, _ = w.(ioEncStringWriter) ww = &sww //ww = bufio.NewWriterSize(w, defEncByteBufSize) } z := ioEncWriter{ w: ww, } return &Encoder{w: &z, hh: h, h: h.getBasicHandle(), e: h.newEncDriver(&z)} } // NewEncoderBytes returns an encoder for encoding directly and efficiently // into a byte slice, using zero-copying to temporary slices. // // It will potentially replace the output byte slice pointed to. // After encoding, the out parameter contains the encoded contents. func NewEncoderBytes(out *[]byte, h Handle) *Encoder { in := *out if in == nil { in = make([]byte, defEncByteBufSize) } z := bytesEncWriter{ b: in, out: out, } return &Encoder{w: &z, hh: h, h: h.getBasicHandle(), e: h.newEncDriver(&z)} } // Encode writes an object into a stream in the codec format. // // Encoding can be configured via the "codec" struct tag for the fields. // // The "codec" key in struct field's tag value is the key name, // followed by an optional comma and options. // // To set an option on all fields (e.g. omitempty on all fields), you // can create a field called _struct, and set flags on it. // // Struct values "usually" encode as maps. Each exported struct field is encoded unless: // - the field's codec tag is "-", OR // - the field is empty and its codec tag specifies the "omitempty" option. // // When encoding as a map, the first string in the tag (before the comma) // is the map key string to use when encoding. // // However, struct values may encode as arrays. This happens when: // - StructToArray Encode option is set, OR // - the codec tag on the _struct field sets the "toarray" option // // Values with types that implement MapBySlice are encoded as stream maps. // // The empty values (for omitempty option) are false, 0, any nil pointer // or interface value, and any array, slice, map, or string of length zero. // // Anonymous fields are encoded inline if no struct tag is present. // Else they are encoded as regular fields. // // Examples: // // type MyStruct struct { // _struct bool `codec:",omitempty"` //set omitempty for every field // Field1 string `codec:"-"` //skip this field // Field2 int `codec:"myName"` //Use key "myName" in encode stream // Field3 int32 `codec:",omitempty"` //use key "Field3". Omit if empty. // Field4 bool `codec:"f4,omitempty"` //use key "f4". Omit if empty. // ... // } // // type MyStruct struct { // _struct bool `codec:",omitempty,toarray"` //set omitempty for every field // //and encode struct as an array // } // // The mode of encoding is based on the type of the value. When a value is seen: // - If an extension is registered for it, call that extension function // - If it implements BinaryMarshaler, call its MarshalBinary() (data []byte, err error) // - Else encode it based on its reflect.Kind // // Note that struct field names and keys in map[string]XXX will be treated as symbols. // Some formats support symbols (e.g. binc) and will properly encode the string // only once in the stream, and use a tag to refer to it thereafter. func (e *Encoder) Encode(v interface{}) (err error) { defer panicToErr(&err) e.encode(v) e.w.atEndOfEncode() return } func (e *Encoder) encode(iv interface{}) { switch v := iv.(type) { case nil: e.e.encodeNil() case reflect.Value: e.encodeValue(v) case string: e.e.encodeString(c_UTF8, v) case bool: e.e.encodeBool(v) case int: e.e.encodeInt(int64(v)) case int8: e.e.encodeInt(int64(v)) case int16: e.e.encodeInt(int64(v)) case int32: e.e.encodeInt(int64(v)) case int64: e.e.encodeInt(v) case uint: e.e.encodeUint(uint64(v)) case uint8: e.e.encodeUint(uint64(v)) case uint16: e.e.encodeUint(uint64(v)) case uint32: e.e.encodeUint(uint64(v)) case uint64: e.e.encodeUint(v) case float32: e.e.encodeFloat32(v) case float64: e.e.encodeFloat64(v) case []interface{}: e.encSliceIntf(v) case []string: e.encSliceStr(v) case []int64: e.encSliceInt64(v) case []uint64: e.encSliceUint64(v) case []uint8: e.e.encodeStringBytes(c_RAW, v) case map[interface{}]interface{}: e.encMapIntfIntf(v) case map[string]interface{}: e.encMapStrIntf(v) case map[string]string: e.encMapStrStr(v) case map[int64]interface{}: e.encMapInt64Intf(v) case map[uint64]interface{}: e.encMapUint64Intf(v) case *string: e.e.encodeString(c_UTF8, *v) case *bool: e.e.encodeBool(*v) case *int: e.e.encodeInt(int64(*v)) case *int8: e.e.encodeInt(int64(*v)) case *int16: e.e.encodeInt(int64(*v)) case *int32: e.e.encodeInt(int64(*v)) case *int64: e.e.encodeInt(*v) case *uint: e.e.encodeUint(uint64(*v)) case *uint8: e.e.encodeUint(uint64(*v)) case *uint16: e.e.encodeUint(uint64(*v)) case *uint32: e.e.encodeUint(uint64(*v)) case *uint64: e.e.encodeUint(*v) case *float32: e.e.encodeFloat32(*v) case *float64: e.e.encodeFloat64(*v) case *[]interface{}: e.encSliceIntf(*v) case *[]string: e.encSliceStr(*v) case *[]int64: e.encSliceInt64(*v) case *[]uint64: e.encSliceUint64(*v) case *[]uint8: e.e.encodeStringBytes(c_RAW, *v) case *map[interface{}]interface{}: e.encMapIntfIntf(*v) case *map[string]interface{}: e.encMapStrIntf(*v) case *map[string]string: e.encMapStrStr(*v) case *map[int64]interface{}: e.encMapInt64Intf(*v) case *map[uint64]interface{}: e.encMapUint64Intf(*v) default: e.encodeValue(reflect.ValueOf(iv)) } } func (e *Encoder) encodeValue(rv reflect.Value) { for rv.Kind() == reflect.Ptr { if rv.IsNil() { e.e.encodeNil() return } rv = rv.Elem() } rt := rv.Type() rtid := reflect.ValueOf(rt).Pointer() // if e.f == nil && e.s == nil { debugf("---->Creating new enc f map for type: %v\n", rt) } var fn encFn var ok bool if useMapForCodecCache { fn, ok = e.f[rtid] } else { for i, v := range e.x { if v == rtid { fn, ok = e.s[i], true break } } } if !ok { // debugf("\tCreating new enc fn for type: %v\n", rt) fi := encFnInfo{ti: getTypeInfo(rtid, rt), e: e, ee: e.e} fn.i = &fi if rtid == rawExtTypId { fn.f = (*encFnInfo).rawExt } else if e.e.isBuiltinType(rtid) { fn.f = (*encFnInfo).builtin } else if xfTag, xfFn := e.h.getEncodeExt(rtid); xfFn != nil { fi.xfTag, fi.xfFn = xfTag, xfFn fn.f = (*encFnInfo).ext } else if supportBinaryMarshal && fi.ti.m { fn.f = (*encFnInfo).binaryMarshal } else { switch rk := rt.Kind(); rk { case reflect.Bool: fn.f = (*encFnInfo).kBool case reflect.String: fn.f = (*encFnInfo).kString case reflect.Float64: fn.f = (*encFnInfo).kFloat64 case reflect.Float32: fn.f = (*encFnInfo).kFloat32 case reflect.Int, reflect.Int8, reflect.Int64, reflect.Int32, reflect.Int16: fn.f = (*encFnInfo).kInt case reflect.Uint8, reflect.Uint64, reflect.Uint, reflect.Uint32, reflect.Uint16: fn.f = (*encFnInfo).kUint case reflect.Invalid: fn.f = (*encFnInfo).kInvalid case reflect.Slice: fn.f = (*encFnInfo).kSlice case reflect.Array: fn.f = (*encFnInfo).kArray case reflect.Struct: fn.f = (*encFnInfo).kStruct // case reflect.Ptr: // fn.f = (*encFnInfo).kPtr case reflect.Interface: fn.f = (*encFnInfo).kInterface case reflect.Map: fn.f = (*encFnInfo).kMap default: fn.f = (*encFnInfo).kErr } } if useMapForCodecCache { if e.f == nil { e.f = make(map[uintptr]encFn, 16) } e.f[rtid] = fn } else { e.s = append(e.s, fn) e.x = append(e.x, rtid) } } fn.f(fn.i, rv) } func (e *Encoder) encRawExt(re RawExt) { if re.Data == nil { e.e.encodeNil() return } if e.hh.writeExt() { e.e.encodeExtPreamble(re.Tag, len(re.Data)) e.w.writeb(re.Data) } else { e.e.encodeStringBytes(c_RAW, re.Data) } } // --------------------------------------------- // short circuit functions for common maps and slices func (e *Encoder) encSliceIntf(v []interface{}) { e.e.encodeArrayPreamble(len(v)) for _, v2 := range v { e.encode(v2) } } func (e *Encoder) encSliceStr(v []string) { e.e.encodeArrayPreamble(len(v)) for _, v2 := range v { e.e.encodeString(c_UTF8, v2) } } func (e *Encoder) encSliceInt64(v []int64) { e.e.encodeArrayPreamble(len(v)) for _, v2 := range v { e.e.encodeInt(v2) } } func (e *Encoder) encSliceUint64(v []uint64) { e.e.encodeArrayPreamble(len(v)) for _, v2 := range v { e.e.encodeUint(v2) } } func (e *Encoder) encMapStrStr(v map[string]string) { e.e.encodeMapPreamble(len(v)) asSymbols := e.h.AsSymbols&AsSymbolMapStringKeysFlag != 0 for k2, v2 := range v { if asSymbols { e.e.encodeSymbol(k2) } else { e.e.encodeString(c_UTF8, k2) } e.e.encodeString(c_UTF8, v2) } } func (e *Encoder) encMapStrIntf(v map[string]interface{}) { e.e.encodeMapPreamble(len(v)) asSymbols := e.h.AsSymbols&AsSymbolMapStringKeysFlag != 0 for k2, v2 := range v { if asSymbols { e.e.encodeSymbol(k2) } else { e.e.encodeString(c_UTF8, k2) } e.encode(v2) } } func (e *Encoder) encMapInt64Intf(v map[int64]interface{}) { e.e.encodeMapPreamble(len(v)) for k2, v2 := range v { e.e.encodeInt(k2) e.encode(v2) } } func (e *Encoder) encMapUint64Intf(v map[uint64]interface{}) { e.e.encodeMapPreamble(len(v)) for k2, v2 := range v { e.e.encodeUint(uint64(k2)) e.encode(v2) } } func (e *Encoder) encMapIntfIntf(v map[interface{}]interface{}) { e.e.encodeMapPreamble(len(v)) for k2, v2 := range v { e.encode(k2) e.encode(v2) } } // ---------------------------------------- func encErr(format string, params ...interface{}) { doPanic(msgTagEnc, format, params...) } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/ext_dep_test.go000066400000000000000000000040331270406071200271430ustar00rootroot00000000000000// //+build ignore // Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec // This file includes benchmarks which have dependencies on 3rdparty // packages (bson and vmihailenco/msgpack) which must be installed locally. // // To run the benchmarks including these 3rdparty packages, first // - Uncomment first line in this file (put // // in front of it) // - Get those packages: // go get github.com/vmihailenco/msgpack // go get labix.org/v2/mgo/bson // - Run: // go test -bi -bench=. import ( "testing" vmsgpack "gopkg.in/vmihailenco/msgpack.v2" "labix.org/v2/mgo/bson" ) func init() { benchCheckers = append(benchCheckers, benchChecker{"v-msgpack", fnVMsgpackEncodeFn, fnVMsgpackDecodeFn}, benchChecker{"bson", fnBsonEncodeFn, fnBsonDecodeFn}, ) } func fnVMsgpackEncodeFn(ts interface{}) ([]byte, error) { return vmsgpack.Marshal(ts) } func fnVMsgpackDecodeFn(buf []byte, ts interface{}) error { return vmsgpack.Unmarshal(buf, ts) } func fnBsonEncodeFn(ts interface{}) ([]byte, error) { return bson.Marshal(ts) } func fnBsonDecodeFn(buf []byte, ts interface{}) error { return bson.Unmarshal(buf, ts) } func Benchmark__Bson_______Encode(b *testing.B) { fnBenchmarkEncode(b, "bson", benchTs, fnBsonEncodeFn) } func Benchmark__Bson_______Decode(b *testing.B) { fnBenchmarkDecode(b, "bson", benchTs, fnBsonEncodeFn, fnBsonDecodeFn, fnBenchNewTs) } func Benchmark__VMsgpack___Encode(b *testing.B) { fnBenchmarkEncode(b, "v-msgpack", benchTs, fnVMsgpackEncodeFn) } func Benchmark__VMsgpack___Decode(b *testing.B) { fnBenchmarkDecode(b, "v-msgpack", benchTs, fnVMsgpackEncodeFn, fnVMsgpackDecodeFn, fnBenchNewTs) } func TestMsgpackPythonGenStreams(t *testing.T) { doTestMsgpackPythonGenStreams(t) } func TestMsgpackRpcSpecGoClientToPythonSvc(t *testing.T) { doTestMsgpackRpcSpecGoClientToPythonSvc(t) } func TestMsgpackRpcSpecPythonClientToGoSvc(t *testing.T) { doTestMsgpackRpcSpecPythonClientToGoSvc(t) } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/helper.go000066400000000000000000000401401270406071200257320ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec // Contains code shared by both encode and decode. import ( "encoding/binary" "fmt" "math" "reflect" "sort" "strings" "sync" "time" "unicode" "unicode/utf8" ) const ( structTagName = "codec" // Support // encoding.BinaryMarshaler: MarshalBinary() (data []byte, err error) // encoding.BinaryUnmarshaler: UnmarshalBinary(data []byte) error // This constant flag will enable or disable it. supportBinaryMarshal = true // Each Encoder or Decoder uses a cache of functions based on conditionals, // so that the conditionals are not run every time. // // Either a map or a slice is used to keep track of the functions. // The map is more natural, but has a higher cost than a slice/array. // This flag (useMapForCodecCache) controls which is used. useMapForCodecCache = false // For some common container types, we can short-circuit an elaborate // reflection dance and call encode/decode directly. // The currently supported types are: // - slices of strings, or id's (int64,uint64) or interfaces. // - maps of str->str, str->intf, id(int64,uint64)->intf, intf->intf shortCircuitReflectToFastPath = true // for debugging, set this to false, to catch panic traces. // Note that this will always cause rpc tests to fail, since they need io.EOF sent via panic. recoverPanicToErr = true ) type charEncoding uint8 const ( c_RAW charEncoding = iota c_UTF8 c_UTF16LE c_UTF16BE c_UTF32LE c_UTF32BE ) // valueType is the stream type type valueType uint8 const ( valueTypeUnset valueType = iota valueTypeNil valueTypeInt valueTypeUint valueTypeFloat valueTypeBool valueTypeString valueTypeSymbol valueTypeBytes valueTypeMap valueTypeArray valueTypeTimestamp valueTypeExt valueTypeInvalid = 0xff ) var ( bigen = binary.BigEndian structInfoFieldName = "_struct" cachedTypeInfo = make(map[uintptr]*typeInfo, 4) cachedTypeInfoMutex sync.RWMutex intfSliceTyp = reflect.TypeOf([]interface{}(nil)) intfTyp = intfSliceTyp.Elem() strSliceTyp = reflect.TypeOf([]string(nil)) boolSliceTyp = reflect.TypeOf([]bool(nil)) uintSliceTyp = reflect.TypeOf([]uint(nil)) uint8SliceTyp = reflect.TypeOf([]uint8(nil)) uint16SliceTyp = reflect.TypeOf([]uint16(nil)) uint32SliceTyp = reflect.TypeOf([]uint32(nil)) uint64SliceTyp = reflect.TypeOf([]uint64(nil)) intSliceTyp = reflect.TypeOf([]int(nil)) int8SliceTyp = reflect.TypeOf([]int8(nil)) int16SliceTyp = reflect.TypeOf([]int16(nil)) int32SliceTyp = reflect.TypeOf([]int32(nil)) int64SliceTyp = reflect.TypeOf([]int64(nil)) float32SliceTyp = reflect.TypeOf([]float32(nil)) float64SliceTyp = reflect.TypeOf([]float64(nil)) mapIntfIntfTyp = reflect.TypeOf(map[interface{}]interface{}(nil)) mapStrIntfTyp = reflect.TypeOf(map[string]interface{}(nil)) mapStrStrTyp = reflect.TypeOf(map[string]string(nil)) mapIntIntfTyp = reflect.TypeOf(map[int]interface{}(nil)) mapInt64IntfTyp = reflect.TypeOf(map[int64]interface{}(nil)) mapUintIntfTyp = reflect.TypeOf(map[uint]interface{}(nil)) mapUint64IntfTyp = reflect.TypeOf(map[uint64]interface{}(nil)) stringTyp = reflect.TypeOf("") timeTyp = reflect.TypeOf(time.Time{}) rawExtTyp = reflect.TypeOf(RawExt{}) mapBySliceTyp = reflect.TypeOf((*MapBySlice)(nil)).Elem() binaryMarshalerTyp = reflect.TypeOf((*binaryMarshaler)(nil)).Elem() binaryUnmarshalerTyp = reflect.TypeOf((*binaryUnmarshaler)(nil)).Elem() rawExtTypId = reflect.ValueOf(rawExtTyp).Pointer() intfTypId = reflect.ValueOf(intfTyp).Pointer() timeTypId = reflect.ValueOf(timeTyp).Pointer() intfSliceTypId = reflect.ValueOf(intfSliceTyp).Pointer() strSliceTypId = reflect.ValueOf(strSliceTyp).Pointer() boolSliceTypId = reflect.ValueOf(boolSliceTyp).Pointer() uintSliceTypId = reflect.ValueOf(uintSliceTyp).Pointer() uint8SliceTypId = reflect.ValueOf(uint8SliceTyp).Pointer() uint16SliceTypId = reflect.ValueOf(uint16SliceTyp).Pointer() uint32SliceTypId = reflect.ValueOf(uint32SliceTyp).Pointer() uint64SliceTypId = reflect.ValueOf(uint64SliceTyp).Pointer() intSliceTypId = reflect.ValueOf(intSliceTyp).Pointer() int8SliceTypId = reflect.ValueOf(int8SliceTyp).Pointer() int16SliceTypId = reflect.ValueOf(int16SliceTyp).Pointer() int32SliceTypId = reflect.ValueOf(int32SliceTyp).Pointer() int64SliceTypId = reflect.ValueOf(int64SliceTyp).Pointer() float32SliceTypId = reflect.ValueOf(float32SliceTyp).Pointer() float64SliceTypId = reflect.ValueOf(float64SliceTyp).Pointer() mapStrStrTypId = reflect.ValueOf(mapStrStrTyp).Pointer() mapIntfIntfTypId = reflect.ValueOf(mapIntfIntfTyp).Pointer() mapStrIntfTypId = reflect.ValueOf(mapStrIntfTyp).Pointer() mapIntIntfTypId = reflect.ValueOf(mapIntIntfTyp).Pointer() mapInt64IntfTypId = reflect.ValueOf(mapInt64IntfTyp).Pointer() mapUintIntfTypId = reflect.ValueOf(mapUintIntfTyp).Pointer() mapUint64IntfTypId = reflect.ValueOf(mapUint64IntfTyp).Pointer() // Id = reflect.ValueOf().Pointer() // mapBySliceTypId = reflect.ValueOf(mapBySliceTyp).Pointer() binaryMarshalerTypId = reflect.ValueOf(binaryMarshalerTyp).Pointer() binaryUnmarshalerTypId = reflect.ValueOf(binaryUnmarshalerTyp).Pointer() intBitsize uint8 = uint8(reflect.TypeOf(int(0)).Bits()) uintBitsize uint8 = uint8(reflect.TypeOf(uint(0)).Bits()) bsAll0x00 = []byte{0, 0, 0, 0, 0, 0, 0, 0} bsAll0xff = []byte{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff} ) type binaryUnmarshaler interface { UnmarshalBinary(data []byte) error } type binaryMarshaler interface { MarshalBinary() (data []byte, err error) } // MapBySlice represents a slice which should be encoded as a map in the stream. // The slice contains a sequence of key-value pairs. type MapBySlice interface { MapBySlice() } // WARNING: DO NOT USE DIRECTLY. EXPORTED FOR GODOC BENEFIT. WILL BE REMOVED. // // BasicHandle encapsulates the common options and extension functions. type BasicHandle struct { extHandle EncodeOptions DecodeOptions } // Handle is the interface for a specific encoding format. // // Typically, a Handle is pre-configured before first time use, // and not modified while in use. Such a pre-configured Handle // is safe for concurrent access. type Handle interface { writeExt() bool getBasicHandle() *BasicHandle newEncDriver(w encWriter) encDriver newDecDriver(r decReader) decDriver } // RawExt represents raw unprocessed extension data. type RawExt struct { Tag byte Data []byte } type extTypeTagFn struct { rtid uintptr rt reflect.Type tag byte encFn func(reflect.Value) ([]byte, error) decFn func(reflect.Value, []byte) error } type extHandle []*extTypeTagFn // AddExt registers an encode and decode function for a reflect.Type. // Note that the type must be a named type, and specifically not // a pointer or Interface. An error is returned if that is not honored. // // To Deregister an ext, call AddExt with 0 tag, nil encfn and nil decfn. func (o *extHandle) AddExt( rt reflect.Type, tag byte, encfn func(reflect.Value) ([]byte, error), decfn func(reflect.Value, []byte) error, ) (err error) { // o is a pointer, because we may need to initialize it if rt.PkgPath() == "" || rt.Kind() == reflect.Interface { err = fmt.Errorf("codec.Handle.AddExt: Takes named type, especially not a pointer or interface: %T", reflect.Zero(rt).Interface()) return } // o cannot be nil, since it is always embedded in a Handle. // if nil, let it panic. // if o == nil { // err = errors.New("codec.Handle.AddExt: extHandle cannot be a nil pointer.") // return // } rtid := reflect.ValueOf(rt).Pointer() for _, v := range *o { if v.rtid == rtid { v.tag, v.encFn, v.decFn = tag, encfn, decfn return } } *o = append(*o, &extTypeTagFn{rtid, rt, tag, encfn, decfn}) return } func (o extHandle) getExt(rtid uintptr) *extTypeTagFn { for _, v := range o { if v.rtid == rtid { return v } } return nil } func (o extHandle) getExtForTag(tag byte) *extTypeTagFn { for _, v := range o { if v.tag == tag { return v } } return nil } func (o extHandle) getDecodeExtForTag(tag byte) ( rv reflect.Value, fn func(reflect.Value, []byte) error) { if x := o.getExtForTag(tag); x != nil { // ext is only registered for base rv = reflect.New(x.rt).Elem() fn = x.decFn } return } func (o extHandle) getDecodeExt(rtid uintptr) (tag byte, fn func(reflect.Value, []byte) error) { if x := o.getExt(rtid); x != nil { tag = x.tag fn = x.decFn } return } func (o extHandle) getEncodeExt(rtid uintptr) (tag byte, fn func(reflect.Value) ([]byte, error)) { if x := o.getExt(rtid); x != nil { tag = x.tag fn = x.encFn } return } type structFieldInfo struct { encName string // encode name // only one of 'i' or 'is' can be set. If 'i' is -1, then 'is' has been set. is []int // (recursive/embedded) field index in struct i int16 // field index in struct omitEmpty bool toArray bool // if field is _struct, is the toArray set? // tag string // tag // name string // field name // encNameBs []byte // encoded name as byte stream // ikind int // kind of the field as an int i.e. int(reflect.Kind) } func parseStructFieldInfo(fname string, stag string) *structFieldInfo { if fname == "" { panic("parseStructFieldInfo: No Field Name") } si := structFieldInfo{ // name: fname, encName: fname, // tag: stag, } if stag != "" { for i, s := range strings.Split(stag, ",") { if i == 0 { if s != "" { si.encName = s } } else { switch s { case "omitempty": si.omitEmpty = true case "toarray": si.toArray = true } } } } // si.encNameBs = []byte(si.encName) return &si } type sfiSortedByEncName []*structFieldInfo func (p sfiSortedByEncName) Len() int { return len(p) } func (p sfiSortedByEncName) Less(i, j int) bool { return p[i].encName < p[j].encName } func (p sfiSortedByEncName) Swap(i, j int) { p[i], p[j] = p[j], p[i] } // typeInfo keeps information about each type referenced in the encode/decode sequence. // // During an encode/decode sequence, we work as below: // - If base is a built in type, en/decode base value // - If base is registered as an extension, en/decode base value // - If type is binary(M/Unm)arshaler, call Binary(M/Unm)arshal method // - Else decode appropriately based on the reflect.Kind type typeInfo struct { sfi []*structFieldInfo // sorted. Used when enc/dec struct to map. sfip []*structFieldInfo // unsorted. Used when enc/dec struct to array. rt reflect.Type rtid uintptr // baseId gives pointer to the base reflect.Type, after deferencing // the pointers. E.g. base type of ***time.Time is time.Time. base reflect.Type baseId uintptr baseIndir int8 // number of indirections to get to base mbs bool // base type (T or *T) is a MapBySlice m bool // base type (T or *T) is a binaryMarshaler unm bool // base type (T or *T) is a binaryUnmarshaler mIndir int8 // number of indirections to get to binaryMarshaler type unmIndir int8 // number of indirections to get to binaryUnmarshaler type toArray bool // whether this (struct) type should be encoded as an array } func (ti *typeInfo) indexForEncName(name string) int { //tisfi := ti.sfi const binarySearchThreshold = 16 if sfilen := len(ti.sfi); sfilen < binarySearchThreshold { // linear search. faster than binary search in my testing up to 16-field structs. for i, si := range ti.sfi { if si.encName == name { return i } } } else { // binary search. adapted from sort/search.go. h, i, j := 0, 0, sfilen for i < j { h = i + (j-i)/2 if ti.sfi[h].encName < name { i = h + 1 } else { j = h } } if i < sfilen && ti.sfi[i].encName == name { return i } } return -1 } func getTypeInfo(rtid uintptr, rt reflect.Type) (pti *typeInfo) { var ok bool cachedTypeInfoMutex.RLock() pti, ok = cachedTypeInfo[rtid] cachedTypeInfoMutex.RUnlock() if ok { return } cachedTypeInfoMutex.Lock() defer cachedTypeInfoMutex.Unlock() if pti, ok = cachedTypeInfo[rtid]; ok { return } ti := typeInfo{rt: rt, rtid: rtid} pti = &ti var indir int8 if ok, indir = implementsIntf(rt, binaryMarshalerTyp); ok { ti.m, ti.mIndir = true, indir } if ok, indir = implementsIntf(rt, binaryUnmarshalerTyp); ok { ti.unm, ti.unmIndir = true, indir } if ok, _ = implementsIntf(rt, mapBySliceTyp); ok { ti.mbs = true } pt := rt var ptIndir int8 // for ; pt.Kind() == reflect.Ptr; pt, ptIndir = pt.Elem(), ptIndir+1 { } for pt.Kind() == reflect.Ptr { pt = pt.Elem() ptIndir++ } if ptIndir == 0 { ti.base = rt ti.baseId = rtid } else { ti.base = pt ti.baseId = reflect.ValueOf(pt).Pointer() ti.baseIndir = ptIndir } if rt.Kind() == reflect.Struct { var siInfo *structFieldInfo if f, ok := rt.FieldByName(structInfoFieldName); ok { siInfo = parseStructFieldInfo(structInfoFieldName, f.Tag.Get(structTagName)) ti.toArray = siInfo.toArray } sfip := make([]*structFieldInfo, 0, rt.NumField()) rgetTypeInfo(rt, nil, make(map[string]bool), &sfip, siInfo) // // try to put all si close together // const tryToPutAllStructFieldInfoTogether = true // if tryToPutAllStructFieldInfoTogether { // sfip2 := make([]structFieldInfo, len(sfip)) // for i, si := range sfip { // sfip2[i] = *si // } // for i := range sfip { // sfip[i] = &sfip2[i] // } // } ti.sfip = make([]*structFieldInfo, len(sfip)) ti.sfi = make([]*structFieldInfo, len(sfip)) copy(ti.sfip, sfip) sort.Sort(sfiSortedByEncName(sfip)) copy(ti.sfi, sfip) } // sfi = sfip cachedTypeInfo[rtid] = pti return } func rgetTypeInfo(rt reflect.Type, indexstack []int, fnameToHastag map[string]bool, sfi *[]*structFieldInfo, siInfo *structFieldInfo, ) { // for rt.Kind() == reflect.Ptr { // // indexstack = append(indexstack, 0) // rt = rt.Elem() // } for j := 0; j < rt.NumField(); j++ { f := rt.Field(j) stag := f.Tag.Get(structTagName) if stag == "-" { continue } if r1, _ := utf8.DecodeRuneInString(f.Name); r1 == utf8.RuneError || !unicode.IsUpper(r1) { continue } // if anonymous and there is no struct tag and its a struct (or pointer to struct), inline it. if f.Anonymous && stag == "" { ft := f.Type for ft.Kind() == reflect.Ptr { ft = ft.Elem() } if ft.Kind() == reflect.Struct { indexstack2 := append(append(make([]int, 0, len(indexstack)+4), indexstack...), j) rgetTypeInfo(ft, indexstack2, fnameToHastag, sfi, siInfo) continue } } // do not let fields with same name in embedded structs override field at higher level. // this must be done after anonymous check, to allow anonymous field // still include their child fields if _, ok := fnameToHastag[f.Name]; ok { continue } si := parseStructFieldInfo(f.Name, stag) // si.ikind = int(f.Type.Kind()) if len(indexstack) == 0 { si.i = int16(j) } else { si.i = -1 si.is = append(append(make([]int, 0, len(indexstack)+4), indexstack...), j) } if siInfo != nil { if siInfo.omitEmpty { si.omitEmpty = true } } *sfi = append(*sfi, si) fnameToHastag[f.Name] = stag != "" } } func panicToErr(err *error) { if recoverPanicToErr { if x := recover(); x != nil { //debug.PrintStack() panicValToErr(x, err) } } } func doPanic(tag string, format string, params ...interface{}) { params2 := make([]interface{}, len(params)+1) params2[0] = tag copy(params2[1:], params) panic(fmt.Errorf("%s: "+format, params2...)) } func checkOverflowFloat32(f float64, doCheck bool) { if !doCheck { return } // check overflow (logic adapted from std pkg reflect/value.go OverflowFloat() f2 := f if f2 < 0 { f2 = -f } if math.MaxFloat32 < f2 && f2 <= math.MaxFloat64 { decErr("Overflow float32 value: %v", f2) } } func checkOverflow(ui uint64, i int64, bitsize uint8) { // check overflow (logic adapted from std pkg reflect/value.go OverflowUint() if bitsize == 0 { return } if i != 0 { if trunc := (i << (64 - bitsize)) >> (64 - bitsize); i != trunc { decErr("Overflow int value: %v", i) } } if ui != 0 { if trunc := (ui << (64 - bitsize)) >> (64 - bitsize); ui != trunc { decErr("Overflow uint value: %v", ui) } } } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/helper_internal.go000066400000000000000000000057571270406071200276450ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec // All non-std package dependencies live in this file, // so porting to different environment is easy (just update functions). import ( "errors" "fmt" "math" "reflect" ) var ( raisePanicAfterRecover = false debugging = true ) func panicValToErr(panicVal interface{}, err *error) { switch xerr := panicVal.(type) { case error: *err = xerr case string: *err = errors.New(xerr) default: *err = fmt.Errorf("%v", panicVal) } if raisePanicAfterRecover { panic(panicVal) } return } func isEmptyValueDeref(v reflect.Value, deref bool) bool { switch v.Kind() { case reflect.Array, reflect.Map, reflect.Slice, reflect.String: return v.Len() == 0 case reflect.Bool: return !v.Bool() case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: return v.Int() == 0 case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr: return v.Uint() == 0 case reflect.Float32, reflect.Float64: return v.Float() == 0 case reflect.Interface, reflect.Ptr: if deref { if v.IsNil() { return true } return isEmptyValueDeref(v.Elem(), deref) } else { return v.IsNil() } case reflect.Struct: // return true if all fields are empty. else return false. // we cannot use equality check, because some fields may be maps/slices/etc // and consequently the structs are not comparable. // return v.Interface() == reflect.Zero(v.Type()).Interface() for i, n := 0, v.NumField(); i < n; i++ { if !isEmptyValueDeref(v.Field(i), deref) { return false } } return true } return false } func isEmptyValue(v reflect.Value) bool { return isEmptyValueDeref(v, true) } func debugf(format string, args ...interface{}) { if debugging { if len(format) == 0 || format[len(format)-1] != '\n' { format = format + "\n" } fmt.Printf(format, args...) } } func pruneSignExt(v []byte, pos bool) (n int) { if len(v) < 2 { } else if pos && v[0] == 0 { for ; v[n] == 0 && n+1 < len(v) && (v[n+1]&(1<<7) == 0); n++ { } } else if !pos && v[0] == 0xff { for ; v[n] == 0xff && n+1 < len(v) && (v[n+1]&(1<<7) != 0); n++ { } } return } func implementsIntf(typ, iTyp reflect.Type) (success bool, indir int8) { if typ == nil { return } rt := typ // The type might be a pointer and we need to keep // dereferencing to the base type until we find an implementation. for { if rt.Implements(iTyp) { return true, indir } if p := rt; p.Kind() == reflect.Ptr { indir++ if indir >= math.MaxInt8 { // insane number of indirections return false, 0 } rt = p.Elem() continue } break } // No luck yet, but if this is a base type (non-pointer), the pointer might satisfy. if typ.Kind() != reflect.Ptr { // Not a pointer, but does the pointer work? if reflect.PtrTo(typ).Implements(iTyp) { return true, -1 } } return false, 0 } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/msgpack.go000066400000000000000000000517111270406071200261060ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. /* MSGPACK Msgpack-c implementation powers the c, c++, python, ruby, etc libraries. We need to maintain compatibility with it and how it encodes integer values without caring about the type. For compatibility with behaviour of msgpack-c reference implementation: - Go intX (>0) and uintX IS ENCODED AS msgpack +ve fixnum, unsigned - Go intX (<0) IS ENCODED AS msgpack -ve fixnum, signed */ package codec import ( "fmt" "io" "math" "net/rpc" ) const ( mpPosFixNumMin byte = 0x00 mpPosFixNumMax = 0x7f mpFixMapMin = 0x80 mpFixMapMax = 0x8f mpFixArrayMin = 0x90 mpFixArrayMax = 0x9f mpFixStrMin = 0xa0 mpFixStrMax = 0xbf mpNil = 0xc0 _ = 0xc1 mpFalse = 0xc2 mpTrue = 0xc3 mpFloat = 0xca mpDouble = 0xcb mpUint8 = 0xcc mpUint16 = 0xcd mpUint32 = 0xce mpUint64 = 0xcf mpInt8 = 0xd0 mpInt16 = 0xd1 mpInt32 = 0xd2 mpInt64 = 0xd3 // extensions below mpBin8 = 0xc4 mpBin16 = 0xc5 mpBin32 = 0xc6 mpExt8 = 0xc7 mpExt16 = 0xc8 mpExt32 = 0xc9 mpFixExt1 = 0xd4 mpFixExt2 = 0xd5 mpFixExt4 = 0xd6 mpFixExt8 = 0xd7 mpFixExt16 = 0xd8 mpStr8 = 0xd9 // new mpStr16 = 0xda mpStr32 = 0xdb mpArray16 = 0xdc mpArray32 = 0xdd mpMap16 = 0xde mpMap32 = 0xdf mpNegFixNumMin = 0xe0 mpNegFixNumMax = 0xff ) // MsgpackSpecRpcMultiArgs is a special type which signifies to the MsgpackSpecRpcCodec // that the backend RPC service takes multiple arguments, which have been arranged // in sequence in the slice. // // The Codec then passes it AS-IS to the rpc service (without wrapping it in an // array of 1 element). type MsgpackSpecRpcMultiArgs []interface{} // A MsgpackContainer type specifies the different types of msgpackContainers. type msgpackContainerType struct { fixCutoff int bFixMin, b8, b16, b32 byte hasFixMin, has8, has8Always bool } var ( msgpackContainerStr = msgpackContainerType{32, mpFixStrMin, mpStr8, mpStr16, mpStr32, true, true, false} msgpackContainerBin = msgpackContainerType{0, 0, mpBin8, mpBin16, mpBin32, false, true, true} msgpackContainerList = msgpackContainerType{16, mpFixArrayMin, 0, mpArray16, mpArray32, true, false, false} msgpackContainerMap = msgpackContainerType{16, mpFixMapMin, 0, mpMap16, mpMap32, true, false, false} ) //--------------------------------------------- type msgpackEncDriver struct { w encWriter h *MsgpackHandle } func (e *msgpackEncDriver) isBuiltinType(rt uintptr) bool { //no builtin types. All encodings are based on kinds. Types supported as extensions. return false } func (e *msgpackEncDriver) encodeBuiltin(rt uintptr, v interface{}) {} func (e *msgpackEncDriver) encodeNil() { e.w.writen1(mpNil) } func (e *msgpackEncDriver) encodeInt(i int64) { switch { case i >= 0: e.encodeUint(uint64(i)) case i >= -32: e.w.writen1(byte(i)) case i >= math.MinInt8: e.w.writen2(mpInt8, byte(i)) case i >= math.MinInt16: e.w.writen1(mpInt16) e.w.writeUint16(uint16(i)) case i >= math.MinInt32: e.w.writen1(mpInt32) e.w.writeUint32(uint32(i)) default: e.w.writen1(mpInt64) e.w.writeUint64(uint64(i)) } } func (e *msgpackEncDriver) encodeUint(i uint64) { switch { case i <= math.MaxInt8: e.w.writen1(byte(i)) case i <= math.MaxUint8: e.w.writen2(mpUint8, byte(i)) case i <= math.MaxUint16: e.w.writen1(mpUint16) e.w.writeUint16(uint16(i)) case i <= math.MaxUint32: e.w.writen1(mpUint32) e.w.writeUint32(uint32(i)) default: e.w.writen1(mpUint64) e.w.writeUint64(uint64(i)) } } func (e *msgpackEncDriver) encodeBool(b bool) { if b { e.w.writen1(mpTrue) } else { e.w.writen1(mpFalse) } } func (e *msgpackEncDriver) encodeFloat32(f float32) { e.w.writen1(mpFloat) e.w.writeUint32(math.Float32bits(f)) } func (e *msgpackEncDriver) encodeFloat64(f float64) { e.w.writen1(mpDouble) e.w.writeUint64(math.Float64bits(f)) } func (e *msgpackEncDriver) encodeExtPreamble(xtag byte, l int) { switch { case l == 1: e.w.writen2(mpFixExt1, xtag) case l == 2: e.w.writen2(mpFixExt2, xtag) case l == 4: e.w.writen2(mpFixExt4, xtag) case l == 8: e.w.writen2(mpFixExt8, xtag) case l == 16: e.w.writen2(mpFixExt16, xtag) case l < 256: e.w.writen2(mpExt8, byte(l)) e.w.writen1(xtag) case l < 65536: e.w.writen1(mpExt16) e.w.writeUint16(uint16(l)) e.w.writen1(xtag) default: e.w.writen1(mpExt32) e.w.writeUint32(uint32(l)) e.w.writen1(xtag) } } func (e *msgpackEncDriver) encodeArrayPreamble(length int) { e.writeContainerLen(msgpackContainerList, length) } func (e *msgpackEncDriver) encodeMapPreamble(length int) { e.writeContainerLen(msgpackContainerMap, length) } func (e *msgpackEncDriver) encodeString(c charEncoding, s string) { if c == c_RAW && e.h.WriteExt { e.writeContainerLen(msgpackContainerBin, len(s)) } else { e.writeContainerLen(msgpackContainerStr, len(s)) } if len(s) > 0 { e.w.writestr(s) } } func (e *msgpackEncDriver) encodeSymbol(v string) { e.encodeString(c_UTF8, v) } func (e *msgpackEncDriver) encodeStringBytes(c charEncoding, bs []byte) { if c == c_RAW && e.h.WriteExt { e.writeContainerLen(msgpackContainerBin, len(bs)) } else { e.writeContainerLen(msgpackContainerStr, len(bs)) } if len(bs) > 0 { e.w.writeb(bs) } } func (e *msgpackEncDriver) writeContainerLen(ct msgpackContainerType, l int) { switch { case ct.hasFixMin && l < ct.fixCutoff: e.w.writen1(ct.bFixMin | byte(l)) case ct.has8 && l < 256 && (ct.has8Always || e.h.WriteExt): e.w.writen2(ct.b8, uint8(l)) case l < 65536: e.w.writen1(ct.b16) e.w.writeUint16(uint16(l)) default: e.w.writen1(ct.b32) e.w.writeUint32(uint32(l)) } } //--------------------------------------------- type msgpackDecDriver struct { r decReader h *MsgpackHandle bd byte bdRead bool bdType valueType } func (d *msgpackDecDriver) isBuiltinType(rt uintptr) bool { //no builtin types. All encodings are based on kinds. Types supported as extensions. return false } func (d *msgpackDecDriver) decodeBuiltin(rt uintptr, v interface{}) {} // Note: This returns either a primitive (int, bool, etc) for non-containers, // or a containerType, or a specific type denoting nil or extension. // It is called when a nil interface{} is passed, leaving it up to the DecDriver // to introspect the stream and decide how best to decode. // It deciphers the value by looking at the stream first. func (d *msgpackDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurther bool) { d.initReadNext() bd := d.bd switch bd { case mpNil: vt = valueTypeNil d.bdRead = false case mpFalse: vt = valueTypeBool v = false case mpTrue: vt = valueTypeBool v = true case mpFloat: vt = valueTypeFloat v = float64(math.Float32frombits(d.r.readUint32())) case mpDouble: vt = valueTypeFloat v = math.Float64frombits(d.r.readUint64()) case mpUint8: vt = valueTypeUint v = uint64(d.r.readn1()) case mpUint16: vt = valueTypeUint v = uint64(d.r.readUint16()) case mpUint32: vt = valueTypeUint v = uint64(d.r.readUint32()) case mpUint64: vt = valueTypeUint v = uint64(d.r.readUint64()) case mpInt8: vt = valueTypeInt v = int64(int8(d.r.readn1())) case mpInt16: vt = valueTypeInt v = int64(int16(d.r.readUint16())) case mpInt32: vt = valueTypeInt v = int64(int32(d.r.readUint32())) case mpInt64: vt = valueTypeInt v = int64(int64(d.r.readUint64())) default: switch { case bd >= mpPosFixNumMin && bd <= mpPosFixNumMax: // positive fixnum (always signed) vt = valueTypeInt v = int64(int8(bd)) case bd >= mpNegFixNumMin && bd <= mpNegFixNumMax: // negative fixnum vt = valueTypeInt v = int64(int8(bd)) case bd == mpStr8, bd == mpStr16, bd == mpStr32, bd >= mpFixStrMin && bd <= mpFixStrMax: if d.h.RawToString { var rvm string vt = valueTypeString v = &rvm } else { var rvm = []byte{} vt = valueTypeBytes v = &rvm } decodeFurther = true case bd == mpBin8, bd == mpBin16, bd == mpBin32: var rvm = []byte{} vt = valueTypeBytes v = &rvm decodeFurther = true case bd == mpArray16, bd == mpArray32, bd >= mpFixArrayMin && bd <= mpFixArrayMax: vt = valueTypeArray decodeFurther = true case bd == mpMap16, bd == mpMap32, bd >= mpFixMapMin && bd <= mpFixMapMax: vt = valueTypeMap decodeFurther = true case bd >= mpFixExt1 && bd <= mpFixExt16, bd >= mpExt8 && bd <= mpExt32: clen := d.readExtLen() var re RawExt re.Tag = d.r.readn1() re.Data = d.r.readn(clen) v = &re vt = valueTypeExt default: decErr("Nil-Deciphered DecodeValue: %s: hex: %x, dec: %d", msgBadDesc, bd, bd) } } if !decodeFurther { d.bdRead = false } return } // int can be decoded from msgpack type: intXXX or uintXXX func (d *msgpackDecDriver) decodeInt(bitsize uint8) (i int64) { switch d.bd { case mpUint8: i = int64(uint64(d.r.readn1())) case mpUint16: i = int64(uint64(d.r.readUint16())) case mpUint32: i = int64(uint64(d.r.readUint32())) case mpUint64: i = int64(d.r.readUint64()) case mpInt8: i = int64(int8(d.r.readn1())) case mpInt16: i = int64(int16(d.r.readUint16())) case mpInt32: i = int64(int32(d.r.readUint32())) case mpInt64: i = int64(d.r.readUint64()) default: switch { case d.bd >= mpPosFixNumMin && d.bd <= mpPosFixNumMax: i = int64(int8(d.bd)) case d.bd >= mpNegFixNumMin && d.bd <= mpNegFixNumMax: i = int64(int8(d.bd)) default: decErr("Unhandled single-byte unsigned integer value: %s: %x", msgBadDesc, d.bd) } } // check overflow (logic adapted from std pkg reflect/value.go OverflowUint() if bitsize > 0 { if trunc := (i << (64 - bitsize)) >> (64 - bitsize); i != trunc { decErr("Overflow int value: %v", i) } } d.bdRead = false return } // uint can be decoded from msgpack type: intXXX or uintXXX func (d *msgpackDecDriver) decodeUint(bitsize uint8) (ui uint64) { switch d.bd { case mpUint8: ui = uint64(d.r.readn1()) case mpUint16: ui = uint64(d.r.readUint16()) case mpUint32: ui = uint64(d.r.readUint32()) case mpUint64: ui = d.r.readUint64() case mpInt8: if i := int64(int8(d.r.readn1())); i >= 0 { ui = uint64(i) } else { decErr("Assigning negative signed value: %v, to unsigned type", i) } case mpInt16: if i := int64(int16(d.r.readUint16())); i >= 0 { ui = uint64(i) } else { decErr("Assigning negative signed value: %v, to unsigned type", i) } case mpInt32: if i := int64(int32(d.r.readUint32())); i >= 0 { ui = uint64(i) } else { decErr("Assigning negative signed value: %v, to unsigned type", i) } case mpInt64: if i := int64(d.r.readUint64()); i >= 0 { ui = uint64(i) } else { decErr("Assigning negative signed value: %v, to unsigned type", i) } default: switch { case d.bd >= mpPosFixNumMin && d.bd <= mpPosFixNumMax: ui = uint64(d.bd) case d.bd >= mpNegFixNumMin && d.bd <= mpNegFixNumMax: decErr("Assigning negative signed value: %v, to unsigned type", int(d.bd)) default: decErr("Unhandled single-byte unsigned integer value: %s: %x", msgBadDesc, d.bd) } } // check overflow (logic adapted from std pkg reflect/value.go OverflowUint() if bitsize > 0 { if trunc := (ui << (64 - bitsize)) >> (64 - bitsize); ui != trunc { decErr("Overflow uint value: %v", ui) } } d.bdRead = false return } // float can either be decoded from msgpack type: float, double or intX func (d *msgpackDecDriver) decodeFloat(chkOverflow32 bool) (f float64) { switch d.bd { case mpFloat: f = float64(math.Float32frombits(d.r.readUint32())) case mpDouble: f = math.Float64frombits(d.r.readUint64()) default: f = float64(d.decodeInt(0)) } checkOverflowFloat32(f, chkOverflow32) d.bdRead = false return } // bool can be decoded from bool, fixnum 0 or 1. func (d *msgpackDecDriver) decodeBool() (b bool) { switch d.bd { case mpFalse, 0: // b = false case mpTrue, 1: b = true default: decErr("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd) } d.bdRead = false return } func (d *msgpackDecDriver) decodeString() (s string) { clen := d.readContainerLen(msgpackContainerStr) if clen > 0 { s = string(d.r.readn(clen)) } d.bdRead = false return } // Callers must check if changed=true (to decide whether to replace the one they have) func (d *msgpackDecDriver) decodeBytes(bs []byte) (bsOut []byte, changed bool) { // bytes can be decoded from msgpackContainerStr or msgpackContainerBin var clen int switch d.bd { case mpBin8, mpBin16, mpBin32: clen = d.readContainerLen(msgpackContainerBin) default: clen = d.readContainerLen(msgpackContainerStr) } // if clen < 0 { // changed = true // panic("length cannot be zero. this cannot be nil.") // } if clen > 0 { // if no contents in stream, don't update the passed byteslice if len(bs) != clen { // Return changed=true if length of passed slice diff from length of bytes in stream if len(bs) > clen { bs = bs[:clen] } else { bs = make([]byte, clen) } bsOut = bs changed = true } d.r.readb(bs) } d.bdRead = false return } // Every top-level decode funcs (i.e. decodeValue, decode) must call this first. func (d *msgpackDecDriver) initReadNext() { if d.bdRead { return } d.bd = d.r.readn1() d.bdRead = true d.bdType = valueTypeUnset } func (d *msgpackDecDriver) currentEncodedType() valueType { if d.bdType == valueTypeUnset { bd := d.bd switch bd { case mpNil: d.bdType = valueTypeNil case mpFalse, mpTrue: d.bdType = valueTypeBool case mpFloat, mpDouble: d.bdType = valueTypeFloat case mpUint8, mpUint16, mpUint32, mpUint64: d.bdType = valueTypeUint case mpInt8, mpInt16, mpInt32, mpInt64: d.bdType = valueTypeInt default: switch { case bd >= mpPosFixNumMin && bd <= mpPosFixNumMax: d.bdType = valueTypeInt case bd >= mpNegFixNumMin && bd <= mpNegFixNumMax: d.bdType = valueTypeInt case bd == mpStr8, bd == mpStr16, bd == mpStr32, bd >= mpFixStrMin && bd <= mpFixStrMax: if d.h.RawToString { d.bdType = valueTypeString } else { d.bdType = valueTypeBytes } case bd == mpBin8, bd == mpBin16, bd == mpBin32: d.bdType = valueTypeBytes case bd == mpArray16, bd == mpArray32, bd >= mpFixArrayMin && bd <= mpFixArrayMax: d.bdType = valueTypeArray case bd == mpMap16, bd == mpMap32, bd >= mpFixMapMin && bd <= mpFixMapMax: d.bdType = valueTypeMap case bd >= mpFixExt1 && bd <= mpFixExt16, bd >= mpExt8 && bd <= mpExt32: d.bdType = valueTypeExt default: decErr("currentEncodedType: Undeciphered descriptor: %s: hex: %x, dec: %d", msgBadDesc, bd, bd) } } } return d.bdType } func (d *msgpackDecDriver) tryDecodeAsNil() bool { if d.bd == mpNil { d.bdRead = false return true } return false } func (d *msgpackDecDriver) readContainerLen(ct msgpackContainerType) (clen int) { bd := d.bd switch { case bd == mpNil: clen = -1 // to represent nil case bd == ct.b8: clen = int(d.r.readn1()) case bd == ct.b16: clen = int(d.r.readUint16()) case bd == ct.b32: clen = int(d.r.readUint32()) case (ct.bFixMin & bd) == ct.bFixMin: clen = int(ct.bFixMin ^ bd) default: decErr("readContainerLen: %s: hex: %x, dec: %d", msgBadDesc, bd, bd) } d.bdRead = false return } func (d *msgpackDecDriver) readMapLen() int { return d.readContainerLen(msgpackContainerMap) } func (d *msgpackDecDriver) readArrayLen() int { return d.readContainerLen(msgpackContainerList) } func (d *msgpackDecDriver) readExtLen() (clen int) { switch d.bd { case mpNil: clen = -1 // to represent nil case mpFixExt1: clen = 1 case mpFixExt2: clen = 2 case mpFixExt4: clen = 4 case mpFixExt8: clen = 8 case mpFixExt16: clen = 16 case mpExt8: clen = int(d.r.readn1()) case mpExt16: clen = int(d.r.readUint16()) case mpExt32: clen = int(d.r.readUint32()) default: decErr("decoding ext bytes: found unexpected byte: %x", d.bd) } return } func (d *msgpackDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte) { xbd := d.bd switch { case xbd == mpBin8, xbd == mpBin16, xbd == mpBin32: xbs, _ = d.decodeBytes(nil) case xbd == mpStr8, xbd == mpStr16, xbd == mpStr32, xbd >= mpFixStrMin && xbd <= mpFixStrMax: xbs = []byte(d.decodeString()) default: clen := d.readExtLen() xtag = d.r.readn1() if verifyTag && xtag != tag { decErr("Wrong extension tag. Got %b. Expecting: %v", xtag, tag) } xbs = d.r.readn(clen) } d.bdRead = false return } //-------------------------------------------------- //MsgpackHandle is a Handle for the Msgpack Schema-Free Encoding Format. type MsgpackHandle struct { BasicHandle // RawToString controls how raw bytes are decoded into a nil interface{}. RawToString bool // WriteExt flag supports encoding configured extensions with extension tags. // It also controls whether other elements of the new spec are encoded (ie Str8). // // With WriteExt=false, configured extensions are serialized as raw bytes // and Str8 is not encoded. // // A stream can still be decoded into a typed value, provided an appropriate value // is provided, but the type cannot be inferred from the stream. If no appropriate // type is provided (e.g. decoding into a nil interface{}), you get back // a []byte or string based on the setting of RawToString. WriteExt bool } func (h *MsgpackHandle) newEncDriver(w encWriter) encDriver { return &msgpackEncDriver{w: w, h: h} } func (h *MsgpackHandle) newDecDriver(r decReader) decDriver { return &msgpackDecDriver{r: r, h: h} } func (h *MsgpackHandle) writeExt() bool { return h.WriteExt } func (h *MsgpackHandle) getBasicHandle() *BasicHandle { return &h.BasicHandle } //-------------------------------------------------- type msgpackSpecRpcCodec struct { rpcCodec } // /////////////// Spec RPC Codec /////////////////// func (c *msgpackSpecRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error { // WriteRequest can write to both a Go service, and other services that do // not abide by the 1 argument rule of a Go service. // We discriminate based on if the body is a MsgpackSpecRpcMultiArgs var bodyArr []interface{} if m, ok := body.(MsgpackSpecRpcMultiArgs); ok { bodyArr = ([]interface{})(m) } else { bodyArr = []interface{}{body} } r2 := []interface{}{0, uint32(r.Seq), r.ServiceMethod, bodyArr} return c.write(r2, nil, false, true) } func (c *msgpackSpecRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error { var moe interface{} if r.Error != "" { moe = r.Error } if moe != nil && body != nil { body = nil } r2 := []interface{}{1, uint32(r.Seq), moe, body} return c.write(r2, nil, false, true) } func (c *msgpackSpecRpcCodec) ReadResponseHeader(r *rpc.Response) error { return c.parseCustomHeader(1, &r.Seq, &r.Error) } func (c *msgpackSpecRpcCodec) ReadRequestHeader(r *rpc.Request) error { return c.parseCustomHeader(0, &r.Seq, &r.ServiceMethod) } func (c *msgpackSpecRpcCodec) ReadRequestBody(body interface{}) error { if body == nil { // read and discard return c.read(nil) } bodyArr := []interface{}{body} return c.read(&bodyArr) } func (c *msgpackSpecRpcCodec) parseCustomHeader(expectTypeByte byte, msgid *uint64, methodOrError *string) (err error) { if c.cls { return io.EOF } // We read the response header by hand // so that the body can be decoded on its own from the stream at a later time. const fia byte = 0x94 //four item array descriptor value // Not sure why the panic of EOF is swallowed above. // if bs1 := c.dec.r.readn1(); bs1 != fia { // err = fmt.Errorf("Unexpected value for array descriptor: Expecting %v. Received %v", fia, bs1) // return // } var b byte b, err = c.br.ReadByte() if err != nil { return } if b != fia { err = fmt.Errorf("Unexpected value for array descriptor: Expecting %v. Received %v", fia, b) return } if err = c.read(&b); err != nil { return } if b != expectTypeByte { err = fmt.Errorf("Unexpected byte descriptor in header. Expecting %v. Received %v", expectTypeByte, b) return } if err = c.read(msgid); err != nil { return } if err = c.read(methodOrError); err != nil { return } return } //-------------------------------------------------- // msgpackSpecRpc is the implementation of Rpc that uses custom communication protocol // as defined in the msgpack spec at https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md type msgpackSpecRpc struct{} // MsgpackSpecRpc implements Rpc using the communication protocol defined in // the msgpack spec at https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md . // Its methods (ServerCodec and ClientCodec) return values that implement RpcCodecBuffered. var MsgpackSpecRpc msgpackSpecRpc func (x msgpackSpecRpc) ServerCodec(conn io.ReadWriteCloser, h Handle) rpc.ServerCodec { return &msgpackSpecRpcCodec{newRPCCodec(conn, h)} } func (x msgpackSpecRpc) ClientCodec(conn io.ReadWriteCloser, h Handle) rpc.ClientCodec { return &msgpackSpecRpcCodec{newRPCCodec(conn, h)} } var _ decDriver = (*msgpackDecDriver)(nil) var _ encDriver = (*msgpackEncDriver)(nil) golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/msgpack_test.py000077500000000000000000000066261270406071200272000ustar00rootroot00000000000000#!/usr/bin/env python # This will create golden files in a directory passed to it. # A Test calls this internally to create the golden files # So it can process them (so we don't have to checkin the files). import msgpack, msgpackrpc, sys, os, threading def get_test_data_list(): # get list with all primitive types, and a combo type l0 = [ -8, -1616, -32323232, -6464646464646464, 192, 1616, 32323232, 6464646464646464, 192, -3232.0, -6464646464.0, 3232.0, 6464646464.0, False, True, None, "someday", "", "bytestring", 1328176922000002000, -2206187877999998000, 0, -6795364578871345152 ] l1 = [ { "true": True, "false": False }, { "true": "True", "false": False, "uint16(1616)": 1616 }, { "list": [1616, 32323232, True, -3232.0, {"TRUE":True, "FALSE":False}, [True, False] ], "int32":32323232, "bool": True, "LONG STRING": "123456789012345678901234567890123456789012345678901234567890", "SHORT STRING": "1234567890" }, { True: "true", 8: False, "false": 0 } ] l = [] l.extend(l0) l.append(l0) l.extend(l1) return l def build_test_data(destdir): l = get_test_data_list() for i in range(len(l)): packer = msgpack.Packer() serialized = packer.pack(l[i]) f = open(os.path.join(destdir, str(i) + '.golden'), 'wb') f.write(serialized) f.close() def doRpcServer(port, stopTimeSec): class EchoHandler(object): def Echo123(self, msg1, msg2, msg3): return ("1:%s 2:%s 3:%s" % (msg1, msg2, msg3)) def EchoStruct(self, msg): return ("%s" % msg) addr = msgpackrpc.Address('localhost', port) server = msgpackrpc.Server(EchoHandler()) server.listen(addr) # run thread to stop it after stopTimeSec seconds if > 0 if stopTimeSec > 0: def myStopRpcServer(): server.stop() t = threading.Timer(stopTimeSec, myStopRpcServer) t.start() server.start() def doRpcClientToPythonSvc(port): address = msgpackrpc.Address('localhost', port) client = msgpackrpc.Client(address, unpack_encoding='utf-8') print client.call("Echo123", "A1", "B2", "C3") print client.call("EchoStruct", {"A" :"Aa", "B":"Bb", "C":"Cc"}) def doRpcClientToGoSvc(port): # print ">>>> port: ", port, " <<<<<" address = msgpackrpc.Address('localhost', port) client = msgpackrpc.Client(address, unpack_encoding='utf-8') print client.call("TestRpcInt.Echo123", ["A1", "B2", "C3"]) print client.call("TestRpcInt.EchoStruct", {"A" :"Aa", "B":"Bb", "C":"Cc"}) def doMain(args): if len(args) == 2 and args[0] == "testdata": build_test_data(args[1]) elif len(args) == 3 and args[0] == "rpc-server": doRpcServer(int(args[1]), int(args[2])) elif len(args) == 2 and args[0] == "rpc-client-python-service": doRpcClientToPythonSvc(int(args[1])) elif len(args) == 2 and args[0] == "rpc-client-go-service": doRpcClientToGoSvc(int(args[1])) else: print("Usage: msgpack_test.py " + "[testdata|rpc-server|rpc-client-python-service|rpc-client-go-service] ...") if __name__ == "__main__": doMain(sys.argv[1:]) golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/rpc.go000066400000000000000000000067541270406071200252540ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec import ( "bufio" "io" "net/rpc" "sync" ) // Rpc provides a rpc Server or Client Codec for rpc communication. type Rpc interface { ServerCodec(conn io.ReadWriteCloser, h Handle) rpc.ServerCodec ClientCodec(conn io.ReadWriteCloser, h Handle) rpc.ClientCodec } // RpcCodecBuffered allows access to the underlying bufio.Reader/Writer // used by the rpc connection. It accomodates use-cases where the connection // should be used by rpc and non-rpc functions, e.g. streaming a file after // sending an rpc response. type RpcCodecBuffered interface { BufferedReader() *bufio.Reader BufferedWriter() *bufio.Writer } // ------------------------------------- // rpcCodec defines the struct members and common methods. type rpcCodec struct { rwc io.ReadWriteCloser dec *Decoder enc *Encoder bw *bufio.Writer br *bufio.Reader mu sync.Mutex cls bool } func newRPCCodec(conn io.ReadWriteCloser, h Handle) rpcCodec { bw := bufio.NewWriter(conn) br := bufio.NewReader(conn) return rpcCodec{ rwc: conn, bw: bw, br: br, enc: NewEncoder(bw, h), dec: NewDecoder(br, h), } } func (c *rpcCodec) BufferedReader() *bufio.Reader { return c.br } func (c *rpcCodec) BufferedWriter() *bufio.Writer { return c.bw } func (c *rpcCodec) write(obj1, obj2 interface{}, writeObj2, doFlush bool) (err error) { if c.cls { return io.EOF } if err = c.enc.Encode(obj1); err != nil { return } if writeObj2 { if err = c.enc.Encode(obj2); err != nil { return } } if doFlush && c.bw != nil { return c.bw.Flush() } return } func (c *rpcCodec) read(obj interface{}) (err error) { if c.cls { return io.EOF } //If nil is passed in, we should still attempt to read content to nowhere. if obj == nil { var obj2 interface{} return c.dec.Decode(&obj2) } return c.dec.Decode(obj) } func (c *rpcCodec) Close() error { if c.cls { return io.EOF } c.cls = true return c.rwc.Close() } func (c *rpcCodec) ReadResponseBody(body interface{}) error { return c.read(body) } // ------------------------------------- type goRpcCodec struct { rpcCodec } func (c *goRpcCodec) WriteRequest(r *rpc.Request, body interface{}) error { // Must protect for concurrent access as per API c.mu.Lock() defer c.mu.Unlock() return c.write(r, body, true, true) } func (c *goRpcCodec) WriteResponse(r *rpc.Response, body interface{}) error { c.mu.Lock() defer c.mu.Unlock() return c.write(r, body, true, true) } func (c *goRpcCodec) ReadResponseHeader(r *rpc.Response) error { return c.read(r) } func (c *goRpcCodec) ReadRequestHeader(r *rpc.Request) error { return c.read(r) } func (c *goRpcCodec) ReadRequestBody(body interface{}) error { return c.read(body) } // ------------------------------------- // goRpc is the implementation of Rpc that uses the communication protocol // as defined in net/rpc package. type goRpc struct{} // GoRpc implements Rpc using the communication protocol defined in net/rpc package. // Its methods (ServerCodec and ClientCodec) return values that implement RpcCodecBuffered. var GoRpc goRpc func (x goRpc) ServerCodec(conn io.ReadWriteCloser, h Handle) rpc.ServerCodec { return &goRpcCodec{newRPCCodec(conn, h)} } func (x goRpc) ClientCodec(conn io.ReadWriteCloser, h Handle) rpc.ClientCodec { return &goRpcCodec{newRPCCodec(conn, h)} } var _ RpcCodecBuffered = (*rpcCodec)(nil) // ensure *rpcCodec implements RpcCodecBuffered golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/simple.go000066400000000000000000000272411270406071200257530ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec import "math" const ( _ uint8 = iota simpleVdNil = 1 simpleVdFalse = 2 simpleVdTrue = 3 simpleVdFloat32 = 4 simpleVdFloat64 = 5 // each lasts for 4 (ie n, n+1, n+2, n+3) simpleVdPosInt = 8 simpleVdNegInt = 12 // containers: each lasts for 4 (ie n, n+1, n+2, ... n+7) simpleVdString = 216 simpleVdByteArray = 224 simpleVdArray = 232 simpleVdMap = 240 simpleVdExt = 248 ) type simpleEncDriver struct { h *SimpleHandle w encWriter //b [8]byte } func (e *simpleEncDriver) isBuiltinType(rt uintptr) bool { return false } func (e *simpleEncDriver) encodeBuiltin(rt uintptr, v interface{}) { } func (e *simpleEncDriver) encodeNil() { e.w.writen1(simpleVdNil) } func (e *simpleEncDriver) encodeBool(b bool) { if b { e.w.writen1(simpleVdTrue) } else { e.w.writen1(simpleVdFalse) } } func (e *simpleEncDriver) encodeFloat32(f float32) { e.w.writen1(simpleVdFloat32) e.w.writeUint32(math.Float32bits(f)) } func (e *simpleEncDriver) encodeFloat64(f float64) { e.w.writen1(simpleVdFloat64) e.w.writeUint64(math.Float64bits(f)) } func (e *simpleEncDriver) encodeInt(v int64) { if v < 0 { e.encUint(uint64(-v), simpleVdNegInt) } else { e.encUint(uint64(v), simpleVdPosInt) } } func (e *simpleEncDriver) encodeUint(v uint64) { e.encUint(v, simpleVdPosInt) } func (e *simpleEncDriver) encUint(v uint64, bd uint8) { switch { case v <= math.MaxUint8: e.w.writen2(bd, uint8(v)) case v <= math.MaxUint16: e.w.writen1(bd + 1) e.w.writeUint16(uint16(v)) case v <= math.MaxUint32: e.w.writen1(bd + 2) e.w.writeUint32(uint32(v)) case v <= math.MaxUint64: e.w.writen1(bd + 3) e.w.writeUint64(v) } } func (e *simpleEncDriver) encLen(bd byte, length int) { switch { case length == 0: e.w.writen1(bd) case length <= math.MaxUint8: e.w.writen1(bd + 1) e.w.writen1(uint8(length)) case length <= math.MaxUint16: e.w.writen1(bd + 2) e.w.writeUint16(uint16(length)) case int64(length) <= math.MaxUint32: e.w.writen1(bd + 3) e.w.writeUint32(uint32(length)) default: e.w.writen1(bd + 4) e.w.writeUint64(uint64(length)) } } func (e *simpleEncDriver) encodeExtPreamble(xtag byte, length int) { e.encLen(simpleVdExt, length) e.w.writen1(xtag) } func (e *simpleEncDriver) encodeArrayPreamble(length int) { e.encLen(simpleVdArray, length) } func (e *simpleEncDriver) encodeMapPreamble(length int) { e.encLen(simpleVdMap, length) } func (e *simpleEncDriver) encodeString(c charEncoding, v string) { e.encLen(simpleVdString, len(v)) e.w.writestr(v) } func (e *simpleEncDriver) encodeSymbol(v string) { e.encodeString(c_UTF8, v) } func (e *simpleEncDriver) encodeStringBytes(c charEncoding, v []byte) { e.encLen(simpleVdByteArray, len(v)) e.w.writeb(v) } //------------------------------------ type simpleDecDriver struct { h *SimpleHandle r decReader bdRead bool bdType valueType bd byte //b [8]byte } func (d *simpleDecDriver) initReadNext() { if d.bdRead { return } d.bd = d.r.readn1() d.bdRead = true d.bdType = valueTypeUnset } func (d *simpleDecDriver) currentEncodedType() valueType { if d.bdType == valueTypeUnset { switch d.bd { case simpleVdNil: d.bdType = valueTypeNil case simpleVdTrue, simpleVdFalse: d.bdType = valueTypeBool case simpleVdPosInt, simpleVdPosInt + 1, simpleVdPosInt + 2, simpleVdPosInt + 3: d.bdType = valueTypeUint case simpleVdNegInt, simpleVdNegInt + 1, simpleVdNegInt + 2, simpleVdNegInt + 3: d.bdType = valueTypeInt case simpleVdFloat32, simpleVdFloat64: d.bdType = valueTypeFloat case simpleVdString, simpleVdString + 1, simpleVdString + 2, simpleVdString + 3, simpleVdString + 4: d.bdType = valueTypeString case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4: d.bdType = valueTypeBytes case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4: d.bdType = valueTypeExt case simpleVdArray, simpleVdArray + 1, simpleVdArray + 2, simpleVdArray + 3, simpleVdArray + 4: d.bdType = valueTypeArray case simpleVdMap, simpleVdMap + 1, simpleVdMap + 2, simpleVdMap + 3, simpleVdMap + 4: d.bdType = valueTypeMap default: decErr("currentEncodedType: Unrecognized d.vd: 0x%x", d.bd) } } return d.bdType } func (d *simpleDecDriver) tryDecodeAsNil() bool { if d.bd == simpleVdNil { d.bdRead = false return true } return false } func (d *simpleDecDriver) isBuiltinType(rt uintptr) bool { return false } func (d *simpleDecDriver) decodeBuiltin(rt uintptr, v interface{}) { } func (d *simpleDecDriver) decIntAny() (ui uint64, i int64, neg bool) { switch d.bd { case simpleVdPosInt: ui = uint64(d.r.readn1()) i = int64(ui) case simpleVdPosInt + 1: ui = uint64(d.r.readUint16()) i = int64(ui) case simpleVdPosInt + 2: ui = uint64(d.r.readUint32()) i = int64(ui) case simpleVdPosInt + 3: ui = uint64(d.r.readUint64()) i = int64(ui) case simpleVdNegInt: ui = uint64(d.r.readn1()) i = -(int64(ui)) neg = true case simpleVdNegInt + 1: ui = uint64(d.r.readUint16()) i = -(int64(ui)) neg = true case simpleVdNegInt + 2: ui = uint64(d.r.readUint32()) i = -(int64(ui)) neg = true case simpleVdNegInt + 3: ui = uint64(d.r.readUint64()) i = -(int64(ui)) neg = true default: decErr("decIntAny: Integer only valid from pos/neg integer1..8. Invalid descriptor: %v", d.bd) } // don't do this check, because callers may only want the unsigned value. // if ui > math.MaxInt64 { // decErr("decIntAny: Integer out of range for signed int64: %v", ui) // } return } func (d *simpleDecDriver) decodeInt(bitsize uint8) (i int64) { _, i, _ = d.decIntAny() checkOverflow(0, i, bitsize) d.bdRead = false return } func (d *simpleDecDriver) decodeUint(bitsize uint8) (ui uint64) { ui, i, neg := d.decIntAny() if neg { decErr("Assigning negative signed value: %v, to unsigned type", i) } checkOverflow(ui, 0, bitsize) d.bdRead = false return } func (d *simpleDecDriver) decodeFloat(chkOverflow32 bool) (f float64) { switch d.bd { case simpleVdFloat32: f = float64(math.Float32frombits(d.r.readUint32())) case simpleVdFloat64: f = math.Float64frombits(d.r.readUint64()) default: if d.bd >= simpleVdPosInt && d.bd <= simpleVdNegInt+3 { _, i, _ := d.decIntAny() f = float64(i) } else { decErr("Float only valid from float32/64: Invalid descriptor: %v", d.bd) } } checkOverflowFloat32(f, chkOverflow32) d.bdRead = false return } // bool can be decoded from bool only (single byte). func (d *simpleDecDriver) decodeBool() (b bool) { switch d.bd { case simpleVdTrue: b = true case simpleVdFalse: default: decErr("Invalid single-byte value for bool: %s: %x", msgBadDesc, d.bd) } d.bdRead = false return } func (d *simpleDecDriver) readMapLen() (length int) { d.bdRead = false return d.decLen() } func (d *simpleDecDriver) readArrayLen() (length int) { d.bdRead = false return d.decLen() } func (d *simpleDecDriver) decLen() int { switch d.bd % 8 { case 0: return 0 case 1: return int(d.r.readn1()) case 2: return int(d.r.readUint16()) case 3: ui := uint64(d.r.readUint32()) checkOverflow(ui, 0, intBitsize) return int(ui) case 4: ui := d.r.readUint64() checkOverflow(ui, 0, intBitsize) return int(ui) } decErr("decLen: Cannot read length: bd%8 must be in range 0..4. Got: %d", d.bd%8) return -1 } func (d *simpleDecDriver) decodeString() (s string) { s = string(d.r.readn(d.decLen())) d.bdRead = false return } func (d *simpleDecDriver) decodeBytes(bs []byte) (bsOut []byte, changed bool) { if clen := d.decLen(); clen > 0 { // if no contents in stream, don't update the passed byteslice if len(bs) != clen { if len(bs) > clen { bs = bs[:clen] } else { bs = make([]byte, clen) } bsOut = bs changed = true } d.r.readb(bs) } d.bdRead = false return } func (d *simpleDecDriver) decodeExt(verifyTag bool, tag byte) (xtag byte, xbs []byte) { switch d.bd { case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4: l := d.decLen() xtag = d.r.readn1() if verifyTag && xtag != tag { decErr("Wrong extension tag. Got %b. Expecting: %v", xtag, tag) } xbs = d.r.readn(l) case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4: xbs, _ = d.decodeBytes(nil) default: decErr("Invalid d.vd for extensions (Expecting extensions or byte array). Got: 0x%x", d.bd) } d.bdRead = false return } func (d *simpleDecDriver) decodeNaked() (v interface{}, vt valueType, decodeFurther bool) { d.initReadNext() switch d.bd { case simpleVdNil: vt = valueTypeNil case simpleVdFalse: vt = valueTypeBool v = false case simpleVdTrue: vt = valueTypeBool v = true case simpleVdPosInt, simpleVdPosInt + 1, simpleVdPosInt + 2, simpleVdPosInt + 3: vt = valueTypeUint ui, _, _ := d.decIntAny() v = ui case simpleVdNegInt, simpleVdNegInt + 1, simpleVdNegInt + 2, simpleVdNegInt + 3: vt = valueTypeInt _, i, _ := d.decIntAny() v = i case simpleVdFloat32: vt = valueTypeFloat v = d.decodeFloat(true) case simpleVdFloat64: vt = valueTypeFloat v = d.decodeFloat(false) case simpleVdString, simpleVdString + 1, simpleVdString + 2, simpleVdString + 3, simpleVdString + 4: vt = valueTypeString v = d.decodeString() case simpleVdByteArray, simpleVdByteArray + 1, simpleVdByteArray + 2, simpleVdByteArray + 3, simpleVdByteArray + 4: vt = valueTypeBytes v, _ = d.decodeBytes(nil) case simpleVdExt, simpleVdExt + 1, simpleVdExt + 2, simpleVdExt + 3, simpleVdExt + 4: vt = valueTypeExt l := d.decLen() var re RawExt re.Tag = d.r.readn1() re.Data = d.r.readn(l) v = &re vt = valueTypeExt case simpleVdArray, simpleVdArray + 1, simpleVdArray + 2, simpleVdArray + 3, simpleVdArray + 4: vt = valueTypeArray decodeFurther = true case simpleVdMap, simpleVdMap + 1, simpleVdMap + 2, simpleVdMap + 3, simpleVdMap + 4: vt = valueTypeMap decodeFurther = true default: decErr("decodeNaked: Unrecognized d.vd: 0x%x", d.bd) } if !decodeFurther { d.bdRead = false } return } //------------------------------------ // SimpleHandle is a Handle for a very simple encoding format. // // simple is a simplistic codec similar to binc, but not as compact. // - Encoding of a value is always preceeded by the descriptor byte (bd) // - True, false, nil are encoded fully in 1 byte (the descriptor) // - Integers (intXXX, uintXXX) are encoded in 1, 2, 4 or 8 bytes (plus a descriptor byte). // There are positive (uintXXX and intXXX >= 0) and negative (intXXX < 0) integers. // - Floats are encoded in 4 or 8 bytes (plus a descriptor byte) // - Lenght of containers (strings, bytes, array, map, extensions) // are encoded in 0, 1, 2, 4 or 8 bytes. // Zero-length containers have no length encoded. // For others, the number of bytes is given by pow(2, bd%3) // - maps are encoded as [bd] [length] [[key][value]]... // - arrays are encoded as [bd] [length] [value]... // - extensions are encoded as [bd] [length] [tag] [byte]... // - strings/bytearrays are encoded as [bd] [length] [byte]... // // The full spec will be published soon. type SimpleHandle struct { BasicHandle } func (h *SimpleHandle) newEncDriver(w encWriter) encDriver { return &simpleEncDriver{w: w, h: h} } func (h *SimpleHandle) newDecDriver(r decReader) decDriver { return &simpleDecDriver{r: r, h: h} } func (_ *SimpleHandle) writeExt() bool { return true } func (h *SimpleHandle) getBasicHandle() *BasicHandle { return &h.BasicHandle } var _ decDriver = (*simpleDecDriver)(nil) var _ encDriver = (*simpleEncDriver)(nil) golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/time.go000066400000000000000000000140631270406071200254160ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec import ( "time" ) var ( timeDigits = [...]byte{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9'} ) // EncodeTime encodes a time.Time as a []byte, including // information on the instant in time and UTC offset. // // Format Description // // A timestamp is composed of 3 components: // // - secs: signed integer representing seconds since unix epoch // - nsces: unsigned integer representing fractional seconds as a // nanosecond offset within secs, in the range 0 <= nsecs < 1e9 // - tz: signed integer representing timezone offset in minutes east of UTC, // and a dst (daylight savings time) flag // // When encoding a timestamp, the first byte is the descriptor, which // defines which components are encoded and how many bytes are used to // encode secs and nsecs components. *If secs/nsecs is 0 or tz is UTC, it // is not encoded in the byte array explicitly*. // // Descriptor 8 bits are of the form `A B C DDD EE`: // A: Is secs component encoded? 1 = true // B: Is nsecs component encoded? 1 = true // C: Is tz component encoded? 1 = true // DDD: Number of extra bytes for secs (range 0-7). // If A = 1, secs encoded in DDD+1 bytes. // If A = 0, secs is not encoded, and is assumed to be 0. // If A = 1, then we need at least 1 byte to encode secs. // DDD says the number of extra bytes beyond that 1. // E.g. if DDD=0, then secs is represented in 1 byte. // if DDD=2, then secs is represented in 3 bytes. // EE: Number of extra bytes for nsecs (range 0-3). // If B = 1, nsecs encoded in EE+1 bytes (similar to secs/DDD above) // // Following the descriptor bytes, subsequent bytes are: // // secs component encoded in `DDD + 1` bytes (if A == 1) // nsecs component encoded in `EE + 1` bytes (if B == 1) // tz component encoded in 2 bytes (if C == 1) // // secs and nsecs components are integers encoded in a BigEndian // 2-complement encoding format. // // tz component is encoded as 2 bytes (16 bits). Most significant bit 15 to // Least significant bit 0 are described below: // // Timezone offset has a range of -12:00 to +14:00 (ie -720 to +840 minutes). // Bit 15 = have\_dst: set to 1 if we set the dst flag. // Bit 14 = dst\_on: set to 1 if dst is in effect at the time, or 0 if not. // Bits 13..0 = timezone offset in minutes. It is a signed integer in Big Endian format. // func encodeTime(t time.Time) []byte { //t := rv.Interface().(time.Time) tsecs, tnsecs := t.Unix(), t.Nanosecond() var ( bd byte btmp [8]byte bs [16]byte i int = 1 ) l := t.Location() if l == time.UTC { l = nil } if tsecs != 0 { bd = bd | 0x80 bigen.PutUint64(btmp[:], uint64(tsecs)) f := pruneSignExt(btmp[:], tsecs >= 0) bd = bd | (byte(7-f) << 2) copy(bs[i:], btmp[f:]) i = i + (8 - f) } if tnsecs != 0 { bd = bd | 0x40 bigen.PutUint32(btmp[:4], uint32(tnsecs)) f := pruneSignExt(btmp[:4], true) bd = bd | byte(3-f) copy(bs[i:], btmp[f:4]) i = i + (4 - f) } if l != nil { bd = bd | 0x20 // Note that Go Libs do not give access to dst flag. _, zoneOffset := t.Zone() //zoneName, zoneOffset := t.Zone() zoneOffset /= 60 z := uint16(zoneOffset) bigen.PutUint16(btmp[:2], z) // clear dst flags bs[i] = btmp[0] & 0x3f bs[i+1] = btmp[1] i = i + 2 } bs[0] = bd return bs[0:i] } // DecodeTime decodes a []byte into a time.Time. func decodeTime(bs []byte) (tt time.Time, err error) { bd := bs[0] var ( tsec int64 tnsec uint32 tz uint16 i byte = 1 i2 byte n byte ) if bd&(1<<7) != 0 { var btmp [8]byte n = ((bd >> 2) & 0x7) + 1 i2 = i + n copy(btmp[8-n:], bs[i:i2]) //if first bit of bs[i] is set, then fill btmp[0..8-n] with 0xff (ie sign extend it) if bs[i]&(1<<7) != 0 { copy(btmp[0:8-n], bsAll0xff) //for j,k := byte(0), 8-n; j < k; j++ { btmp[j] = 0xff } } i = i2 tsec = int64(bigen.Uint64(btmp[:])) } if bd&(1<<6) != 0 { var btmp [4]byte n = (bd & 0x3) + 1 i2 = i + n copy(btmp[4-n:], bs[i:i2]) i = i2 tnsec = bigen.Uint32(btmp[:]) } if bd&(1<<5) == 0 { tt = time.Unix(tsec, int64(tnsec)).UTC() return } // In stdlib time.Parse, when a date is parsed without a zone name, it uses "" as zone name. // However, we need name here, so it can be shown when time is printed. // Zone name is in form: UTC-08:00. // Note that Go Libs do not give access to dst flag, so we ignore dst bits i2 = i + 2 tz = bigen.Uint16(bs[i:i2]) i = i2 // sign extend sign bit into top 2 MSB (which were dst bits): if tz&(1<<13) == 0 { // positive tz = tz & 0x3fff //clear 2 MSBs: dst bits } else { // negative tz = tz | 0xc000 //set 2 MSBs: dst bits //tzname[3] = '-' (TODO: verify. this works here) } tzint := int16(tz) if tzint == 0 { tt = time.Unix(tsec, int64(tnsec)).UTC() } else { // For Go Time, do not use a descriptive timezone. // It's unnecessary, and makes it harder to do a reflect.DeepEqual. // The Offset already tells what the offset should be, if not on UTC and unknown zone name. // var zoneName = timeLocUTCName(tzint) tt = time.Unix(tsec, int64(tnsec)).In(time.FixedZone("", int(tzint)*60)) } return } func timeLocUTCName(tzint int16) string { if tzint == 0 { return "UTC" } var tzname = []byte("UTC+00:00") //tzname := fmt.Sprintf("UTC%s%02d:%02d", tzsign, tz/60, tz%60) //perf issue using Sprintf. inline below. //tzhr, tzmin := tz/60, tz%60 //faster if u convert to int first var tzhr, tzmin int16 if tzint < 0 { tzname[3] = '-' // (TODO: verify. this works here) tzhr, tzmin = -tzint/60, (-tzint)%60 } else { tzhr, tzmin = tzint/60, tzint%60 } tzname[4] = timeDigits[tzhr/10] tzname[5] = timeDigits[tzhr%10] tzname[7] = timeDigits[tzmin/10] tzname[8] = timeDigits[tzmin%10] return string(tzname) //return time.FixedZone(string(tzname), int(tzint)*60) } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/codec/z_helper_test.go000066400000000000000000000044141270406071200273260ustar00rootroot00000000000000// Copyright (c) 2012, 2013 Ugorji Nwoke. All rights reserved. // Use of this source code is governed by a BSD-style license found in the LICENSE file. package codec // All non-std package dependencies related to testing live in this file, // so porting to different environment is easy (just update functions). // // Also, this file is called z_helper_test, to give a "hint" to compiler // that its init() function should be called last. (not guaranteed by spec) import ( "errors" "reflect" "flag" "testing" ) var ( testLogToT = true failNowOnFail = true ) func init() { testInitFlags() benchInitFlags() flag.Parse() testInit() benchInit() } func checkErrT(t *testing.T, err error) { if err != nil { logT(t, err.Error()) failT(t) } } func checkEqualT(t *testing.T, v1 interface{}, v2 interface{}, desc string) (err error) { if err = deepEqual(v1, v2); err != nil { logT(t, "Not Equal: %s: %v. v1: %v, v2: %v", desc, err, v1, v2) failT(t) } return } func logT(x interface{}, format string, args ...interface{}) { if t, ok := x.(*testing.T); ok && t != nil && testLogToT { t.Logf(format, args...) } else if b, ok := x.(*testing.B); ok && b != nil && testLogToT { b.Logf(format, args...) } else { debugf(format, args...) } } func failT(t *testing.T) { if failNowOnFail { t.FailNow() } else { t.Fail() } } func deepEqual(v1, v2 interface{}) (err error) { if !reflect.DeepEqual(v1, v2) { err = errors.New("Not Match") } return } func approxDataSize(rv reflect.Value) (sum int) { switch rk := rv.Kind(); rk { case reflect.Invalid: case reflect.Ptr, reflect.Interface: sum += int(rv.Type().Size()) sum += approxDataSize(rv.Elem()) case reflect.Slice: sum += int(rv.Type().Size()) for j := 0; j < rv.Len(); j++ { sum += approxDataSize(rv.Index(j)) } case reflect.String: sum += int(rv.Type().Size()) sum += rv.Len() case reflect.Map: sum += int(rv.Type().Size()) for _, mk := range rv.MapKeys() { sum += approxDataSize(mk) sum += approxDataSize(rv.MapIndex(mk)) } case reflect.Struct: //struct size already includes the full data size. //sum += int(rv.Type().Size()) for j := 0; j < rv.NumField(); j++ { sum += approxDataSize(rv.Field(j)) } default: //pure value types sum += int(rv.Type().Size()) } return } golang-github-hashicorp-go-msgpack-0.0~git20150518.0.fa3f638/msgpack.org.md000066400000000000000000000024321270406071200256060ustar00rootroot00000000000000**MessagePack and [Binc](http://github.com/ugorji/binc) Codec for [Go](http://golang.org) Language.** *A High Performance, Feature-Rich, Idiomatic encode/decode and rpc library*. To install: go get github.com/ugorji/go/codec Source: [http://github.com/ugorji/go] Online documentation: [http://godoc.org/github.com/ugorji/go/codec] Typical usage: ```go // create and use decoder/encoder var ( v interface{} // value to decode/encode into r io.Reader w io.Writer b []byte mh codec.MsgpackHandle ) dec = codec.NewDecoder(r, &mh) dec = codec.NewDecoderBytes(b, &mh) err = dec.Decode(&v) enc = codec.NewEncoder(w, &mh) enc = codec.NewEncoderBytes(&b, &mh) err = enc.Encode(v) //RPC Server go func() { for { conn, err := listener.Accept() rpcCodec := codec.GoRpc.ServerCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h) rpc.ServeCodec(rpcCodec) } }() //RPC Communication (client side) conn, err = net.Dial("tcp", "localhost:5555") rpcCodec := codec.GoRpc.ClientCodec(conn, h) //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h) client := rpc.NewClientWithCodec(rpcCodec) ```