Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

wire: only borrow/return binaryFreeList buffers at the message level #1426

Closed
wants to merge 52 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
8d005ab
wire/bench_test: report allocs in benchmarks
cfromknecht Jan 25, 2020
92351be
wire/bench: add witness block
cfromknecht Jan 25, 2020
e766de6
wire/common: optimize Read/WriteVarInt
cfromknecht Jan 25, 2020
a6da5b3
wire: introduce Read/WriteVarIntBuf to reuse buffers between invocations
cfromknecht Jan 25, 2020
35ac17f
wire/msgtx: use Read/WriteVarIntBuf in tx serialization
cfromknecht Jan 25, 2020
904f63c
wire/msgtx: reuse tx-level buffer for version and locktime
cfromknecht Jan 25, 2020
292e8f8
wire/common: add optimized Read/WriteVarBytesBuf
cfromknecht Jan 25, 2020
155d751
wire/msgtx: introduce optimized read/writeOutPointBuf
cfromknecht Jan 25, 2020
d24e5c7
wire/msgtx: introduce optimized writeTxInBuf
cfromknecht Jan 25, 2020
05b0ba0
wire/msgtx: use writeTxInBuf in txn encoding
cfromknecht Jan 25, 2020
ede651d
wire/msgtx: introduce optimized readScriptBuf
cfromknecht Apr 30, 2019
15d2e47
wire/msgtx: introduce optimized readTxInBuf
cfromknecht Apr 30, 2019
1f8a45b
wire/msgtx: use readTxInBuf in txn serialization
cfromknecht Jan 25, 2020
84d80e0
wire/msgtx: introduce optimized WriteTxOutBuf
cfromknecht Jan 25, 2020
ac4db9d
wire/msgtx: use WriteTxOutBuf in txn serialization
cfromknecht Jan 25, 2020
080cad5
wire/msgtx: introduce optimized readTxOutBuf
cfromknecht Jan 25, 2020
79976d0
wire/msgtx: use readTxOutBuf in txn serialization
cfromknecht Jan 25, 2020
56f1965
wire/msgtx: introduce optimized writeTxWitnessBuf
cfromknecht Jan 25, 2020
11fe696
wire/msgtx: use writeTxWitnessBuf in txn serialization
cfromknecht Jan 25, 2020
0f7d652
wire/msgtx: use readScriptBuf in txn serialization
cfromknecht Jan 25, 2020
84f1451
wire/bench_test: introduce optimized readBlockHeaderBuf
cfromknecht Jan 25, 2020
d733f6d
wire/blockheader: introduce optimized writeBlockHeaderBuf
cfromknecht Jan 25, 2020
ef0fd86
wire/invvect: add optimized readInvVectBuf and writeInvVectBuf
cfromknecht Jan 25, 2020
0e1207f
wire/msggetblocks: optimize by reusing small buffer
cfromknecht Jan 25, 2020
db91c07
wire/msgblock: use only one small buffer per block encode/decode
cfromknecht Jan 25, 2020
549d33f
wire/msgblock: optimize DeserializeTxLoc by reusing small buffers
cfromknecht Jan 25, 2020
b8f2f38
wire/msggetheaders: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
0645d64
wire/msgheaders: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
8f0a986
wire/msggetcfheaders: use single small buffer for encode/decode
cfromknecht Jan 25, 2020
73693d9
wire/msgcfheaders: optimize encode/decode by using one small buffer
cfromknecht Jan 25, 2020
4b4dfaa
wire/msggetcfcheckpt: optimize by removing read/writeElement
cfromknecht Jan 25, 2020
cdd28f0
wire/msgcfcheckpt: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
22cdfc2
wire/msginv: optimize by reusing small buffers
cfromknecht Jan 25, 2020
10c5459
wire/msggetdata: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
6d3d4b2
wiree/msggetcfilters: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
af9fdaa
wire/msgcfilter: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
a1c1620
wire/msgnotfound: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
33c8877
wire/invvect: remove unused readInvVect and writeInvVect
cfromknecht Jan 25, 2020
6147d8b
wire/common: add optimized writeVarStrBuf an readVarStrBuf
cfromknecht Jan 25, 2020
49aba92
wire/msgreject: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
518fa70
wire/netaddress: add optimiezed read/writeNetAddressBuf
cfromknecht Jan 25, 2020
f4388ef
wire/msgmerkleblock: optimize serialization by reusing small buffers
cfromknecht Jan 25, 2020
a5affa7
wire/msgping: remove usage for read/writeElement
cfromknecht Jan 25, 2020
08c24c9
wire/msgpong: remove usage of read/writeElement
cfromknecht Jan 25, 2020
b92f8e2
wire/msgtx: remove unused writeTxWitness
cfromknecht Jan 25, 2020
0b121ec
wire/msgtx: remove unused readTxOut
cfromknecht Jan 25, 2020
055a6a8
wire/msgtx: remove unused writeTxIn
cfromknecht Jan 25, 2020
0b67c05
wire/msgtx: remove unused readTxIn
cfromknecht Jan 25, 2020
49ac379
wire/msgtx: remove unused readScript
cfromknecht Jan 25, 2020
512571b
wire/msgtx: remove unused read/writeOutPoint
cfromknecht Jan 25, 2020
e7ed275
wire/msgtx: use tx-level script slab
cfromknecht Jan 25, 2020
86ba486
wire/msgblock+msgtx: user block-level script slab
cfromknecht Jan 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
437 changes: 430 additions & 7 deletions wire/bench_test.go

Large diffs are not rendered by default.

103 changes: 98 additions & 5 deletions wire/blockheader.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,16 +113,109 @@ func NewBlockHeader(version int32, prevHash, merkleRootHash *chainhash.Hash,
// readBlockHeader reads a bitcoin block header from r. See Deserialize for
// decoding block headers stored to disk, such as in a database, as opposed to
// decoding from the wire.
//
// DEPRECATED: Use readBlockHeaderBuf instead.
func readBlockHeader(r io.Reader, pver uint32, bh *BlockHeader) error {
return readElements(r, &bh.Version, &bh.PrevBlock, &bh.MerkleRoot,
(*uint32Time)(&bh.Timestamp), &bh.Bits, &bh.Nonce)
buf := binarySerializer.Borrow()
err := readBlockHeaderBuf(r, pver, bh, buf)
binarySerializer.Return(buf)
return err
}

// readBlockHeaderBuf reads a bitcoin block header from r. See Deserialize for
// decoding block headers stored to disk, such as in a database, as opposed to
// decoding from the wire.
//
// If b is non-nil, the provided buffer will be used for serializing small
// values. Otherwise a buffer will be drawn from the binarySerializer's pool
// and return when the method finishes.
//
// NOTE: b MUST either be nil or at least an 8-byte slice.
func readBlockHeaderBuf(r io.Reader, pver uint32, bh *BlockHeader,
buf []byte) error {

if _, err := io.ReadFull(r, buf[:4]); err != nil {
return err
}
bh.Version = int32(littleEndian.Uint32(buf[:4]))

if _, err := io.ReadFull(r, bh.PrevBlock[:]); err != nil {
return err
}

if _, err := io.ReadFull(r, bh.MerkleRoot[:]); err != nil {
return err
}

if _, err := io.ReadFull(r, buf[:4]); err != nil {
return err
}
bh.Timestamp = time.Unix(int64(littleEndian.Uint32(buf[:4])), 0)

if _, err := io.ReadFull(r, buf[:4]); err != nil {
return err
}
bh.Bits = littleEndian.Uint32(buf[:4])

if _, err := io.ReadFull(r, buf[:4]); err != nil {
return err
}
bh.Nonce = littleEndian.Uint32(buf[:4])

return nil
}

// writeBlockHeader writes a bitcoin block header to w. See Serialize for
// encoding block headers to be stored to disk, such as in a database, as
// opposed to encoding for the wire.
//
// DEPRECATED: Use writeBlockHeaderBuf instead.
func writeBlockHeader(w io.Writer, pver uint32, bh *BlockHeader) error {
sec := uint32(bh.Timestamp.Unix())
return writeElements(w, bh.Version, &bh.PrevBlock, &bh.MerkleRoot,
sec, bh.Bits, bh.Nonce)
buf := binarySerializer.Borrow()
err := writeBlockHeaderBuf(w, pver, bh, buf)
binarySerializer.Return(buf)
return err
}

// writeBlockHeaderBuf writes a bitcoin block header to w. See Serialize for
// encoding block headers to be stored to disk, such as in a database, as
// opposed to encoding for the wire.
//
// If b is non-nil, the provided buffer will be used for serializing small
// values. Otherwise a buffer will be drawn from the binarySerializer's pool
// and return when the method finishes.
//
// NOTE: b MUST either be nil or at least an 8-byte slice.
func writeBlockHeaderBuf(w io.Writer, pver uint32, bh *BlockHeader,
buf []byte) error {

littleEndian.PutUint32(buf[:4], uint32(bh.Version))
if _, err := w.Write(buf[:4]); err != nil {
return err
}

if _, err := w.Write(bh.PrevBlock[:]); err != nil {
return err
}

if _, err := w.Write(bh.MerkleRoot[:]); err != nil {
return err
}

littleEndian.PutUint32(buf[:4], uint32(bh.Timestamp.Unix()))
if _, err := w.Write(buf[:4]); err != nil {
return err
}

littleEndian.PutUint32(buf[:4], bh.Bits)
if _, err := w.Write(buf[:4]); err != nil {
return err
}

littleEndian.PutUint32(buf[:4], bh.Nonce)
if _, err := w.Write(buf[:4]); err != nil {
return err
}

return nil
}
167 changes: 128 additions & 39 deletions wire/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -474,19 +474,29 @@ func writeElements(w io.Writer, elements ...interface{}) error {

// ReadVarInt reads a variable length integer from r and returns it as a uint64.
func ReadVarInt(r io.Reader, pver uint32) (uint64, error) {
discriminant, err := binarySerializer.Uint8(r)
if err != nil {
buf := binarySerializer.Borrow()
n, err := ReadVarIntBuf(r, pver, buf)
binarySerializer.Return(buf)
return n, err
}

// ReadVarIntBuf reads a variable length integer from r using a preallocated
// scratch buffer and returns it as a uint64.
//
// NOTE: buf MUST at least an 8-byte slice.
func ReadVarIntBuf(r io.Reader, pver uint32, buf []byte) (uint64, error) {
if _, err := io.ReadFull(r, buf[:1]); err != nil {
return 0, err
}
discriminant := buf[0]

var rv uint64
switch discriminant {
case 0xff:
sv, err := binarySerializer.Uint64(r, littleEndian)
if err != nil {
if _, err := io.ReadFull(r, buf); err != nil {
return 0, err
}
rv = sv
rv = littleEndian.Uint64(buf)

// The encoding is not canonical if the value could have been
// encoded using fewer bytes.
Expand All @@ -497,11 +507,10 @@ func ReadVarInt(r io.Reader, pver uint32) (uint64, error) {
}

case 0xfe:
sv, err := binarySerializer.Uint32(r, littleEndian)
if err != nil {
if _, err := io.ReadFull(r, buf[:4]); err != nil {
return 0, err
}
rv = uint64(sv)
rv = uint64(littleEndian.Uint32(buf[:4]))

// The encoding is not canonical if the value could have been
// encoded using fewer bytes.
Expand All @@ -512,11 +521,10 @@ func ReadVarInt(r io.Reader, pver uint32) (uint64, error) {
}

case 0xfd:
sv, err := binarySerializer.Uint16(r, littleEndian)
if err != nil {
if _, err := io.ReadFull(r, buf[:2]); err != nil {
return 0, err
}
rv = uint64(sv)
rv = uint64(littleEndian.Uint16(buf[:2]))

// The encoding is not canonical if the value could have been
// encoded using fewer bytes.
Expand All @@ -536,31 +544,45 @@ func ReadVarInt(r io.Reader, pver uint32) (uint64, error) {
// WriteVarInt serializes val to w using a variable number of bytes depending
// on its value.
func WriteVarInt(w io.Writer, pver uint32, val uint64) error {
if val < 0xfd {
return binarySerializer.PutUint8(w, uint8(val))
}
buf := binarySerializer.Borrow()
err := WriteVarIntBuf(w, pver, val, buf)
binarySerializer.Return(buf)
return err
}

if val <= math.MaxUint16 {
err := binarySerializer.PutUint8(w, 0xfd)
if err != nil {
return err
}
return binarySerializer.PutUint16(w, littleEndian, uint16(val))
}
// WriteVarIntBuf serializes val to w using a variable number of bytes depending
// on its value using a preallocated scratch buffer.
//
// NOTE: buf MUST at least an 8-byte slice.
func WriteVarIntBuf(w io.Writer, pver uint32, val uint64, buf []byte) error {
switch {
case val < 0xfd:
buf[0] = uint8(val)
_, err := w.Write(buf[:1])
return err

if val <= math.MaxUint32 {
err := binarySerializer.PutUint8(w, 0xfe)
if err != nil {
case val <= math.MaxUint16:
buf[0] = 0xfd
littleEndian.PutUint16(buf[1:3], uint16(val))
_, err := w.Write(buf[:3])
return err

case val <= math.MaxUint32:
buf[0] = 0xfe
littleEndian.PutUint32(buf[1:5], uint32(val))
_, err := w.Write(buf[:5])
return err

default:
buf[0] = 0xff
if _, err := w.Write(buf[:1]); err != nil {
return err
}
return binarySerializer.PutUint32(w, littleEndian, uint32(val))
}

err := binarySerializer.PutUint8(w, 0xff)
if err != nil {
littleEndian.PutUint64(buf, val)
_, err := w.Write(buf)
return err
}
return binarySerializer.PutUint64(w, littleEndian, val)
}

// VarIntSerializeSize returns the number of bytes it would take to serialize
Expand Down Expand Up @@ -593,7 +615,26 @@ func VarIntSerializeSize(val uint64) int {
// maximum block payload size since it helps protect against memory exhaustion
// attacks and forced panics through malformed messages.
func ReadVarString(r io.Reader, pver uint32) (string, error) {
count, err := ReadVarInt(r, pver)
buf := binarySerializer.Borrow()
str, err := readVarStringBuf(r, pver, buf)
binarySerializer.Return(buf)
return str, err
}

// readVarStringBuf reads a variable length string from r and returns it as a Go
// string. A variable length string is encoded as a variable length integer
// containing the length of the string followed by the bytes that represent the
// string itself. An error is returned if the length is greater than the
// maximum block payload size since it helps protect against memory exhaustion
// attacks and forced panics through malformed messages.
//
// If b is non-nil, the provided buffer will be used for serializing small
// values. Otherwise a buffer will be drawn from the binarySerializer's pool
// and return when the method finishes.
//
// NOTE: b MUST either be nil or at least an 8-byte slice.
func readVarStringBuf(r io.Reader, pver uint32, buf []byte) (string, error) {
count, err := ReadVarIntBuf(r, pver, buf)
if err != nil {
return "", err
}
Expand All @@ -607,22 +648,39 @@ func ReadVarString(r io.Reader, pver uint32) (string, error) {
return "", messageError("ReadVarString", str)
}

buf := make([]byte, count)
_, err = io.ReadFull(r, buf)
str := make([]byte, count)
_, err = io.ReadFull(r, str)
if err != nil {
return "", err
}
return string(buf), nil
return string(str), nil
}

// WriteVarString serializes str to w as a variable length integer containing
// the length of the string followed by the bytes that represent the string
// itself.
func WriteVarString(w io.Writer, pver uint32, str string) error {
err := WriteVarInt(w, pver, uint64(len(str)))
buf := binarySerializer.Borrow()
err := writeVarStringBuf(w, pver, str, buf)
binarySerializer.Return(buf)
return err
}

// writeVarStringBuf serializes str to w as a variable length integer containing
// the length of the string followed by the bytes that represent the string
// itself.
//
// If b is non-nil, the provided buffer will be used for serializing small
// values. Otherwise a buffer will be drawn from the binarySerializer's pool
// and return when the method finishes.
//
// NOTE: b MUST either be nil or at least an 8-byte slice.
func writeVarStringBuf(w io.Writer, pver uint32, str string, buf []byte) error {
err := WriteVarIntBuf(w, pver, uint64(len(str)), buf)
if err != nil {
return err
}

_, err = w.Write([]byte(str))
return err
}
Expand All @@ -637,7 +695,25 @@ func WriteVarString(w io.Writer, pver uint32, str string) error {
func ReadVarBytes(r io.Reader, pver uint32, maxAllowed uint32,
fieldName string) ([]byte, error) {

count, err := ReadVarInt(r, pver)
buf := binarySerializer.Borrow()
b, err := ReadVarBytesBuf(r, pver, buf, maxAllowed, fieldName)
binarySerializer.Return(buf)
return b, err
}

// ReadVarBytesBuf reads a variable length byte array. A byte array is encoded
// as a varInt containing the length of the array followed by the bytes
// themselves. An error is returned if the length is greater than the
// passed maxAllowed parameter which helps protect against memory exhaustion
// attacks and forced panics through malformed messages. The fieldName
// parameter is only used for the error message so it provides more context in
// the error. If b is non-nil, the provided buffer will be used for serializing
// small values. Otherwise a buffer will be drawn from the binarySerializer's
// pool and return when the method finishes.
func ReadVarBytesBuf(r io.Reader, pver uint32, buf []byte, maxAllowed uint32,
fieldName string) ([]byte, error) {

count, err := ReadVarIntBuf(r, pver, buf)
if err != nil {
return nil, err
}
Expand All @@ -651,19 +727,32 @@ func ReadVarBytes(r io.Reader, pver uint32, maxAllowed uint32,
return nil, messageError("ReadVarBytes", str)
}

b := make([]byte, count)
_, err = io.ReadFull(r, b)
bytes := make([]byte, count)
_, err = io.ReadFull(r, bytes)
if err != nil {
return nil, err
}
return b, nil
return bytes, nil
}

// WriteVarBytes serializes a variable length byte array to w as a varInt
// containing the number of bytes, followed by the bytes themselves.
func WriteVarBytes(w io.Writer, pver uint32, bytes []byte) error {
buf := binarySerializer.Borrow()
err := WriteVarBytesBuf(w, pver, bytes, buf)
binarySerializer.Return(buf)
return err
}

// WriteVarBytesBuf serializes a variable length byte array to w as a varInt
// containing the number of bytes, followed by the bytes themselves. If b is
// non-nil, the provided buffer will be used for serializing small values.
// Otherwise a buffer will be drawn from the binarySerializer's pool and return
// when the method finishes.
func WriteVarBytesBuf(w io.Writer, pver uint32, bytes, buf []byte) error {
slen := uint64(len(bytes))
err := WriteVarInt(w, pver, slen)

err := WriteVarIntBuf(w, pver, slen, buf)
if err != nil {
return err
}
Expand Down
Loading