Panic: chacha20poly1305: invalid buffer overlap

[ridenaio]

Hi, we've caught this panic:

panic: chacha20poly1305: invalid buffer overlap

goroutine 992523603 [running]:
golang.org/x/crypto/chacha20poly1305.(*chacha20poly1305).seal(0xc0139f9160, 0xc065a41460, 0x2, 0x20, 0xc00a82a1b0, 0xc, 0xc, 0xc065a41460, 0x3, 0x20, ...)
        /home/travis/gopath/pkg/mod/golang.org/x/crypto@v0.0.0-20200820211705-5c72a883971a/chacha20poly1305/chacha20poly1305_amd64.go:59 +0x3a2
golang.org/x/crypto/chacha20poly1305.(*chacha20poly1305).Seal(0xc0139f9160, 0xc065a41460, 0x2, 0x20, 0xc00a82a1b0, 0xc, 0xc, 0xc065a41460, 0x3, 0x20, ...)
        /home/travis/gopath/pkg/mod/golang.org/x/crypto@v0.0.0-20200820211705-5c72a883971a/chacha20poly1305/chacha20poly1305.go:62 +0xf9
github.com/flynn/noise.aeadCipher.Encrypt(0x2606440, 0xc0139f9160, 0x23b7bd8, 0xc065a41460, 0x2, 0x20, 0x1, 0x0, 0x0, 0x0, ...)
        /home/travis/gopath/pkg/mod/github.com/flynn/noise@v0.0.0-20180327030543-2492fe189ae6/cipher_suite.go:184 +0xf5
github.com/flynn/noise.(*CipherState).Encrypt(...)
        /home/travis/gopath/pkg/mod/github.com/flynn/noise@v0.0.0-20180327030543-2492fe189ae6/state.go:36
github.com/libp2p/go-libp2p-noise.(*secureSession).encrypt(0xc004086c80, 0xc065a41460, 0x2, 0x20, 0xc065a41460, 0x3, 0x20, 0x0, 0x1, 0x0, ...)
        /home/travis/gopath/pkg/mod/github.com/libp2p/go-libp2p-noise@v0.1.1/crypto.go:26 +0xc2
github.com/libp2p/go-libp2p-noise.(*secureSession).Write(0xc004086c80, 0xc065a41460, 0x3, 0x20, 0x0, 0x0, 0x0)
        /home/travis/gopath/pkg/mod/github.com/libp2p/go-libp2p-noise@v0.1.1/rw.go:116 +0x203
github.com/libp2p/go-mplex.(*Multiplex).doWriteMsg(0xc017e34100, 0xc065a41460, 0x3, 0x20, 0xbff44a516fa99201, 0x4f137c64a0d28)
        /home/travis/gopath/pkg/mod/github.com/libp2p/go-mplex@v0.1.2/multiplex.go:277 +0x6a
github.com/libp2p/go-mplex.(*Multiplex).handleOutgoing(0xc017e34100)
        /home/travis/gopath/pkg/mod/github.com/libp2p/go-mplex@v0.1.2/multiplex.go:198 +0x138
created by github.com/libp2p/go-mplex.NewMultiplex
        /home/travis/gopath/pkg/mod/github.com/libp2p/go-mplex@v0.1.2/multiplex.go:106 +0x21f

[marten-seemann]

This doesn't look like it's a bug in noise:

go-libp2p-noise/rw.go

Lines 102 to 116 in 02dc2ad

	if total < MaxPlaintextLength {
	cbuf = pool.Get(total + poly1305.TagSize + LengthPrefixLength)
	} else {
	cbuf = pool.Get(MaxTransportMsgLength + LengthPrefixLength)
	}
	defer pool.Put(cbuf)
	for written < total {
	end := written + MaxPlaintextLength
	if end > total {
	end = total
	}
	b, err := s.encrypt(cbuf[:LengthPrefixLength], data[written:end])

The panic occurs because the two slices passed to encrypt in the last line of the code here have some non-zero overlap. As cbuf is a slice we're getting out of the buffer pool, and data is a slice that was also obtained from the same pool (by go-mplex), this is only possible if we continue using a slice from the buffer after we've already but it back.
This will be hard to debug. Might be related to libp2p/go-msgio#22.

Do you have any advice how we can reliably reproduce this crash?

[marten-seemann]

If this actually caused by an incorrect use of the buffer pool, this might be detectable by Go's race detector.
@ridenaio, could you run your node with race detector, and see if it reports any race conditions related to go-buffer-pool?

[Stebalien]

Also ipfs/kubo#6802.

[Stebalien]

This could also be triggered by, e.g, a concurrent read/write on a single stream.