go 1.19 causes different checksums in TestMultistoreSnapshot_Checksum

[tac0turtle]

Summary of Bug

with go 1.19 we witnessed a different hash being produced in TestMultistoreSnapshot_Checksum. We should identify and fix this if possible otherwise report it to the go team.

@odeke-em do you have any ideas?

[alexanderbez]

For ref, here are the release notes for GO 1.19

https://tip.golang.org/doc/go1.19

We should look at Minor changes to the library.

[mmsqe]

seems extra bytes 010000ffff get removed by zlib in 1.19, diff comes from deflate close (1.18 vs 1.19)

[elias-orijtech]

I can confirm @mmsqe's finding. The Go commit that triggers the difference is golang/go@2719b1a. The underlying issue is that snapshots.StreamWriter double-closes the zlib writer, which results in the trailer written twice. For posterity,

// Close implements io.Closer interface
func (sw *StreamWriter) Close() error {
	if err := sw.protoWriter.Close(); err != nil {
		sw.chunkWriter.CloseWithError(err)
		return err
	}
	if err := sw.zWriter.Close(); err != nil {
		sw.chunkWriter.CloseWithError(err)
		return err
	}
	if err := sw.bufWriter.Flush(); err != nil {
		sw.chunkWriter.CloseWithError(err)
		return err
	}
	return sw.chunkWriter.Close()
}

Note that sw.protoWriter.Close() closes the underlying writer,

func (this *varintWriter) Close() error {
	if closer, ok := this.w.(io.Closer); ok {
		return closer.Close()
	}
	return nil
}

too.

A double-close before Go 1.19 would result in the compress/flate.compressor.close being called twice; in Go 1.19 the double close is silently ignored.

In any case, this issue is fixed as a side-effect of #13371.

[julienrbrt]

Thanks for the explanation!

In any case, this issue is fixed as a side-effect of #13371.

TestManager_Take (/snapshots) and TestMultistoreSnapshot_Checksum/Format_1 (store/rootmulti) still fail.
Seems like for the same reason you've explained above.

[mmsqe]

TestManager_Take (/snapshots) and TestMultistoreSnapshot_Checksum/Format_1 (store/rootmulti) still fail. Seems like for the same reason you've explained above.

yes, but data written in 1.18 still can be decompressed correctly in 1.19.