Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s2: Add LZ4 block converter #748

Merged
merged 14 commits into from
Feb 17, 2023
Merged

s2: Add LZ4 block converter #748

merged 14 commits into from
Feb 17, 2023

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented Feb 7, 2023 •
edited
Loading

This allows converting compressed LZ4 blocks to S2 (or snappy) blocks without decompression.

LZ4 -> S2 seems to be same size on average.
LZ4 -> Snappy is usually worse.

Single threaded performance

Speed (excluding LZ4 encoding):

BenchmarkLZ4Converter_ConvertBlock/html-32         	   28237	     42827 ns/op	2390.99 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    2138	    541816 ns/op	1295.80 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  514826	      2328 ns/op	52874.24 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	34821668	        33.48 ns/op	5973.00 MB/s	         2.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  198241	      5975 ns/op	17136.81 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	    7002	    173440 ns/op	2361.63 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	    5940	    196951 ns/op	 772.22 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	    6656	    177228 ns/op	 706.32 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    2355	    510435 ns/op	 836.06 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    1700	    694444 ns/op	 693.88 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   37118	     32141 ns/op	3689.60 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	    6961	    172253 ns/op	1070.05 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	19923691	        59.82 ns/op	2139.66 MB/s	         0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	 3180837	       375.2 ns/op	2665.40 MB/s	        16.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	  184214	      6350 ns/op	1574.70 MB/s	        90.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	   74031	     15521 ns/op	1288.54 MB/s	        -5.000 b_saved	       0 B/op	       0 allocs/op

Assembly speed (amd64)

BenchmarkLZ4Converter_ConvertBlock/html-32         	   47457	     24463 ns/op	4185.89 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    3506	    330277 ns/op	2125.75 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  450177	      2718 ns/op	45294.89 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	76887589	        15.52 ns/op	12887.16 MB/s	         2.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  279540	      4322 ns/op	23694.21 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	   10000	    107485 ns/op	3810.75 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	   10000	    117764 ns/op	1291.47 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	   10000	    100578 ns/op	1244.60 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    3793	    313021 ns/op	1363.34 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    2988	    399888 ns/op	1204.99 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   57486	     19277 ns/op	6151.76 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	   10000	    115641 ns/op	1593.90 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	38400122	        31.21 ns/op	4101.10 MB/s	         0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	 6509028	       179.9 ns/op	5559.38 MB/s	        16.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	  368212	      3244 ns/op	3082.28 MB/s	        90.00 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	  141013	      8303 ns/op	2408.69 MB/s	        -5.000 b_saved	       0 B/op	       0 allocs/op

Reference compression speed:

BenchmarkCompressBlockReference/html/default-32                    14070         82449 ns/op    1241.98 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/urls/default-32                     1215        970890 ns/op     723.14 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg/default-32                    193770          5904 ns/op    20849.30 MB/s          0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/default-32              8297767           144.1 ns/op  1387.77 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pdf/default-32                     94203         12694 ns/op    8066.76 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/html4/default-32                   12174         97969 ns/op    4180.90 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1/default-32                     3613        333851 ns/op     455.56 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt2/default-32                     4683        260579 ns/op     480.39 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt3/default-32                     1268        947209 ns/op     450.54 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt4/default-32                     1083       1097426 ns/op     439.08 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pb/default-32                      18357         64771 ns/op    1830.90 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/gaviota/default-32                  3942        295275 ns/op     624.23 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/default-32            11448295           105.7 ns/op  1210.45 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/default-32            1000000          1021 ns/op     979.26 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/default-32            116739         10114 ns/op     988.68 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/default-32             49216         23409 ns/op     854.39 MB/s           0 B/op          0 allocs/op

BenchmarkCompressBlockReference/html/better-32                      6649        174667 ns/op     586.26 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/urls/better-32                       627       1905706 ns/op     368.41 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg/better-32                      52425         22783 ns/op    5402.88 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/better-32               2772865           433.3 ns/op   461.61 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pdf/better-32                       9210        127051 ns/op     805.97 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/html4/better-32                     5835        201146 ns/op    2036.33 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1/better-32                      2034        566702 ns/op     268.38 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt2/better-32                      2386        500580 ns/op     250.07 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt3/better-32                       758       1556541 ns/op     274.17 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt4/better-32                       591       2013515 ns/op     239.31 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/pb/better-32                        7836        155117 ns/op     764.51 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/gaviota/better-32                   2473        484975 ns/op     380.06 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/better-32              4322678           275.5 ns/op   464.59 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/better-32              468687          2533 ns/op     394.76 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/better-32              49606         23720 ns/op     421.59 MB/s           0 B/op          0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/better-32              14823         81300 ns/op     246.00 MB/s           0 B/op          0 allocs/op

Size comparisons (using Go lz4 encoder):

=== RUN   TestLZ4Converter_ConvertBlock/html
    lz4convert_test.go:42: input size: 102400
    lz4convert_test.go:43: lz4 size: 21195
    lz4convert_test.go:60: lz4->snappy size: 21828
    lz4convert_test.go:79: lz4->s2 size: 20636
    lz4convert_test.go:91: s2 (default) size: 20865
    lz4convert_test.go:95: s2 (better) size: 18969
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 559
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -633
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 330
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 2226
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -229
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1667
    --- PASS: TestLZ4Converter_ConvertBlock/html (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/urls
    lz4convert_test.go:42: input size: 702087
    lz4convert_test.go:43: lz4 size: 292514
    lz4convert_test.go:60: lz4->snappy size: 297926
    lz4convert_test.go:79: lz4->s2 size: 296457
    lz4convert_test.go:91: s2 (default) size: 286538
    lz4convert_test.go:95: s2 (better) size: 248076
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -3943
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -5412
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 5976
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 44438
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 9919
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 48381
    --- PASS: TestLZ4Converter_ConvertBlock/urls (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/jpg
    lz4convert_test.go:42: input size: 123093
    lz4convert_test.go:43: lz4 size: 123522
    lz4convert_test.go:60: lz4->snappy size: 123040
    lz4convert_test.go:79: lz4->s2 size: 123040
    lz4convert_test.go:91: s2 (default) size: 123097
    lz4convert_test.go:95: s2 (better) size: 123097
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 482
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 482
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 425
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 425
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -57
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: -57
    --- PASS: TestLZ4Converter_ConvertBlock/jpg (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/jpg_200b
    lz4convert_test.go:42: input size: 200
    lz4convert_test.go:43: lz4 size: 155
    lz4convert_test.go:60: lz4->snappy size: 153
    lz4convert_test.go:79: lz4->s2 size: 153
    lz4convert_test.go:91: s2 (default) size: 153
    lz4convert_test.go:95: s2 (better) size: 147
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 2
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 2
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 2
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 0
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 6
    --- PASS: TestLZ4Converter_ConvertBlock/jpg_200b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/pdf
    lz4convert_test.go:42: input size: 102400
    lz4convert_test.go:43: lz4 size: 83152
    lz4convert_test.go:60: lz4->snappy size: 83428
    lz4convert_test.go:79: lz4->s2 size: 83016
    lz4convert_test.go:91: s2 (default) size: 84199
    lz4convert_test.go:95: s2 (better) size: 82884
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 136
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -276
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -1047
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 268
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -1183
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 132
    --- PASS: TestLZ4Converter_ConvertBlock/pdf (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/html4
    lz4convert_test.go:42: input size: 409600
    lz4convert_test.go:43: lz4 size: 81908
    lz4convert_test.go:60: lz4->snappy size: 84886
    lz4convert_test.go:79: lz4->s2 size: 80068
    lz4convert_test.go:91: s2 (default) size: 20867
    lz4convert_test.go:95: s2 (better) size: 18979
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 1840
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -2978
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 61041
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 62929
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 59201
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 61089
    --- PASS: TestLZ4Converter_ConvertBlock/html4 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1
    lz4convert_test.go:42: input size: 152089
    lz4convert_test.go:43: lz4 size: 79672
    lz4convert_test.go:60: lz4->snappy size: 79567
    lz4convert_test.go:79: lz4->s2 size: 79566
    lz4convert_test.go:91: s2 (default) size: 85931
    lz4convert_test.go:95: s2 (better) size: 71608
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 105
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -6259
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -6365
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 7958
    --- PASS: TestLZ4Converter_ConvertBlock/txt1 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt2
    lz4convert_test.go:42: input size: 125179
    lz4convert_test.go:43: lz4 size: 70801
    lz4convert_test.go:60: lz4->snappy size: 72231
    lz4convert_test.go:79: lz4->s2 size: 72228
    lz4convert_test.go:91: s2 (default) size: 79572
    lz4convert_test.go:95: s2 (better) size: 65938
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -1427
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -1430
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -8771
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 4863
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -7344
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 6290
    --- PASS: TestLZ4Converter_ConvertBlock/txt2 (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt3
    lz4convert_test.go:42: input size: 426754
    lz4convert_test.go:43: lz4 size: 207038
    lz4convert_test.go:60: lz4->snappy size: 206693
    lz4convert_test.go:79: lz4->s2 size: 206654
    lz4convert_test.go:91: s2 (default) size: 220380
    lz4convert_test.go:95: s2 (better) size: 184936
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 384
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 345
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -13342
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 22102
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -13726
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 21718
    --- PASS: TestLZ4Converter_ConvertBlock/txt3 (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/txt4
    lz4convert_test.go:42: input size: 481861
    lz4convert_test.go:43: lz4 size: 277731
    lz4convert_test.go:60: lz4->snappy size: 286863
    lz4convert_test.go:79: lz4->s2 size: 286856
    lz4convert_test.go:91: s2 (default) size: 318193
    lz4convert_test.go:95: s2 (better) size: 264987
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -9125
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -9132
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -40462
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 12744
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -31337
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 21869
    --- PASS: TestLZ4Converter_ConvertBlock/txt4 (0.01s)
=== RUN   TestLZ4Converter_ConvertBlock/pb
    lz4convert_test.go:42: input size: 118588
    lz4convert_test.go:43: lz4 size: 19003
    lz4convert_test.go:60: lz4->snappy size: 21130
    lz4convert_test.go:79: lz4->s2 size: 19002
    lz4convert_test.go:91: s2 (default) size: 18603
    lz4convert_test.go:95: s2 (better) size: 17686
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 1
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -2127
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 400
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 1317
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 399
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1316
    --- PASS: TestLZ4Converter_ConvertBlock/pb (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/gaviota
    lz4convert_test.go:42: input size: 184320
    lz4convert_test.go:43: lz4 size: 71749
    lz4convert_test.go:60: lz4->snappy size: 63392
    lz4convert_test.go:79: lz4->s2 size: 62446
    lz4convert_test.go:91: s2 (default) size: 65016
    lz4convert_test.go:95: s2 (better) size: 55395
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 9303
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 8357
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 6733
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 16354
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -2570
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 7051
    --- PASS: TestLZ4Converter_ConvertBlock/gaviota (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_128b
    lz4convert_test.go:42: input size: 128
    lz4convert_test.go:43: lz4 size: 84
    lz4convert_test.go:60: lz4->snappy size: 84
    lz4convert_test.go:79: lz4->s2 size: 84
    lz4convert_test.go:91: s2 (default) size: 80
    lz4convert_test.go:95: s2 (better) size: 76
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 0
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 0
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 4
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 8
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 4
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 8
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_128b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_1000b
    lz4convert_test.go:42: input size: 1000
    lz4convert_test.go:43: lz4 size: 807
    lz4convert_test.go:60: lz4->snappy size: 791
    lz4convert_test.go:79: lz4->s2 size: 791
    lz4convert_test.go:91: s2 (default) size: 772
    lz4convert_test.go:95: s2 (better) size: 744
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 16
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 16
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 35
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 63
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: 19
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 47
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_1000b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_10000b
    lz4convert_test.go:42: input size: 10000
    lz4convert_test.go:43: lz4 size: 6969
    lz4convert_test.go:60: lz4->snappy size: 6879
    lz4convert_test.go:79: lz4->s2 size: 6879
    lz4convert_test.go:91: s2 (default) size: 6931
    lz4convert_test.go:95: s2 (better) size: 6216
    lz4convert_test.go:97: lz4 -> s2 bytes saved: 90
    lz4convert_test.go:98: lz4 -> snappy bytes saved: 90
    lz4convert_test.go:99: data -> s2 (default) bytes saved: 38
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 753
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -52
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 663
    --- PASS: TestLZ4Converter_ConvertBlock/txt1_10000b (0.00s)
=== RUN   TestLZ4Converter_ConvertBlock/txt1_20000b
    lz4convert_test.go:42: input size: 20000
    lz4convert_test.go:43: lz4 size: 12750
    lz4convert_test.go:60: lz4->snappy size: 12755
    lz4convert_test.go:79: lz4->s2 size: 12755
    lz4convert_test.go:91: s2 (default) size: 13513
    lz4convert_test.go:95: s2 (better) size: 11489
    lz4convert_test.go:97: lz4 -> s2 bytes saved: -5
    lz4convert_test.go:98: lz4 -> snappy bytes saved: -5
    lz4convert_test.go:99: data -> s2 (default) bytes saved: -763
    lz4convert_test.go:100: data -> s2 (better) bytes saved: 1261
    lz4convert_test.go:101: direct data -> s2 (default) compared to converted from lz4: -758
    lz4convert_test.go:102: direct data -> s2 (better) compared to converted from lz4: 1266

@klauspost
s2: Add LZ4 block converter
fe0c088 
## Single threaded performance

Speed excluding LZ4 encoding:

```
BenchmarkLZ4Converter_ConvertBlock/html-32         	   22450	     49962 ns/op	2049.58 MB/s	       559.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/urls-32         	    1933	    608889 ns/op	1153.06 MB/s	     -3943 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg-32          	  428572	      2816 ns/op	43715.74 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/jpg_200b-32     	  413655	      2793 ns/op	44075.51 MB/s	       482.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pdf-32          	  163200	      7226 ns/op	14171.69 MB/s	       136.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/html4-32        	    5451	    214569 ns/op	1908.94 MB/s	      1840 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1-32         	    5100	    233525 ns/op	 651.27 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt2-32         	    5997	    205362 ns/op	 609.55 MB/s	     -1427 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt3-32         	    1948	    616892 ns/op	 691.78 MB/s	       384.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt4-32         	    1419	    835555 ns/op	 576.70 MB/s	     -9125 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/pb-32           	   29461	     40528 ns/op	2926.08 MB/s	         1.000 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/gaviota-32      	    5454	    214264 ns/op	 860.25 MB/s	      9303 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_128b-32    	    5216	    233042 ns/op	 652.62 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_1000b-32   	    5000	    232401 ns/op	 654.43 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_10000b-32  	    4999	    232989 ns/op	 652.77 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
BenchmarkLZ4Converter_ConvertBlock/txt1_20000b-32  	    4999	    233068 ns/op	 652.55 MB/s	       106.0 b_saved	       0 B/op	       0 allocs/op
```

Reference comnpression speed:

```
BenchmarkCompressBlockReference/html/default-32 	   14392	     82754 ns/op	1237.41 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/html/better-32  	    6663	    178830 ns/op	 572.61 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/urls/default-32 	    1174	   1002967 ns/op	 700.01 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/urls/better-32  	     607	   1942559 ns/op	 361.42 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg/default-32  	  196694	      6023 ns/op	20436.73 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg/better-32   	   51148	     23613 ns/op	5212.85 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/default-32         	  197408	      6015 ns/op	20464.42 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/jpg_200b/better-32          	   51267	     23509 ns/op	5236.04 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pdf/default-32              	   91663	     12891 ns/op	7943.73 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pdf/better-32               	    8433	    128542 ns/op	 796.62 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/html4/default-32            	   10000	    100385 ns/op	4080.29 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/html4/better-32             	    5712	    204505 ns/op	2002.88 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1/default-32             	    3427	    340552 ns/op	 446.60 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1/better-32              	    2032	    576497 ns/op	 263.82 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt2/default-32             	    4359	    266102 ns/op	 470.42 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt2/better-32              	    2306	    509822 ns/op	 245.53 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt3/default-32             	    1219	    971355 ns/op	 439.34 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt3/better-32              	     748	   1598189 ns/op	 267.02 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt4/default-32             	    1056	   1129044 ns/op	 426.79 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt4/better-32              	     579	   2064813 ns/op	 233.37 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pb/default-32               	   17960	     65621 ns/op	1807.16 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/pb/better-32                	    7062	    159908 ns/op	 741.60 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/gaviota/default-32          	    3870	    302586 ns/op	 609.15 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/gaviota/better-32           	    2396	    495081 ns/op	 372.30 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/default-32        	    3426	    342366 ns/op	 444.23 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_128b/better-32         	    2066	    581844 ns/op	 261.39 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/default-32       	    3524	    340442 ns/op	 446.74 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_1000b/better-32        	    1996	    577981 ns/op	 263.14 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/default-32      	    3423	    340195 ns/op	 447.06 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_10000b/better-32       	    2013	    578202 ns/op	 263.04 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/default-32      	    3518	    340101 ns/op	 447.19 MB/s	       0 B/op	       0 allocs/op
BenchmarkCompressBlockReference/txt1_20000b/better-32       	    2031	    581532 ns/op	 261.53 MB/s	       0 B/op	       0 allocs/op
```

Size comparisons (using Go lz4 encoder):

```
=== RUN   TestLZ4Converter_ConvertBlock
=== RUN   TestLZ4Converter_ConvertBlock/html
    lz4convert_test.go:34: input size: 102400
    lz4convert_test.go:35: lz4 size: 21195
    lz4convert_test.go:52: lz4->s2 size: 20636
    lz4convert_test.go:64: s2 (better) size: 18969
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 559
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 2226
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 1667
=== RUN   TestLZ4Converter_ConvertBlock/urls
    lz4convert_test.go:34: input size: 702087
    lz4convert_test.go:35: lz4 size: 292514
    lz4convert_test.go:52: lz4->s2 size: 296457
    lz4convert_test.go:64: s2 (better) size: 248076
    lz4convert_test.go:66: lz4 -> s2 bytes saved: -3943
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 44438
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 48381
=== RUN   TestLZ4Converter_ConvertBlock/jpg
    lz4convert_test.go:34: input size: 123093
    lz4convert_test.go:35: lz4 size: 123522
    lz4convert_test.go:52: lz4->s2 size: 123040
    lz4convert_test.go:64: s2 (better) size: 123097
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 482
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 425
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: -57
=== RUN   TestLZ4Converter_ConvertBlock/jpg_200b
    lz4convert_test.go:34: input size: 123093
    lz4convert_test.go:35: lz4 size: 123522
    lz4convert_test.go:52: lz4->s2 size: 123040
    lz4convert_test.go:64: s2 (better) size: 123097
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 482
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 425
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: -57
=== RUN   TestLZ4Converter_ConvertBlock/pdf
    lz4convert_test.go:34: input size: 102400
    lz4convert_test.go:35: lz4 size: 83152
    lz4convert_test.go:52: lz4->s2 size: 83016
    lz4convert_test.go:64: s2 (better) size: 82884
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 136
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 268
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 132
=== RUN   TestLZ4Converter_ConvertBlock/html4
    lz4convert_test.go:34: input size: 409600
    lz4convert_test.go:35: lz4 size: 81908
    lz4convert_test.go:52: lz4->s2 size: 80068
    lz4convert_test.go:64: s2 (better) size: 18979
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 1840
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 62929
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 61089
=== RUN   TestLZ4Converter_ConvertBlock/txt1
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt2
    lz4convert_test.go:34: input size: 125179
    lz4convert_test.go:35: lz4 size: 70801
    lz4convert_test.go:52: lz4->s2 size: 72228
    lz4convert_test.go:64: s2 (better) size: 65938
    lz4convert_test.go:66: lz4 -> s2 bytes saved: -1427
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 4863
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 6290
=== RUN   TestLZ4Converter_ConvertBlock/txt3
    lz4convert_test.go:34: input size: 426754
    lz4convert_test.go:35: lz4 size: 207038
    lz4convert_test.go:52: lz4->s2 size: 206654
    lz4convert_test.go:64: s2 (better) size: 184936
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 384
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 22102
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 21718
=== RUN   TestLZ4Converter_ConvertBlock/txt4
    lz4convert_test.go:34: input size: 481861
    lz4convert_test.go:35: lz4 size: 277731
    lz4convert_test.go:52: lz4->s2 size: 286856
    lz4convert_test.go:64: s2 (better) size: 264987
    lz4convert_test.go:66: lz4 -> s2 bytes saved: -9125
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 12744
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 21869
=== RUN   TestLZ4Converter_ConvertBlock/pb
    lz4convert_test.go:34: input size: 118588
    lz4convert_test.go:35: lz4 size: 19003
    lz4convert_test.go:52: lz4->s2 size: 19002
    lz4convert_test.go:64: s2 (better) size: 17686
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 1
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 1317
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 1316
=== RUN   TestLZ4Converter_ConvertBlock/gaviota
    lz4convert_test.go:34: input size: 184320
    lz4convert_test.go:35: lz4 size: 71749
    lz4convert_test.go:52: lz4->s2 size: 62446
    lz4convert_test.go:64: s2 (better) size: 55395
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 9303
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 16354
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7051
=== RUN   TestLZ4Converter_ConvertBlock/txt1_128b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt1_1000b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt1_10000b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
=== RUN   TestLZ4Converter_ConvertBlock/txt1_20000b
    lz4convert_test.go:34: input size: 152089
    lz4convert_test.go:35: lz4 size: 79672
    lz4convert_test.go:52: lz4->s2 size: 79566
    lz4convert_test.go:64: s2 (better) size: 71608
    lz4convert_test.go:66: lz4 -> s2 bytes saved: 106
    lz4convert_test.go:67: data -> s2 (better) bytes saved: 8064
    lz4convert_test.go:68: direct data -> s2 better compared to converted from lz4: 7958
```
@klauspost
Tweak tests.
519a51c
@klauspost
Return error when truncated.
b691e88
@klauspost
Improve speed with inlining
b214753
@klauspost
Tweak benchmarks
0751108
@klauspost klauspost force-pushed the add-lz4-block-converter branch from 8ffba47 to 0751108 Compare February 14, 2023 14:49
@klauspost klauspost force-pushed the add-lz4-block-converter branch from 207c000 to 95a0452 Compare February 14, 2023 20:02
@klauspost klauspost merged commit 7e21226 into master Feb 17, 2023
@klauspost klauspost deleted the add-lz4-block-converter branch February 17, 2023 14:43
kodiakhq bot referenced this pull request in cloudquery/filetypes Mar 1, 2023
@cq-bot
fix(deps): Update module github.com/klauspost/compress to v1.16.0 (#84)
a9c9da3 
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/klauspost/compress](https://github.com/klauspost/compress) | indirect | minor | `v1.15.11` -> `v1.16.0` |

---

### ⚠ Dependency Lookup Warnings ⚠

Warnings were logged while processing this repo. Please check the Dependency Dashboard for more information.

---

### Release Notes

<details>
<summary>klauspost/compress</summary>

### [`v1.16.0`](https://github.com/klauspost/compress/releases/tag/v1.16.0)

[Compare Source](https://github.com/klauspost/compress/compare/v1.15.15...v1.16.0)

#### What's Changed

-   s2: Add Dictionary support by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/685](https://github.com/klauspost/compress/pull/685)
-   s2: Add Compression Size Estimate by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/752](https://github.com/klauspost/compress/pull/752)
-   s2: Add support for custom stream encoder by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/755](https://github.com/klauspost/compress/pull/755)
-   s2: Add LZ4 block converter by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/748](https://github.com/klauspost/compress/pull/748)
-   s2: Support io.ReaderAt in ReadSeeker by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/747](https://github.com/klauspost/compress/pull/747)
-   s2c/s2sx: Use concurrent decoding by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/746](https://github.com/klauspost/compress/pull/746)
-   tests: Upgrade to Go 1.20 by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/749](https://github.com/klauspost/compress/pull/749)
-   Update all (command) dependencies by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/758](https://github.com/klauspost/compress/pull/758)

**Full Changelog**: klauspost/compress@v1.15.15...v1.16.0

### [`v1.15.15`](https://github.com/klauspost/compress/releases/tag/v1.15.15)

[Compare Source](https://github.com/klauspost/compress/compare/v1.15.14...v1.15.15)

#### What's Changed

-   zstd: Add delta encoding support by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/728](https://github.com/klauspost/compress/pull/728)
-   huff0: Reduce bounds checking by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/734](https://github.com/klauspost/compress/pull/734)
-   huff0: Assembler improvements by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/736](https://github.com/klauspost/compress/pull/736)
-   deflate: Improve level 7-9 by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/739](https://github.com/klauspost/compress/pull/739)
-   gzhttp: Add SuffixETag() and DropETag() options to prevent ETag collisions on compressed responses by [@&#8203;willbicks](https://github.com/willbicks) in [https://github.com/klauspost/compress/pull/740](https://github.com/klauspost/compress/pull/740)
-   zstd: Don't allocate dataStorage when using byteBuf by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/741](https://github.com/klauspost/compress/pull/741)
-   huff0: Speed up compression of short blocks by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/744](https://github.com/klauspost/compress/pull/744)
-   zstd: Handle dicts by pointer, always by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/743](https://github.com/klauspost/compress/pull/743)
-   fse: Optimize compression by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/745](https://github.com/klauspost/compress/pull/745)
-   Retract v1.14.1-v.1.14.3 by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/742](https://github.com/klauspost/compress/pull/742)

#### New Contributors

-   [@&#8203;willbicks](https://github.com/willbicks) made their first contribution in [https://github.com/klauspost/compress/pull/740](https://github.com/klauspost/compress/pull/740)

**Full Changelog**: klauspost/compress@v1.15.14...v1.15.15

### [`v1.15.14`](https://github.com/klauspost/compress/releases/tag/v1.15.14)

[Compare Source](https://github.com/klauspost/compress/compare/v1.15.13...v1.15.14)

#### What's Changed

-   flate: Improve speed in big stateless blocks. by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/718](https://github.com/klauspost/compress/pull/718)
-   zstd: Trigger BCE by switching on lengths by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/716](https://github.com/klauspost/compress/pull/716)
-   zstd: Shave some instructions off the amd64 asm by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/720](https://github.com/klauspost/compress/pull/720)
-   export NoGzipResponseWriter for custom ResponseWriter wrappers by [@&#8203;harshavardhana](https://github.com/harshavardhana) in [https://github.com/klauspost/compress/pull/722](https://github.com/klauspost/compress/pull/722)
-   s2: Add example for indexing and existing stream by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/723](https://github.com/klauspost/compress/pull/723)
-   tests: Tweak fuzz tests by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/719](https://github.com/klauspost/compress/pull/719)

#### New Contributors

-   [@&#8203;harshavardhana](https://github.com/harshavardhana) made their first contribution in [https://github.com/klauspost/compress/pull/722](https://github.com/klauspost/compress/pull/722)

**Full Changelog**: klauspost/compress@v1.15.13...v1.15.14

### [`v1.15.13`](https://github.com/klauspost/compress/releases/tag/v1.15.13)

[Compare Source](https://github.com/klauspost/compress/compare/v1.15.12...v1.15.13)

#### What's Changed

-   zstd: Add MaxEncodedSize to encoder by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/691](https://github.com/klauspost/compress/pull/691)
-   zstd: Improve "best" end search by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/693](https://github.com/klauspost/compress/pull/693)
-   zstd: Replace bytes.Equal with smaller comparisons by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/695](https://github.com/klauspost/compress/pull/695)
-   zstd: Faster CRC checking/skipping by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/696](https://github.com/klauspost/compress/pull/696)
-   zstd: Rewrite matchLen to make it inlineable by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/701](https://github.com/klauspost/compress/pull/701)
-   zstd: Write table clearing in a way that the compiler recognizes by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/702](https://github.com/klauspost/compress/pull/702)
-   zstd: Use individual reset threshold by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/703](https://github.com/klauspost/compress/pull/703)
-   huff0: Check for zeros earlier in Scratch.countSimple by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/704](https://github.com/klauspost/compress/pull/704)
-   zstd: Improve best compression's match selection by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/705](https://github.com/klauspost/compress/pull/705)
-   zstd: Select best match using selection trees by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/706](https://github.com/klauspost/compress/pull/706)
-   zstd: sync xxhash with final accepted patch upstream by [@&#8203;lizthegrey](https://github.com/lizthegrey) in [https://github.com/klauspost/compress/pull/707](https://github.com/klauspost/compress/pull/707)
-   zstd: Import xxhash v2.2.0 by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/708](https://github.com/klauspost/compress/pull/708)

**Full Changelog**: klauspost/compress@v1.15.12...v1.15.13

### [`v1.15.12`](https://github.com/klauspost/compress/releases/tag/v1.15.12)

[Compare Source](https://github.com/klauspost/compress/compare/v1.15.11...v1.15.12)

#### What's Changed

-   zstd: Tweak decoder allocs. by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/680](https://github.com/klauspost/compress/pull/680)
-   gzhttp: Always delete `HeaderNoCompression` by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/683](https://github.com/klauspost/compress/pull/683)

**Full Changelog**: klauspost/compress@v1.15.11...v1.15.12

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 3am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xMDkuMSIsInVwZGF0ZWRJblZlciI6IjM0LjE1NC4wIn0=-->
kodiakhq bot referenced this pull request in cloudquery/plugin-pb-go Aug 1, 2023
@cq-bot
fix(deps): Update module github.com/klauspost/compress to v1.16.7 (#82)
12e9d67 
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github.com/klauspost/compress](https://github.com/klauspost/compress) | indirect | minor | `v1.15.15` -> `v1.16.7` |

---

### Release Notes

<details>
<summary>klauspost/compress (github.com/klauspost/compress)</summary>

### [`v1.16.7`](https://github.com/klauspost/compress/releases/tag/v1.16.7)

[Compare Source](https://github.com/klauspost/compress/compare/v1.16.6...v1.16.7)

#### What's Changed

-   zstd: Fix default level first dictionary encode by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/829](https://github.com/klauspost/compress/pull/829)
-   docs: Fix typo in security advisory URL by [@&#8203;vcabbage](https://github.com/vcabbage) in [https://github.com/klauspost/compress/pull/830](https://github.com/klauspost/compress/pull/830)
-   s2: add GetBufferCapacity() method by [@&#8203;GiedriusS](https://github.com/GiedriusS) in [https://github.com/klauspost/compress/pull/832](https://github.com/klauspost/compress/pull/832)

#### New Contributors

-   [@&#8203;vcabbage](https://github.com/vcabbage) made their first contribution in [https://github.com/klauspost/compress/pull/830](https://github.com/klauspost/compress/pull/830)
-   [@&#8203;GiedriusS](https://github.com/GiedriusS) made their first contribution in [https://github.com/klauspost/compress/pull/832](https://github.com/klauspost/compress/pull/832)

**Full Changelog**: klauspost/compress@v1.16.6...v1.16.7

### [`v1.16.6`](https://github.com/klauspost/compress/releases/tag/v1.16.6)

[Compare Source](https://github.com/klauspost/compress/compare/v1.16.5...v1.16.6)

#### What's Changed

-   zstd: correctly ignore WithEncoderPadding(1) by [@&#8203;ianlancetaylor](https://github.com/ianlancetaylor) in [https://github.com/klauspost/compress/pull/806](https://github.com/klauspost/compress/pull/806)
-   gzhttp: Handle informational headers by [@&#8203;rtribotte](https://github.com/rtribotte) in [https://github.com/klauspost/compress/pull/815](https://github.com/klauspost/compress/pull/815)
-   zstd: Add amd64 match length assembly by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/824](https://github.com/klauspost/compress/pull/824)
-   s2: Improve Better compression slightly by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/663](https://github.com/klauspost/compress/pull/663)
-   s2: Clean up matchlen assembly by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/825](https://github.com/klauspost/compress/pull/825)

#### New Contributors

-   [@&#8203;rtribotte](https://github.com/rtribotte) made their first contribution in [https://github.com/klauspost/compress/pull/815](https://github.com/klauspost/compress/pull/815)
-   [@&#8203;dveeden](https://github.com/dveeden) made their first contribution in [https://github.com/klauspost/compress/pull/816](https://github.com/klauspost/compress/pull/816)

**Full Changelog**: klauspost/compress@v1.16.5...v1.16.6

### [`v1.16.5`](https://github.com/klauspost/compress/releases/tag/v1.16.5)

[Compare Source](https://github.com/klauspost/compress/compare/v1.16.4...v1.16.5)

#### What's Changed

-   zstd: readByte needs to use io.ReadFull by [@&#8203;jnoxon](https://github.com/jnoxon) in [https://github.com/klauspost/compress/pull/802](https://github.com/klauspost/compress/pull/802)
-   gzip: Fix WriterTo after initial read by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/804](https://github.com/klauspost/compress/pull/804)

#### New Contributors

-   [@&#8203;jnoxon](https://github.com/jnoxon) made their first contribution in [https://github.com/klauspost/compress/pull/802](https://github.com/klauspost/compress/pull/802)

**Full Changelog**: klauspost/compress@v1.16.4...v1.16.5

### [`v1.16.4`](https://github.com/klauspost/compress/releases/tag/v1.16.4)

[Compare Source](https://github.com/klauspost/compress/compare/v1.16.3...v1.16.4)

#### What's Changed

-   s2: Fix huge block overflow by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/779](https://github.com/klauspost/compress/pull/779)
-   s2: Allow CustomEncoder fallback by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/780](https://github.com/klauspost/compress/pull/780)
-   zstd: Fix amd64 not always detecting corrupt data by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/785](https://github.com/klauspost/compress/pull/785)
-   zstd: Improve zstd best efficiency by [@&#8203;klauspost](https://github.com/klauspost) and [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/784](https://github.com/klauspost/compress/pull/784)
-   zstd: Make load(32|64)32 safer and smaller by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/788](https://github.com/klauspost/compress/pull/788)
-   zstd: Fix quick reject on long backmatches by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/787](https://github.com/klauspost/compress/pull/787)
-   zstd: Revert table size change  by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/789](https://github.com/klauspost/compress/pull/789)
-   zstd: Respect WithAllLitEntropyCompression by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/792](https://github.com/klauspost/compress/pull/792)
-   zstd: Fix back-referenced offset by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/793](https://github.com/klauspost/compress/pull/793)
-   zstd: Load source value at start of loop by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/794](https://github.com/klauspost/compress/pull/794)
-   zstd: Shorten checksum code by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/795](https://github.com/klauspost/compress/pull/795)
-   zstd: Fix fallback on incompressible block by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/798](https://github.com/klauspost/compress/pull/798)
-   gzhttp: Suppport ResponseWriter Unwrap() in gzhttp handler by [@&#8203;jgimenez](https://github.com/jgimenez) in [https://github.com/klauspost/compress/pull/799](https://github.com/klauspost/compress/pull/799)

#### New Contributors

-   [@&#8203;jgimenez](https://github.com/jgimenez) made their first contribution in [https://github.com/klauspost/compress/pull/799](https://github.com/klauspost/compress/pull/799)

**Full Changelog**: klauspost/compress@v1.16.3...v1.16.4

### [`v1.16.3`](https://github.com/klauspost/compress/releases/tag/v1.16.3)

[Compare Source](https://github.com/klauspost/compress/compare/v1.16.2...v1.16.3)

**Full Changelog**: klauspost/compress@v1.16.2...v1.16.3

### [`v1.16.2`](https://github.com/klauspost/compress/releases/tag/v1.16.2)

[Compare Source](https://github.com/klauspost/compress/compare/v1.16.1...v1.16.2)

#### What's Changed

-   Fix Goreleaser permissions by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/777](https://github.com/klauspost/compress/pull/777)

**Full Changelog**: klauspost/compress@v1.16.1...v1.16.2

### [`v1.16.1`](https://github.com/klauspost/compress/releases/tag/v1.16.1)

[Compare Source](https://github.com/klauspost/compress/compare/v1.16.0...v1.16.1)

#### What's Changed

-   zstd: Speed up + improve best encoder by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/776](https://github.com/klauspost/compress/pull/776)
-   s2: Add Intel LZ4s converter by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/766](https://github.com/klauspost/compress/pull/766)
-   gzhttp: Add BREACH mitigation by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/762](https://github.com/klauspost/compress/pull/762)
-   gzhttp: Remove a few unneeded allocs by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/768](https://github.com/klauspost/compress/pull/768)
-   gzhttp: Fix crypto/rand.Read usage by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/770](https://github.com/klauspost/compress/pull/770)
-   gzhttp: Use SHA256 as paranoid option by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/769](https://github.com/klauspost/compress/pull/769)
-   gzhttp: Use strings for randomJitter to skip a copy by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/767](https://github.com/klauspost/compress/pull/767)
-   zstd: Fix ineffective block size check by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/771](https://github.com/klauspost/compress/pull/771)
-   zstd: Check FSE init values by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/772](https://github.com/klauspost/compress/pull/772)
-   zstd: Report EOF from byteBuf.readBig by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/773](https://github.com/klauspost/compress/pull/773)
-   huff0: Speed up compress1xDo by [@&#8203;greatroar](https://github.com/greatroar) in [https://github.com/klauspost/compress/pull/774](https://github.com/klauspost/compress/pull/774)
-   tests: Remove fuzz printing by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/775](https://github.com/klauspost/compress/pull/775)
-   tests: Add CICD Fuzz testing by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/763](https://github.com/klauspost/compress/pull/763)
-   ci: set minimal permissions to GitHub Workflows by [@&#8203;diogoteles08](https://github.com/diogoteles08) in [https://github.com/klauspost/compress/pull/765](https://github.com/klauspost/compress/pull/765)

#### New Contributors

-   [@&#8203;diogoteles08](https://github.com/diogoteles08) made their first contribution in [https://github.com/klauspost/compress/pull/765](https://github.com/klauspost/compress/pull/765)

**Full Changelog**: klauspost/compress@v1.16.0...v1.16.1

### [`v1.16.0`](https://github.com/klauspost/compress/releases/tag/v1.16.0)

[Compare Source](https://github.com/klauspost/compress/compare/v1.15.15...v1.16.0)

#### What's Changed

-   s2: Add Dictionary support by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/685](https://github.com/klauspost/compress/pull/685)
-   s2: Add Compression Size Estimate by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/752](https://github.com/klauspost/compress/pull/752)
-   s2: Add support for custom stream encoder by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/755](https://github.com/klauspost/compress/pull/755)
-   s2: Add LZ4 block converter by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/748](https://github.com/klauspost/compress/pull/748)
-   s2: Support io.ReaderAt in ReadSeeker by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/747](https://github.com/klauspost/compress/pull/747)
-   s2c/s2sx: Use concurrent decoding by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/746](https://github.com/klauspost/compress/pull/746)
-   tests: Upgrade to Go 1.20 by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/749](https://github.com/klauspost/compress/pull/749)
-   Update all (command) dependencies by [@&#8203;klauspost](https://github.com/klauspost) in [https://github.com/klauspost/compress/pull/758](https://github.com/klauspost/compress/pull/758)

**Full Changelog**: klauspost/compress@v1.15.15...v1.16.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi4yNi4xIiwidXBkYXRlZEluVmVyIjoiMzYuMjYuMSIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@klauspost