Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Document that read::GzDecoder consumes bytes after end of gzip #367

Merged
merged 1 commit into from
Aug 8, 2023
Merged

Document that read::GzDecoder consumes bytes after end of gzip #367

merged 1 commit into from
Aug 8, 2023

Conversation

jongiddy
Copy link
Contributor

@jongiddy jongiddy commented Jul 30, 2023 •
edited
Loading

Add tests showing that the GzDecoders in bufread and write support reading immediately after end of gzip data.

read::GzDecoder can read past the end of gzip data, and a subsequent read of the inner type loses those bytes.

@jongiddy jongiddy force-pushed the note-read-loss branch 3 times, most recently from cee4538 to 5328174 Compare July 30, 2023 11:55
@jongiddy jongiddy mentioned this pull request Jul 30, 2023
Open
Byron
Byron approved these changes Jul 31, 2023
View reviewed changes
Copy link
Member

@Byron Byron left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot codifying the subtleties that thus far where only kept in issues, while improving the documentation to help users avoid those pitfalls. It's much appreciated!

I have made a few tweaks and believe this PR is good to go.

@jongiddy Could you ping me here in about a week's time so I can hit the merge button? Maybe there is other reviews during that period. (I will also try to get back here, but maybe I miss the mark)

@jongiddy @Byron
Document that read::GzDecoder consumes bytes after end of gzip
b2079e3 
Add tests showing that the `GzDecoder`s in `bufread` and `write`
support reading immediately after end of gzip data.

Co-authored-by: Sebastian Thiel <sebastian.thiel@icloud.com>
@jongiddy
Copy link
Contributor Author

jongiddy commented Aug 1, 2023

I added a note to the read module describing the behaviour of read decoders and how to get the typically expected behaviour.

Byron
Byron reviewed Aug 1, 2023
View reviewed changes
src/lib.rs
/// data while decoding. If the caller requires subsequent reads to start
/// immediately following the compressed data wrap the `Read` type in a
/// [`BufReader`] and use the `BufReader` with the equivalent decoder from the
/// `bufread` module and also for the subsequent reads.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, I think that's a good, which naturally covers all implementations.

@jongiddy jongiddy mentioned this pull request Aug 5, 2023
Open
@Byron Byron merged commit 7f96363 into rust-lang:main Aug 8, 2023
10 checks passed
@jongiddy jongiddy deleted the note-read-loss branch August 8, 2023 06:44
bors added a commit to rust-lang/cargo that referenced this pull request Sep 1, 2023
@bors
Auto merge of #12609 - rust-lang:renovate/compatible, r=epage
774161b 
chore(deps): update compatible

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [anyhow](https://togithub.com/dtolnay/anyhow) | workspace.dependencies | patch | `1.0.72` -> `1.0.75` |
| [base64](https://togithub.com/marshallpierce/rust-base64) | workspace.dependencies | patch | `0.21.2` -> `0.21.3` |
| [bytesize](https://togithub.com/hyunsik/bytesize) | workspace.dependencies | minor | `1.2` -> `1.3` |
| [clap](https://togithub.com/clap-rs/clap) | workspace.dependencies | minor | `4.3.23` -> `4.4.2` |
| [filetime](https://togithub.com/alexcrichton/filetime) | workspace.dependencies | patch | `0.2.21` -> `0.2.22` |
| [flate2](https://togithub.com/rust-lang/flate2-rs) | workspace.dependencies | patch | `1.0.26` -> `1.0.27` |
| [memchr](https://togithub.com/BurntSushi/memchr) | workspace.dependencies | minor | `2.5.0` -> `2.6.2` |
| [openssl](https://togithub.com/sfackler/rust-openssl) | workspace.dependencies | patch | `0.10.55` -> `0.10.57` |
| [serde-untagged](https://togithub.com/dtolnay/serde-untagged) | workspace.dependencies | patch | `0.1.0` -> `0.1.1` |
| [serde_json](https://togithub.com/serde-rs/json) | workspace.dependencies | patch | `1.0.104` -> `1.0.105` |
| [snapbox](https://togithub.com/assert-rs/trycmd/tree/main/crates/snapbox) ([source](https://togithub.com/assert-rs/trycmd)) | workspace.dependencies | patch | `0.4.11` -> `0.4.12` |
| [syn](https://togithub.com/dtolnay/syn) | workspace.dependencies | patch | `2.0.28` -> `2.0.29` |
| [tar](https://togithub.com/alexcrichton/tar-rs) | workspace.dependencies | patch | `0.4.39` -> `0.4.40` |
| [tempfile](https://stebalien.com/projects/tempfile-rs/) ([source](https://togithub.com/Stebalien/tempfile)) | workspace.dependencies | minor | `3.7.0` -> `3.8.0` |
| [thiserror](https://togithub.com/dtolnay/thiserror) | workspace.dependencies | patch | `1.0.44` -> `1.0.47` |
| [unicase](https://togithub.com/seanmonstar/unicase) | workspace.dependencies | minor | `2.6.0` -> `2.7.0` |
| [url](https://togithub.com/servo/rust-url) | workspace.dependencies | patch | `2.4.0` -> `2.4.1` |

---

### Release Notes

<details>
<summary>dtolnay/anyhow (anyhow)</summary>

### [`v1.0.75`](https://togithub.com/dtolnay/anyhow/releases/tag/1.0.75)

[Compare Source](https://togithub.com/dtolnay/anyhow/compare/1.0.74...1.0.75)

-   Partially work around rust-analyzer bug ([rust-lang/rust-analyzer#9911)

### [`v1.0.74`](https://togithub.com/dtolnay/anyhow/releases/tag/1.0.74)

[Compare Source](https://togithub.com/dtolnay/anyhow/compare/1.0.73...1.0.74)

-   Add bootstrap workaround to allow rustc to depend on anyhow ([#&#8203;320](https://togithub.com/dtolnay/anyhow/issues/320), thanks [`@&#8203;RalfJung](https://togithub.com/RalfJung))`

### [`v1.0.73`](https://togithub.com/dtolnay/anyhow/releases/tag/1.0.73)

[Compare Source](https://togithub.com/dtolnay/anyhow/compare/1.0.72...1.0.73)

-   Update backtrace support to nightly's new Error::provide API ([rust-lang/rust#113464, [#&#8203;319](https://togithub.com/dtolnay/anyhow/issues/319))

</details>

<details>
<summary>marshallpierce/rust-base64 (base64)</summary>

### [`v0.21.3`](https://togithub.com/marshallpierce/rust-base64/blob/HEAD/RELEASE-NOTES.md#0213)

[Compare Source](https://togithub.com/marshallpierce/rust-base64/compare/v0.21.2...v0.21.3)

-   Implement `source` instead of `cause` on Error types
-   Roll back MSRV to 1.48.0 so Debian can continue to live in a time warp
-   Slightly faster chunked encoding for short inputs
-   Decrease binary size

</details>

<details>
<summary>hyunsik/bytesize (bytesize)</summary>

### [`v1.3.0`](https://togithub.com/hyunsik/bytesize/releases/tag/v1.3.0): Release 1.3.0

[Compare Source](https://togithub.com/hyunsik/bytesize/compare/v1.2.0...v1.3.0)

#### Changes

-   Improved performance by eliminating String creation by utilizing the original \&str slice [#&#8203;31](https://togithub.com/hyunsik/bytesize/issues/31) ([`@&#8203;ChanTsune](https://togithub.com/ChanTsune))`

</details>

<details>
<summary>clap-rs/clap (clap)</summary>

### [`v4.4.2`](https://togithub.com/clap-rs/clap/blob/HEAD/CHANGELOG.md#442---2023-08-31)

[Compare Source](https://togithub.com/clap-rs/clap/compare/v4.4.1...v4.4.2)

##### Performance

-   Improve build times by removing `once_cell` dependency

### [`v4.4.1`](https://togithub.com/clap-rs/clap/blob/HEAD/CHANGELOG.md#441---2023-08-28)

[Compare Source](https://togithub.com/clap-rs/clap/compare/v4.4.0...v4.4.1)

##### Features

-   Stabilize `Command::styles`

### [`v4.4.0`](https://togithub.com/clap-rs/clap/blob/HEAD/CHANGELOG.md#440---2023-08-24)

[Compare Source](https://togithub.com/clap-rs/clap/compare/v4.3.24...v4.4.0)

##### Compatibility

-   Update MSRV to 1.70.0

### [`v4.3.24`](https://togithub.com/clap-rs/clap/blob/HEAD/CHANGELOG.md#4324---2023-08-23)

[Compare Source](https://togithub.com/clap-rs/clap/compare/v4.3.23...v4.3.24)

##### Fixes

-   Ensure column padding is preserved in `--help` with custom templates

</details>

<details>
<summary>alexcrichton/filetime (filetime)</summary>

### [`v0.2.22`](https://togithub.com/alexcrichton/filetime/compare/0.2.21...0.2.22)

[Compare Source](https://togithub.com/alexcrichton/filetime/compare/0.2.21...0.2.22)

</details>

<details>
<summary>rust-lang/flate2-rs (flate2)</summary>

### [`v1.0.27`](https://togithub.com/rust-lang/flate2-rs/releases/tag/1.0.27)

[Compare Source](https://togithub.com/rust-lang/flate2-rs/compare/1.0.26...1.0.27)

#### What's Changed

-   Move GzHeader into GzState by [`@&#8203;jongiddy](https://togithub.com/jongiddy)` in [rust-lang/flate2-rs#344
-   Move blocked_partial_header_read test to read module by [`@&#8203;jongiddy](https://togithub.com/jongiddy)` in [rust-lang/flate2-rs#345
-   Move gzip header parsing out of bufread module by [`@&#8203;jongiddy](https://togithub.com/jongiddy)` in [rust-lang/flate2-rs#346
-   Fix a comment on the `Compression` struct by [`@&#8203;JohnTitor](https://togithub.com/JohnTitor)` in [rust-lang/flate2-rs#351
-   Add notes about multiple streams to `GzDecoder` by [`@&#8203;JohnTitor](https://togithub.com/JohnTitor)` in [rust-lang/flate2-rs#347
-   better error message when compiling with `--no-default-features` or `default-features = false` by [`@&#8203;Byron](https://togithub.com/Byron)` in [rust-lang/flate2-rs#360
-   Fix Read encoder examples by [`@&#8203;markgoddard](https://togithub.com/markgoddard)` in [rust-lang/flate2-rs#356
-   Add CIFuzz Github action by [`@&#8203;DavidKorczynski](https://togithub.com/DavidKorczynski)` in [rust-lang/flate2-rs#326
-   Fix GzDecoder Write partial filenames and comments by [`@&#8203;jongiddy](https://togithub.com/jongiddy)` in [rust-lang/flate2-rs#323
-   Fix header CRC calculation of trailing zeros by [`@&#8203;jongiddy](https://togithub.com/jongiddy)` in [rust-lang/flate2-rs#363
-   Fix broken link on README.md by [`@&#8203;wcampbell0x2a](https://togithub.com/wcampbell0x2a)` in [rust-lang/flate2-rs#366
-   Recommend MultiGzDecoder over GzDecoder in docs by [`@&#8203;jsha](https://togithub.com/jsha)` in [rust-lang/flate2-rs#324
-   Add functionality for custom (de)compress instances by [`@&#8203;PierreV23](https://togithub.com/PierreV23)` in [rust-lang/flate2-rs#361
-   Add maintenance document by [`@&#8203;Byron](https://togithub.com/Byron)` in [rust-lang/flate2-rs#362
-   Document that `read::GzDecoder` consumes bytes after end of gzip by [`@&#8203;jongiddy](https://togithub.com/jongiddy)` in [rust-lang/flate2-rs#367
-   prepare 1.0.27 release by [`@&#8203;Byron](https://togithub.com/Byron)` in [rust-lang/flate2-rs#369

#### New Contributors

-   [`@&#8203;Byron](https://togithub.com/Byron)` made their first contribution in [rust-lang/flate2-rs#360
-   [`@&#8203;markgoddard](https://togithub.com/markgoddard)` made their first contribution in [rust-lang/flate2-rs#356
-   [`@&#8203;jsha](https://togithub.com/jsha)` made their first contribution in [rust-lang/flate2-rs#324
-   [`@&#8203;PierreV23](https://togithub.com/PierreV23)` made their first contribution in [rust-lang/flate2-rs#361

**Full Changelog**: rust-lang/flate2-rs@1.0.26...1.0.27

</details>

<details>
<summary>BurntSushi/memchr (memchr)</summary>

### [`v2.6.2`](https://togithub.com/BurntSushi/memchr/compare/2.6.1...2.6.2)

[Compare Source](https://togithub.com/BurntSushi/memchr/compare/2.6.1...2.6.2)

### [`v2.6.1`](https://togithub.com/BurntSushi/memchr/compare/2.6.0...2.6.1)

[Compare Source](https://togithub.com/BurntSushi/memchr/compare/2.6.0...2.6.1)

### [`v2.6.0`](https://togithub.com/BurntSushi/memchr/compare/2.5.0...2.6.0)

[Compare Source](https://togithub.com/BurntSushi/memchr/compare/2.5.0...2.6.0)

</details>

<details>
<summary>sfackler/rust-openssl (openssl)</summary>

### [`v0.10.57`](https://togithub.com/sfackler/rust-openssl/releases/tag/openssl-v0.10.57)

[Compare Source](https://togithub.com/sfackler/rust-openssl/compare/openssl-v0.10.56...openssl-v0.10.57)

#### What's Changed

-   Expose chacha20\_poly1305 on LibreSSL by [`@&#8203;alex](https://togithub.com/alex)` in [sfackler/rust-openssl#2011
-   Add openssl::cipher_ctx::CipherCtx::clone by [`@&#8203;johntyner](https://togithub.com/johntyner)` in [sfackler/rust-openssl#2017
-   Add X509VerifyParam::set_email by [`@&#8203;dhouck](https://togithub.com/dhouck)` in [sfackler/rust-openssl#2018
-   Add perl-FindBin dep for fedora by [`@&#8203;JadedBlueEyes](https://togithub.com/JadedBlueEyes)` in [sfackler/rust-openssl#2023
-   Update to bitflags 2.2.1. by [`@&#8203;qwandor](https://togithub.com/qwandor)` in [sfackler/rust-openssl#1906
-   Release openssl v0.10.57 and openssl-sys v0.9.92 by [`@&#8203;alex](https://togithub.com/alex)` in [sfackler/rust-openssl#2025

#### New Contributors

-   [`@&#8203;johntyner](https://togithub.com/johntyner)` made their first contribution in [sfackler/rust-openssl#2017
-   [`@&#8203;dhouck](https://togithub.com/dhouck)` made their first contribution in [sfackler/rust-openssl#2018
-   [`@&#8203;JadedBlueEyes](https://togithub.com/JadedBlueEyes)` made their first contribution in [sfackler/rust-openssl#2023
-   [`@&#8203;qwandor](https://togithub.com/qwandor)` made their first contribution in [sfackler/rust-openssl#1906

**Full Changelog**: sfackler/rust-openssl@openssl-v0.10.56...openssl-v0.10.57

### [`v0.10.56`](https://togithub.com/sfackler/rust-openssl/releases/tag/openssl-v0.10.56): openssl v0.10.56

[Compare Source](https://togithub.com/sfackler/rust-openssl/compare/openssl-v0.10.55...openssl-v0.10.56)

</details>

<details>
<summary>dtolnay/serde-untagged (serde-untagged)</summary>

### [`v0.1.1`](https://togithub.com/dtolnay/serde-untagged/compare/0.1.0...0.1.1)

[Compare Source](https://togithub.com/dtolnay/serde-untagged/compare/0.1.0...0.1.1)

</details>

<details>
<summary>serde-rs/json (serde_json)</summary>

### [`v1.0.105`](https://togithub.com/serde-rs/json/releases/tag/v1.0.105)

[Compare Source](https://togithub.com/serde-rs/json/compare/v1.0.104...v1.0.105)

-   Support bool in map keys ([#&#8203;1054](https://togithub.com/serde-rs/json/issues/1054))

</details>

<details>
<summary>assert-rs/trycmd (snapbox)</summary>

### [`v0.4.12`](https://togithub.com/assert-rs/trycmd/compare/snapbox-v0.4.11...snapbox-v0.4.12)

[Compare Source](https://togithub.com/assert-rs/trycmd/compare/snapbox-v0.4.11...snapbox-v0.4.12)

</details>

<details>
<summary>dtolnay/syn (syn)</summary>

### [`v2.0.29`](https://togithub.com/dtolnay/syn/releases/tag/2.0.29)

[Compare Source](https://togithub.com/dtolnay/syn/compare/2.0.28...2.0.29)

-   Partially work around rust-analyzer bug ([rust-lang/rust-analyzer#9911)

</details>

<details>
<summary>alexcrichton/tar-rs (tar)</summary>

### [`v0.4.40`](https://togithub.com/alexcrichton/tar-rs/compare/0.4.39...0.4.40)

[Compare Source](https://togithub.com/alexcrichton/tar-rs/compare/0.4.39...0.4.40)

</details>

<details>
<summary>Stebalien/tempfile (tempfile)</summary>

### [`v3.8.0`](https://togithub.com/Stebalien/tempfile/blob/HEAD/CHANGELOG.md#380)

[Compare Source](https://togithub.com/Stebalien/tempfile/compare/v3.7.1...v3.8.0)

-   Added `with_prefix` and `with_prefix_in` to `TempDir` and `NamedTempFile` to make it easier to create temporary files/directories with nice prefixes.
-   Misc cleanups.

### [`v3.7.1`](https://togithub.com/Stebalien/tempfile/blob/HEAD/CHANGELOG.md#371)

[Compare Source](https://togithub.com/Stebalien/tempfile/compare/v3.7.0...v3.7.1)

-   Tempfile builds on haiku again.
-   Under the hood, we've switched from the unlinkat/linkat syscalls to the regular unlink/link syscalls where possible.

</details>

<details>
<summary>dtolnay/thiserror (thiserror)</summary>

### [`v1.0.47`](https://togithub.com/dtolnay/thiserror/releases/tag/1.0.47)

[Compare Source](https://togithub.com/dtolnay/thiserror/compare/1.0.46...1.0.47)

-   Work around rust-analyzer bug ([rust-lang/rust-analyzer#9911)

### [`v1.0.46`](https://togithub.com/dtolnay/thiserror/releases/tag/1.0.46)

[Compare Source](https://togithub.com/dtolnay/thiserror/compare/1.0.45...1.0.46)

-   Add bootstrap workaround to allow rustc to depend on thiserror ([#&#8203;248](https://togithub.com/dtolnay/thiserror/issues/248), thanks [`@&#8203;RalfJung](https://togithub.com/RalfJung))`

### [`v1.0.45`](https://togithub.com/dtolnay/thiserror/releases/tag/1.0.45)

[Compare Source](https://togithub.com/dtolnay/thiserror/compare/1.0.44...1.0.45)

-   Update backtrace support to nightly's new Error::provide API ([rust-lang/rust#113464, [#&#8203;246](https://togithub.com/dtolnay/thiserror/issues/246))

</details>

<details>
<summary>seanmonstar/unicase (unicase)</summary>

### [`v2.7.0`](https://togithub.com/seanmonstar/unicase/releases/tag/v2.7.0)

[Compare Source](https://togithub.com/seanmonstar/unicase/compare/v2.6.0...v2.7.0)

#### What's Changed

-   Update to Unicode 15.0.0 by [`@&#8203;seanmonstar](https://togithub.com/seanmonstar)` in [seanmonstar/unicase#59

</details>

<details>
<summary>servo/rust-url (url)</summary>

### [`v2.4.1`](https://togithub.com/servo/rust-url/releases/tag/v2.4.1)

[Compare Source](https://togithub.com/servo/rust-url/compare/v2.4.0...v2.4.1)

##### What's Changed

-   Move debugger_visualizer tests to separate crate by [`@&#8203;lucacasonato](https://togithub.com/lucacasonato)` in [servo/rust-url#853
-   Remove obsolete badge references by [`@&#8203;atouchet](https://togithub.com/atouchet)` in [servo/rust-url#852
-   Fix trailing spaces in scheme / pathname / search setters by [`@&#8203;lucacasonato](https://togithub.com/lucacasonato)` in [servo/rust-url#848
-   fix: implement std::error::Error for data-url by [`@&#8203;lucacasonato](https://togithub.com/lucacasonato)` in [servo/rust-url#698
-   Enable the GitHub merge queue by [`@&#8203;mrobinson](https://togithub.com/mrobinson)` in [servo/rust-url#851
-   Rewrite WPT runner by [`@&#8203;lucacasonato](https://togithub.com/lucacasonato)` in [servo/rust-url#857
-   Implement std::error::Error for InvalidBase64 by [`@&#8203;lucacasonato](https://togithub.com/lucacasonato)` in [servo/rust-url#856
-   Add `--generate-link-to-definition` option when building on docs.rs by [`@&#8203;GuillaumeGomez](https://togithub.com/GuillaumeGomez)` in [servo/rust-url#858
-   Stabilize debugger_visualizer feature by [`@&#8203;lucacasonato](https://togithub.com/lucacasonato)` in [servo/rust-url#855
-   Update WPT data and expectations by [`@&#8203;lucacasonato](https://togithub.com/lucacasonato)` in [servo/rust-url#859
-   Fix no_std Support for idna by [`@&#8203;domenukk](https://togithub.com/domenukk)` in [servo/rust-url#843
-   Fix panic in set_path for file URLs  by [`@&#8203;valenting](https://togithub.com/valenting)` in [servo/rust-url#865

##### New Contributors

-   [`@&#8203;mrobinson](https://togithub.com/mrobinson)` made their first contribution in [servo/rust-url#851
-   [`@&#8203;GuillaumeGomez](https://togithub.com/GuillaumeGomez)` made their first contribution in [servo/rust-url#858
-   [`@&#8203;domenukk](https://togithub.com/domenukk)` made their first contribution in [servo/rust-url#843

**Full Changelog**: servo/rust-url@v2.4.0...v2.4.1

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "before 5am on the first day of the month" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://togithub.com/renovatebot/renovate/discussions) if that's undesired.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/rust-lang/cargo).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNi42OC4xIiwidXBkYXRlZEluVmVyIjoiMzYuNjguMSIsInRhcmdldEJyYW5jaCI6Im1hc3RlciJ9-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Byron Byron Byron approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jongiddy @Byron