Represent Option<bool> and Option<char> as 8bit and 32bit values

[mrhota]

Closes #5977.

This is a rebase and re-issue of @Yoric's work from #31215.

CC @pnkfelix @luqmana -- anyone else?

[rust-highfive]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @Aatch (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

[Aatch]

It looks ok to me, but I'd like some other @rust-lang/compiler people to look at it as well.

One possible concern is that we're currently trying to get rid of this code in favour of using the layout code in librustc.

[eddyb]

This is a problematic time to actually make changes to layout code, you'd at least want to wait for #36151, but I'll take a look.

[eddyb]

Ah, seems that you only changed trans::adt which will most definitely result in ICEs because it won't match ty::layout.
Also, this introduces new special-cases instead of generalizing old ones (the zero/nullptr optimization).
I can mentor a more robust implementation after #36151, if either @mrhota, @camlorn or someone else wants to work on it.

[luqmana]

Yea, aside from the missing ty::layout changes it looks good to me. But i'll defer to @Aatch and @eddyb and yea, it'd probably be best to at least wait for #36151 first.

[eddyb]

My thoughts from IRC:

a proper niche optimization keeps track of the first niche with values to spare and goes looking for a second one after that one is full
right now we do that with a boolean for nullable but we could always track an u64 range, until the range is full

anything less will likely to have too much added complexity for the added benefits and will impact maintenance negatively

thinking about it more, a further strength reduction would be using field offsets instead of paths
this would even work with niches shared by multiple variants, even if they're different types

[ahicks92]

I'll probably take a look at this if no one beats me to it, but I consider struct layout optimization to be more important and intend to try to do that first.

[bors]

☔ The latest upstream changes (presumably #36016) made this pull request unmergeable. Please resolve the merge conflicts.

[arthurprs]

This is pretty cool. Good work!

[ahicks92]

Just read this. It's interesting, and kind of goes in line with some other stuff I was bringing up on IRC.

It should be pretty easy to port this to layout. If we're going to though, I think we should go all out and handle the case of n zero-sized variants and 1 variant with a forbidden field, as opposed to just one-and-one. Unlike null pointer, we can do that here without much effort.

[alexcrichton]

Closing due to inactivity, but feel free to resubmit with a rebase!