Feature request: raw data API for PyString

[indygreg]

PyUnicode internally stores its data in various variations. See https://docs.python.org/3/c-api/unicode.html.

PyO3's PyString currently only allows you to get at UTF-8 / Rust str compatible variations of the data.

rust-cpython - by contrast - exposes a PyString.data() returning a PyStringData enum:

pub enum PyStringData<'a> {
    Latin1(&'a [u8]),
    Utf8(&'a [u8]),
    Utf16(&'a [u16]),
    Utf32(&'a [u32]),
}

This API enables Rust to have access to the raw bytes backing a Python string, not the UTF-8 normalization of it (if different).

PyOxidizer was relying on this API for testing. (There are some low-level tests around encoding handling that need to verify exact byte sequences and Python string representations are being handled properly.)

While I'm certainly capable of using unsafe Python C APIs to get at the raw string data to close this feature gap, I was curious if PyO3 would be interested in a PR to expose a PyStringData enumeration for PyString instances. Here is my proposal:

1. PyString gains a pub fn data(&self) -> PyStringData<'_>
2. PyStringData is an enum with a variant for each internal Python string variation.
3. PyString.data() calls out to PyUnicode_READY() + PyUnicode_{KIND, DATA, GET_LENGTH} and constructs a PyStringData with a slice.

I'd be willing to contribute a PR for this feature if there is interest.

[davidhewitt]

👍 I'm ok with adding this, however I think it's going to be quite hard. See my comments on #1777

FWIW given that PEPs 393 & 623 are removing the "legacy" kinds (and if an object is in the legacy state, we can convert it with PyUnicode_READY), I'd vote we aim to expose just a three-member enum which matches the remaining kinds exposed:

pub enum PyStringData<'a> {
    Ucs1(&'a [u8]),
    Ucs2(&'a [u16]),
    Ucs4(&'a [u32]),
}

There's perhaps room for an Ascii variant too. (A special-case of Ucs1 when the ascii flag is set.)