Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Derive macro for IntoPyObject #4495

Merged
merged 17 commits into from
Oct 26, 2024
Merged

Conversation

Icxolu
Copy link
Contributor

@Icxolu Icxolu commented Aug 26, 2024

Initial implementation of #[derive(IntoPyObject)]. #[derive(FromPyObject)] was used as a base/reference for this implementation.

This currently supports #[pyo3(transparent)] as the only real option, which just forwards to the inner IntoPyObject impl.

  • Newtype structs (#[pyo3(transparent)] named and single tuple): forward impl.
  • Named structs: PyDict with the field names as keys
  • Tuple structs: PyTuple in declaration order
  • Enums: matches on the variant and treats it as an anonymous struct with the rules above.
  • Lifetimes and Generics are supported, 'py is treated special and is used a the Python<'py> lifetime if given.

We still need lots of tests here, but I thought I'll put this up as a POC already. Hopefully this can smoothen the migration to IntoPyObject a bit.

One thing not needed initially, but may be a nice to have at some point is some like #[pyo3(into_py_with = "...")] analogous to #[pyo3(from_py_with = "...")]

Closes #4458

Copy link
Member

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks brilliant, thank you so much! I am travelling with family for the next few days so it might be the weekend before I have a chance to do any reviewing... 🙏

@Icxolu
Copy link
Contributor Author

Icxolu commented Aug 28, 2024

No worries, enjoy your trip.

@Icxolu Icxolu marked this pull request as ready for review August 29, 2024 22:18
Copy link
Member

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks spot on, thanks very much. I have given this initial review from my phone so haven't read the implementation in full detail, looks great in general.

I have a bunch of various suggestions, mostly related to interactions with FromPyObject.

guide/src/migration.md Show resolved Hide resolved
body: TokenStream,
}

struct NamedStructField<'a> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should consider the interaction with FromPyObject's options like #[pyo3(item)] and #[pyo3(attribute)]. I am fine if we just parse them and reject cases where we don't want to support them for IntoPyObject (yet); all I'd like to avoid is that we don't silently ignore those options.

If there is an unsupported case then users can split their FromPyObject and IntoPyObject derives onto different structs, and they are then aware their conversions will not round-trip.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, we could definitely parse them, but I'm not sure what we would do with them in this case. Due to the PyDict return type this always uses set_item. So I think it makes only sense to parse them, if there is a way (I'm not aware of any) to create an anonymous object and set arbitrary attributes on it. I guess we could accept item to allow renaming and to make a round-trip possible if it is given.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Let's handle item round trips here and error on attribute? Users can always make two separate types if they need #[attribute] for their FromPyObject implementation for some reason.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added parsing for #[pyo3(item)] and #[pyo3(attribute)] (like for FromPyObject), roundtrip tests in test_frompy_intopy_roundtrip.rs and ui tests for the unsupported cases.

pyo3-macros-backend/src/intopyobject.rs Outdated Show resolved Hide resolved
tests/ui/invalid_intopy_derive.stderr Outdated Show resolved Hide resolved
tests/test_intopyobject.rs Outdated Show resolved Hide resolved
Copy link
Member

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the long review cycle, this is looking great and I think we're nearly there! Main change I would like to see is just on the interaction with the FromPyObject attributes, where I think it makes our future development simpler if we handle that before releasing.

guide/src/conversions/traits.md Outdated Show resolved Hide resolved
guide/src/migration.md Outdated Show resolved Hide resolved
Crate(CrateAttribute),
}

impl Parse for ContainerPyO3Attribute {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reading all these structures it feels like there's a ton of boilerplate which might be interesting to refactor into common macro_rules! patterns (maybe). Not for this PR, but maybe a non-urgent follow up to explore sometime.

Comment on lines +179 to +291
_ => bail_spanned!(
fields.span() => "cannot derive `IntoPyObject` for empty structs"
),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder, does it make sense for it to be empty tuple and/or empty dict? Perhaps again we leave this for a follow up and consider FromPyObject at the same time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I think revisiting this in a followup (perhaps after 0.23) makes sense.

body: TokenStream,
}

struct NamedStructField<'a> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. Let's handle item round trips here and error on attribute? Users can always make two separate types if they need #[attribute] for their FromPyObject implementation for some reason.

@Icxolu
Copy link
Contributor Author

Icxolu commented Oct 20, 2024

No worries, thanks for the comments. I'll have a look in a few days.

Copy link
Member

@davidhewitt davidhewitt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks perfect, thanks so much! I'm really optimistic this will help users have a good experience upgrading to 0.23! 🚀

@davidhewitt davidhewitt added this pull request to the merge queue Oct 26, 2024
Merged via the queue into PyO3:main with commit c8eb45b Oct 26, 2024
44 checks passed
@Icxolu Icxolu deleted the derive-intopyobject branch October 26, 2024 08:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

ENH: boiler plate for IntoPy<PyObject>
2 participants