generate python constructor types for s3 services

[trim21]

Which issue does this PR close?

Closes #.

Rationale for this change

Not sure if this is a good idea for opendal developers or not, this add extra burden in keeping types synced with builder.

What changes are included in this PR?

add type annotation for s3 operator.

Are there any user-facing changes?

Yes, add argument typing checking for users using s3 service.

[messense]

It's a nice addition IMO, but it does add more maintenance burden. I'd be happy to include it if we can add a pytest to at least validate it to some extend, like it has specified all possible arguments. But it does not seem easy to do because the Python binding api does not expose any information about the config of each service.

Maybe we could also consider exposing XXXConfig to python and also accept that in Operator(config=XXXConfig()) so it's strongly typed?

[trim21]

like it has specified all possible arguments

Some arguments are optional, I don't think this is possible?

Maybe we could also consider exposing XXXConfig to python and also accept that in Operator(config=XXXConfig()) so it's strongly typed?

Then how we type XXXConfig.__init__?

[messense]

To clarify, I meant all possible config arguments are included in .pyi.

Then how we type XXXConfig.__init__?

Just like any other #[pyclass], it'll be easy to validate because we can use ast to parse .pyi while using inspect module to get argspec of the #[pyclass] constructor, then you can compare them in test case.

[trim21]

To clarify, I meant all possible config arguments are included in .pyi.

Then how we type XXXConfig.__init__?

Just like any other #[pyclass], it'll be easy to validate because we can use ast to parse .pyi while using inspect module to get argspec of the #[pyclass] constructor, then you can compare them in test case.

in that case，I think we can generate Operator.init overloads from config directly？

[messense]

I think we can generate Operator.init overloads from config directly？

Totally

[trim21]

I'll have a try

[Xuanwo]

Node.js, C++, and Java bindings all have similar issues. I'm wondering if there is a general solution for this.

[trim21]

I don't know rust very well, so I write a very simple rust script to generat pyi for s3.

I'm guessing maybe most of these rust code can be improved.

I do not know internal details about opendal., but looks like S3Config doesn' match options in docs https://opendal.apache.org/docs/rust/opendal/services/struct.S3.html

[Xuanwo]

Hi, I like this direction. I believe we can integrate it into our workflow more effectively. Let me give it a try.

[Xuanwo]

Hi, @trim21. I plan to add some helper utilities to our dev tools. Would you be interested in adding a Python code generator to it later? I'll ping you on the PR.

[trim21]

Hi, @trim21. I plan to add some helper utilities to our dev tools. Would you be interested in adding a Python code generator to it later? I'll ping you on the PR.

ok

[trim21]

oops, didn't notice there is already a empty dev crate.

[trim21]

I think we should generate some kind of ir and generate types for each languages from ir

[Xuanwo]

I think we should generate some kind of ir and generate types for each languages from ir

100%!

Working on this now.

[trim21]

I think we should generate some kind of ir and generate types for each languages from ir

100%!

Working on this now.

I'll close this for now and wait for your pr. feel free to cherry pick code if you like