Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Stream for RowCursor #122

Open
v3xro opened this issue Aug 1, 2024 · 6 comments
Open

Implement Stream for RowCursor #122

v3xro opened this issue Aug 1, 2024 · 6 comments
Labels
enhancement New feature or request

Comments

@v3xro
Copy link
Contributor

v3xro commented Aug 1, 2024

This would make usage within other async code more ergonomic.

@loyd
Copy link
Collaborator

loyd commented Aug 4, 2024

Makes sense, possibly under dedicated futures03 feature

@loyd loyd added the enhancement New feature or request label Aug 4, 2024
@v3xro
Copy link
Contributor Author

v3xro commented Aug 5, 2024

I've had an initial attempt at implementing stream using https://github.com/tokio-rs/async-stream. However, during testing against a non-toy database I found cases where the deserialized strings are invalid utf8 which points to some buffer corruption somewhere. Not entirely sure where to start debugging that as it does not happen when you use the cursor directly manually. this is the current branch in case that's helpful: https://github.com/v3xro/clickhouse.rs/tree/feat/impl-stream

Edit 1: I found tokio-rs/async-stream#106 so will re-check, seems quite likely
Edit 2: That not the issue but #24 looks suspicious

@v3xro
Copy link
Contributor Author

v3xro commented Aug 5, 2024

@loyd do you have an idea how to fix #24? The sqlx type signature linked in the ticket did not inspire me to a solution 😁

Looking at the code, it looks like you would need to move the lifetime of the buffer into the cursor type so you can properly have the serde 'de lifetime be a subordinate of that? Ok, as I was writing that I realised that's what you meant by GAT being stable (which is the case now unlike 3 years ago).

Edit: did more research, found #46 and now wondering whether it would even be possible to provide a generic Stream API without implementing the copy variant of serde deserializers

@loyd
Copy link
Collaborator

loyd commented Aug 7, 2024

#24 affects your case only if you want to use borrowed rows in Stream impl. They are fundamentally incompatible (anyone can call StreamExt::buffered), only T: for<'b> Deserialize<'b> can be used in that case (see Query::fetch_all).

So, you should add such restrictions on v3xro@d25b1bb#diff-f0c63cae3ed0328ebbe14ac22d86b43bc8a40146944b03c8326681e4f3802443R200 as where T: for<'b> Deserialize<'b> and note in docstrings that only owned T is supported

@loyd
Copy link
Collaborator

loyd commented Aug 7, 2024

And I'm sure we should avoid async-stream. It is a good implementation, but has disadvantages:

  • It's based on macro and isn't covered by rustfmt
  • It's based on TLS with some corner cases (don't provide issues to avoid linking)

It seems it can be easily replaced with unfold here.

@loyd
Copy link
Collaborator

loyd commented Aug 7, 2024

do you have an idea how to fix #24?

I do, but not ideal ones (with breaking changes in API sadly). Let's continue in #24 if you want to help with the issue or have ideas

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants