Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Async iterator streams wip #271

Closed
wants to merge 14 commits into from
Closed

Async iterator streams wip #271

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
082939b
Re-enable content_length in read_row_group, implement a wrapper JsPa…
H-Plus-Time Jul 31, 2023
7aa06de
*Carefully* bump web-sys, enable unstable web apis, minimal logic for…
H-Plus-Time Aug 1, 2023
5ed5f3d
Fast-forward to latest version of arrow (w/ export adaptions), satisf…
H-Plus-Time Aug 2, 2023
2da4734
wip ffi for arrow1
kylebarron Aug 7, 2023
ab7c571
add make_array
kylebarron Aug 7, 2023
e4b630e
Merge remote-tracking branch 'upstream/main' into async-iterator-stre…
H-Plus-Time Aug 7, 2023
8e477e5
Address stylistic feedback
H-Plus-Time Aug 7, 2023
c490fde
Complete (rough) low-level streaming for arrow2 (the whole stream mat…
H-Plus-Time Aug 7, 2023
ec51142
Implement AsyncWrite trait wrapper for ReadableStreamController, reas…
H-Plus-Time Aug 10, 2023
18bdf74
Apply cargo fmt, switch to pull-based stream for arrow1, add close fo…
H-Plus-Time Aug 11, 2023
4cc7468
Add streaming read tests
H-Plus-Time Aug 11, 2023
d3e7204
Merge remote-tracking branch 'upstream/main' into async-iterator-stre…
H-Plus-Time Aug 11, 2023
83079e0
Merge remote-tracking branch 'upstream/kyle/wip-arrow1-ffi' into asyn…
H-Plus-Time Aug 17, 2023
aa6b38e
Reasonably functional ReadableStream of FFIArrowRecordBatch structs
H-Plus-Time Aug 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .cargo/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
[build]
rustflags = ["--cfg=web_sys_unstable_apis"]
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this mean? Is this unstable on the Rust side or JS side? Are there browser implications? Does this work on all browsers that support Wasm?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unstable on the rust side - it's required for use of the web_sys::ReadableStreamDefaultController (ReadableStream itself doesn't require it). It looks like the controller class isn't globally defined on browsers older than ~Feb 2020 (~7% in most jurisdictions), but calling new ReadableStream({start, pull, cancel}) works as far back as at least Chrome 57 (Firefox 52-65 and Edge 16-18 appear to be the only versions where WebAssembly support was available before ReadableStream constructors). Eyeballing the caniuse figures, its about 0.1% of browsers with !ReadableStream && Webassembly.

I was a little leery of the potential for the Rust side method names to change, but it looks like the web-sys codegen for optional arguments (<original_method_name>with) is pretty stable.


[target.wasm32-unknown-unknown]
rustflags = ["--cfg=web_sys_unstable_apis"]
Loading