Web Playground with WASM #9834
Replies: 5 comments 12 replies
-
My opinion is that the WASM bindings have two primary usecases:
Thus I would personally suggest focusing first on the SQL API (as it has the lowest surface area) and then maybe the dataframe API |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your insight 👍 My personal short-term plan for this binding would look like:
I'll open a tracking issue in the binding repo for easier collaboration. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
BTW: What would you think about moving your playground to https://github.com/datafusion-contrib (I can set you up a repo there with full admin rights) ? I think that might allow more people to find and potentially contribute to your project if that is something you are interested with |
Beta Was this translation helpful? Give feedback.
-
|
Singlestore as well - https://docs.singlestore.com/cloud/reference/code-engine-powered-by-wasm/create-wasm-udfs/ |
Beta Was this translation helpful? Give feedback.
-
|
Just came back from holiday, sorry for the delay.
I'd like to! There are two repos, one for the Web playground and one for the WASM binding. Maybe we need two positions in datafusion-contrib? |
Beta Was this translation helpful? Give feedback.
-
|
awesome @waynexia what should they be called? |
Beta Was this translation helpful? Give feedback.
-
|
Looks good to me 👍 I'll transfer the code tomorrow |
Beta Was this translation helpful? Give feedback.
-
|
New repos created: https://github.com/datafusion-contrib/datafusion-wasm-playground 🚀 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @waynexia . Super nice project! I've been looking at your project and trying to build something similar. I'm running into some performance issues that I noticed you've handled better in your datafustion playground. I'm building with Rust on both sides - a wasmtime host that loads a WASM module (also in Rust, using DataFusion). I've got a .wit file for the interface and I'm building with The thing is, my WASM file ends up at 46MB, and wasmtime takes about 3 minutes just to get the native code ready to run. I was poking around your playground and noticed your WASM file is quite a bit smaller (32MB) and runs way faster. Any chance you could share what you did to get those better numbers? |
Beta Was this translation helpful? Give feedback.
-
|
I have noticed that the datafusion-cli binary is also substantially larger these days. Maybe they are related 🤔 I have been thinking of filing a ticket to investigate / look into this code size |
Beta Was this translation helpful? Give feedback.
-
|
I filed #13813 to collect things to improve, including binary size. |
Beta Was this translation helpful? Give feedback.
-
|
For the building part, I prefer using I'm not using wasmtime, there might be some differences on different target hosts (mine is focused on |
Beta Was this translation helpful? Give feedback.
-
|
I am trying to consolidating various WASM related improvements here: Let's move additional discussion there |
Beta Was this translation helpful? Give feedback.
-
|
@findepi filed a ticket to discuss making a playground like this part of DataFusion |
Beta Was this translation helpful? Give feedback.
-
Hi! I've made a WASM binding for DataFusion (https://github.com/waynexia/datafusion-wasm) and a Web Playground based on it (repo, demo)
It now supports basic queries (like
create,insert,selectandexplain). Generally speaking, DataFusion works pretty well in WASM!My definition of this Web Playground is something like DuckDB WASM, or the
datafusion-cliin browser. But not sure about what the shape ofdatafusion-wasmbinding should be. I.e., expose as many APIs as possible like an embedded computing library, or just focus on the core SQL/DataFrame interface. I would like to hear feedback from you.Beta Was this translation helpful? Give feedback.
All reactions