-
Notifications
You must be signed in to change notification settings - Fork 334
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DataLoader
s 5: add support for external binary DataLoader
s (PATH)
#4521
Conversation
DataLoader
s 4: add support for external binary DataLoader
s (PATH)DataLoader
s 5: add support for external binary DataLoader
s (PATH)
26110a1
to
f3cff17
Compare
ce95b4d
to
788acd1
Compare
Size changes
|
14e7ea7
to
3e4babe
Compare
60df8f9
to
b425400
Compare
c1f6af8
to
d8aa5e4
Compare
I forgot to update the |
@@ -0,0 +1,188 @@ | |||
use std::io::Read; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You should document somewhere what we expect from a loader, eg in a //!
-level docstring
In particular:
- what arguments to expect (
--recording-id
) - what exit code to use for unsupported file types
- what and where to output (.rrd data to
stdout
)
This should also be added as a how-to guide on rerun.io/docs!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have an upcoming PR that adds a web guide, I'll add a link to that guide in the doc string then
**COMMIT PER COMMIT!** _That's the nasty one... there's always a nasty one 😬_ Introduces the `DataLoader` trait, which can load any file, either from a filepath by reading from the local filesystem, or from a pre-loaded file's contents. `DataLoader`s are run in parallel where possible, and have complete say over what data they are interested in or not (i.e. they are not registered based on an extension, which is very limited in practice). I've decided to commit small example assets for the types we support, cause I feel we really need a fast way of checking whether everything still works from time to time. It's pretty light but might be controversial, I figure. Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd}` - [x] Native: `File > Open > examples/assets/*` - [x] Native: `Drag-n-drop > examples/assets/*` - [x] Web: `File > Open > examples/assets/*` - [x] Web: `Drag-n-drop > examples/assets/*` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
What the title says. ![image](https://github.com/rerun-io/rerun/assets/2910679/68f2e499-f4df-4e75-95f6-0ed7f479c5e6) Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md}` - [x] Native: `File > Open > examples/assets/*` - [x] Native: `Drag-n-drop > examples/assets/*` - [x] Web: `File > Open > examples/assets/*` - [x] Web: `Drag-n-drop > examples/assets/*` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
`.ply` support, but point clouds only, at least for now. No feature flag: I had some, and tbh it only made thing needlessly hard to work with for very little gains. `.ply` are so common anyway, they always come up in user requests. Here's an example from [there](https://github.com/HuangCongQing/Point-Clouds-Visualization/blob/master/2open3D/data/fragment.ply): ![image](https://github.com/rerun-io/rerun/assets/2910679/8242c5a2-037d-46c6-9816-ab9b240a7473) Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets/example.{glb,gltf,obj,jpg,png,rrd,txt,md,ply}` - [x] Native: `File > Open > examples/assets/*` - [x] Native: `Drag-n-drop > examples/assets/*` - [x] Web: `File > Open > examples/assets/*` - [x] Web: `Drag-n-drop > examples/assets/*` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
b425400
to
18a7f24
Compare
Load folders filled with whatever. Go nuts. The loader has no particular semantics (e.g. HIVE partitioning): it just loads everything in a folder using the appropriate loader(s) for each file. Checks: - [x] `cargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets` - [x] Native: `File > Open > examples/assets/` - [x] Native: `Drag-n-drop > examples/assets/` --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - TODO: register custom loaders - TODO: high level docs and guides for everything related to loading files
51560f5
to
fea2a31
Compare
Introduce a first-class protocol for `DataLoader`s -- whether they are builtin, custom, or external -- to announce that they don't support loading a given piece of data. This fixes one of the oldest issue when loading files in Rerun: loading unsupported data would always bottom out in either the image or websocket paths, leading to unrelated errors very confusing to users. The loading process is now a two-pass process: 1. Dispatch the data to be loaded to all loaders, which will respond ASAP whether they can do it or not. 1.1. If at least one compatible loader is found, go to 2 1.2. Otherwise, fail early with a nice error message for the end user 2. Dispatch the actual data loading. This has important/subtle ramifications on the threading model, but other than that is what you'd expect. Checks: - [x] Native: CLI examples/assets/* - [x] Native: CLI examples/assets - [x] Native: CLI examples/assets/* containing unsupported files - [x] Native: CLI examples/assets containing unsupported files - [x] Native: File>Open examples/assets/* - [x] Native: File>Open examples/assets - [x] Native: File>Open examples/assets/* containing unsupported files - [x] Native: File>Open examples/assets containing unsupported files - [x] Native: Drag-n-drop examples/assets/* - [x] Native: Drag-n-drop examples/assets - [x] Native: Drag-n-drop examples/assets/* containing unsupported files - [x] Native: Drag-n-drop examples/assets containing unsupported files - [x] Web: File>Open examples/assets/* - [x] Web: Drag-n-drop examples/assets/* - [x] Web: File>Open examples/assets/* containing unsupported files - [x] Web: Drag-n-drop examples/assets/* containing unsupported files --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567
Add support for registering custom loaders, the same way we offer support for registering custom space views and store subscribers. Checks: - [x] cargo r -p custom_data_loader -- Cargo.toml --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567
Some docs for the whole thing. It heavily relies on linking to the examples for exact details and to avoid rot. It's probably extremely bad. Help welcome. --- Part of a series of PRs to make it possible to load _any_ file from the local filesystem, by any means, on web and native: - #4516 - #4517 - #4518 - #4519 - #4520 - #4521 - #4565 - #4566 - #4567 Co-authored-by: Nikolaus West <niko@rerun.io>
* Uses: rerun-io/egui_tiles#67 * Uses: lampsitter/egui_commonmark#51 * Split off from #6171 * Closes #5280 ### Test * [x] image loading * [x] gltf ### Checklist * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested the web demo (if applicable): * Using examples from latest `main` build: [rerun.io/viewer](https://rerun.io/viewer/pr/6448?manifest_url=https://app.rerun.io/version/main/examples_manifest.json) * Using full set of examples from `nightly` build: [rerun.io/viewer](https://rerun.io/viewer/pr/6448?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json) * [x] The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG * [x] If applicable, add a new check to the [release checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)! - [PR Build Summary](https://build.rerun.io/pr/6448) - [Recent benchmark results](https://build.rerun.io/graphs/crates.html) - [Wasm size tracking](https://build.rerun.io/graphs/sizes.html) To run all checks from `main`, comment on the PR with `@rerun-bot full-check`. # egui changelog so far Full diff at https://github.com/emilk/egui/compare/0.27.0..HEAD #### ecolor * Fix `hex_color!` macro by re-exporting `color_hex` crate from `ecolor` [#4372](emilk/egui#4372) (thanks [@dataphract](https://github.com/dataphract)!) * Remove `extra_asserts` and `extra_debug_asserts` feature flags [#4478](emilk/egui#4478) #### eframe * Add `register_native_texture` in `eframe::Frame` [#4246](emilk/egui#4246) (thanks [@Chaojimengnan](https://github.com/Chaojimengnan)!) * Early-out from context switching the `glow` backend [#4284](emilk/egui#4284) * Fix `ViewportCommand::InnerSize` not resizing viewport on Wayland (#4211) [#4211](emilk/egui#4211) (thanks [@rustbasic](https://github.com/rustbasic)!) * Only avoid glow context switching on Windows [#4296](emilk/egui#4296) * Improve IME support with new `Event::Ime` [#4358](emilk/egui#4358) (thanks [@rustbasic](https://github.com/rustbasic)!) * Allow users to create viewports larger than monitor on Windows & macOS [#4337](emilk/egui#4337) (thanks [@lopo12123](https://github.com/lopo12123)!) * Use `objc2` and its framework crates [#4395](emilk/egui#4395) (thanks [@madsmtm](https://github.com/madsmtm)!) * Update to Rust 1.76 [#4411](emilk/egui#4411) * Egui-winit: emit physical key presses when a non-Latin layout is active [#4461](emilk/egui#4461) (thanks [@TicClick](https://github.com/TicClick)!) * IME for chinese [#4436](emilk/egui#4436) (thanks [@rustbasic](https://github.com/rustbasic)!) * Fix : In Windows, the 'egui_demo_app' screen does not appear. [#4410](emilk/egui#4410) (thanks [@rustbasic](https://github.com/rustbasic)!) * Fix: Window position creeps between executions on scaled monitors [#4443](emilk/egui#4443) (thanks [@avery-radmacher](https://github.com/avery-radmacher)!) * Update `image` crate to 0.25 [#4160](emilk/egui#4160) * Ignore synthetic key presses [#4514](emilk/egui#4514) (thanks [@hut](https://github.com/hut)!) * Fix: still track mouse when dragging outside web canvas [#4522](emilk/egui#4522) * Add `NativeOptions::persistence_path` [#4423](emilk/egui#4423) (thanks [@lucasmerlin](https://github.com/lucasmerlin)!) * Use ResizeObserver instead of `resize` event [#4536](emilk/egui#4536) (thanks [@jprochazk](https://github.com/jprochazk)!) * Fix: Don't `.forget()` RAF closure [#4551](emilk/egui#4551) (thanks [@jprochazk](https://github.com/jprochazk)!) #### egui_extras * Update `image` crate to 0.25 [#4160](emilk/egui#4160) #### egui_plot * `Plot::Items:allow_hover` give possibility to masked the interaction on hovered item [#2558](emilk/egui#2558) (thanks [@haricot](https://github.com/haricot)!) * Expose `ClosestElem` and `PlotConfig` [#4380](emilk/egui#4380) (thanks [@Narcha](https://github.com/Narcha)!) * Disable interaction for `ScrollArea` and `Plot` when UI is disabled [#4457](emilk/egui#4457) (thanks [@varphone](https://github.com/varphone)!) * Make sure plot size is positive [#4429](emilk/egui#4429) (thanks [@rustbasic](https://github.com/rustbasic)!) * Introduce lifetime to `egui_plot::Plot` to replace `'static` fields [#4435](emilk/egui#4435) (thanks [@Fabus1184](https://github.com/Fabus1184)!) * Hide all other series when alt-clicking in the legend [#4549](emilk/egui#4549) (thanks [@abey79](https://github.com/abey79)!) * Plot now respects the `interact_radius` set in the UI's style [#4520](emilk/egui#4520) (thanks [@YgorSouza](https://github.com/YgorSouza)!) #### egui_glow * Enable egui_glow's winit feature on wasm (#4420) [#4421](emilk/egui#4421) (thanks [@simon-frankau](https://github.com/simon-frankau)!) #### egui-wgpu * Update to wgpu 0.20 [#4433](emilk/egui#4433) (thanks [@KeKsBoTer](https://github.com/KeKsBoTer)!) * Revert update to wgpu 0.20 => downgrade to wgpu 0.19.1 [#4559](emilk/egui#4559) #### egui-winit * Update `webbrowser` to `v1.0.0` [#4394](emilk/egui#4394) (thanks [@torokati44](https://github.com/torokati44)!) * Emit physical key presses when a non-Latin layout is active [#4461](emilk/egui#4461) (thanks [@TicClick](https://github.com/TicClick)!) * IME for chinese [#4436](emilk/egui#4436) (thanks [@rustbasic](https://github.com/rustbasic)!) * Fix: Window position creeps between executions on scaled monitors [#4443](emilk/egui#4443) (thanks [@avery-radmacher](https://github.com/avery-radmacher)!) * Ignore synthetic key presses [#4514](emilk/egui#4514) (thanks [@hut](https://github.com/hut)!) #### egui * Improve the UI for changing the egui theme [#4257](emilk/egui#4257) * Change the resize cursor when you reach the resize limit [#4275](emilk/egui#4275) * Make `TextEdit` an atomic widget [#4276](emilk/egui#4276) * Overload operators for `Rect + Margin`, `Rect - Margin` etc [#4277](emilk/egui#4277) * Implement blinking text cursor in `TextEdit` [#4279](emilk/egui#4279) * Rename `fn scroll2` to `fn scroll` [#4282](emilk/egui#4282) * Change `Frame::multiply_with_opacity` to multiply in gamma space [#4283](emilk/egui#4283) * Support order on windows [#4301](emilk/egui#4301) (thanks [@alexparlett](https://github.com/alexparlett)!) * Fix wrong replacement function in deprecation notice of `drag_released*` [#4314](emilk/egui#4314) (thanks [@sornas](https://github.com/sornas)!) * Consider layer transform when positioning text agent [#4319](emilk/egui#4319) (thanks [@juancampa](https://github.com/juancampa)!) * Fix incorrect line breaks [#4377](emilk/egui#4377) (thanks [@juancampa](https://github.com/juancampa)!) * Fix `hex_color!` macro by re-exporting `color_hex` crate from `ecolor` [#4372](emilk/egui#4372) (thanks [@dataphract](https://github.com/dataphract)!) * Change `Ui::allocate_painter` to inherit properties from `Ui` [#4343](emilk/egui#4343) (thanks [@varphone](https://github.com/varphone)!) * Use parent `Ui`s style for popups [#4325](emilk/egui#4325) (thanks [@alexparlett](https://github.com/alexparlett)!) * Fix : take `rounding` into account when using `Slider::trailing_fill` [#4308](emilk/egui#4308) (thanks [@rustbasic](https://github.com/rustbasic)!) * Add a way to specify Undoer settings and construct Undoers more easily [#4357](emilk/egui#4357) (thanks [@valadaptive](https://github.com/valadaptive)!) * Add xtask crate [#4293](emilk/egui#4293) (thanks [@YgorSouza](https://github.com/YgorSouza)!) * Add `ViewportCommand::RequestCut`, `RequestCopy` and `RequestPaste` to trigger Clipboard actions [#4035](emilk/egui#4035) (thanks [@bu5hm4nn](https://github.com/bu5hm4nn)!) * Fix `Panel` incorrect size [#4351](emilk/egui#4351) (thanks [@zhatuokun](https://github.com/zhatuokun)!) * Improve IME support with new `Event::Ime` [#4358](emilk/egui#4358) (thanks [@rustbasic](https://github.com/rustbasic)!) * Allow users to create viewports larger than monitor on Windows & macOS [#4337](emilk/egui#4337) (thanks [@lopo12123](https://github.com/lopo12123)!) * Added ability to define colors at UV coordinates along a path [#4353](emilk/egui#4353) (thanks [@murl-digital](https://github.com/murl-digital)!) * Eframe: update ViewportBuilder.with_icon() documentation [#4408](emilk/egui#4408) (thanks [@roccoblues](https://github.com/roccoblues)!) * Update to Rust 1.76 [#4411](emilk/egui#4411) * Add a `Display` impl for `Vec2`, `Pos2`, and `Rect` [#4428](emilk/egui#4428) (thanks [@tgross35](https://github.com/tgross35)!) * Remove `extra_asserts` and `extra_debug_asserts` feature flags [#4478](emilk/egui#4478) * Egui-winit: emit physical key presses when a non-Latin layout is active [#4461](emilk/egui#4461) (thanks [@TicClick](https://github.com/TicClick)!) * Disable interaction for `ScrollArea` and `Plot` when UI is disabled [#4457](emilk/egui#4457) (thanks [@varphone](https://github.com/varphone)!) * Update ahash 0.8.6 -> 0.8.11 [#4507](emilk/egui#4507) (thanks [@hellodword](https://github.com/hellodword)!) * `include_image!` now accepts expressions [#4521](emilk/egui#4521) (thanks [@YgorSouza](https://github.com/YgorSouza)!) * Remove `Event::Scroll` and handle it in egui [#4524](emilk/egui#4524) * Remove scroll latency for smooth trackpads [#4526](emilk/egui#4526) * Smooth out zooming with discreet scroll wheel [#4530](emilk/egui#4530) * Add `Options::line_scroll_speed` and `scroll_zoom_speed` [#4532](emilk/egui#4532) * Don't panic when replacement glyph is not found [#4542](emilk/egui#4542) (thanks [@RyanBluth](https://github.com/RyanBluth)!) * Make `TextEdit::return_key` optional [#4543](emilk/egui#4543) (thanks [@doonv](https://github.com/doonv)!) * Add `TextEdit::hint_text_font` [#4517](emilk/egui#4517) (thanks [@zaaarf](https://github.com/zaaarf)!) * Add `Options::reduce_texture_memory` to free up RAM [#4431](emilk/egui#4431) (thanks [@varphone](https://github.com/varphone)!) * Fix `Ui::scroll_with_delta` only scrolling if the `ScrollArea` is focused [#4303](emilk/egui#4303) (thanks [@lucasmerlin](https://github.com/lucasmerlin)!) * Add support for text truncation to `egui::Style` [#4556](emilk/egui#4556) (thanks [@abey79](https://github.com/abey79)!) * Hide toolip when opening `ComboBox` drop-down [#4546](emilk/egui#4546) (thanks [@abey79](https://github.com/abey79)!) * Better spacing and sizes for (menu) buttons [#4558](emilk/egui#4558) #### epaint * Add `RectShape::blur_width` to implement shadows [#4267](emilk/egui#4267) * Overload operators for `Rect + Margin`, `Rect - Margin` etc [#4277](emilk/egui#4277) * Fix incorrect line breaks [#4377](emilk/egui#4377) (thanks [@juancampa](https://github.com/juancampa)!) * Fix `hex_color!` macro by re-exporting `color_hex` crate from `ecolor` [#4372](emilk/egui#4372) (thanks [@dataphract](https://github.com/dataphract)!) * Add `emath::OrderedFloat` (moved from `epaint::util::OrderedFloat`) [#4389](emilk/egui#4389) * Added ability to define colors at UV coordinates along a path [#4353](emilk/egui#4353) (thanks [@murl-digital](https://github.com/murl-digital)!) * Add a `Display` impl for `Vec2`, `Pos2`, and `Rect` [#4428](emilk/egui#4428) (thanks [@tgross35](https://github.com/tgross35)!) * Remove `extra_asserts` and `extra_debug_asserts` feature flags [#4478](emilk/egui#4478) * Make `epaint::mutex::RwLock` allow `?Sized` types [#4485](emilk/egui#4485) (thanks [@crumblingstatue](https://github.com/crumblingstatue)!) * Update ahash 0.8.6 -> 0.8.11 [#4507](emilk/egui#4507) (thanks [@hellodword](https://github.com/hellodword)!) * Don't panic when replacement glyph is not found [#4542](emilk/egui#4542) (thanks [@RyanBluth](https://github.com/RyanBluth)!) --------- Co-authored-by: Antoine Beyeler <antoine@rerun.io> Co-authored-by: Andreas Reich <r_andreas2@web.de>
Turn any executable in your
PATH
into aDataLoader
by logging data to stdout.Includes examples for all supported languages.
Checks:
rerun-loader-rust-file
just prints help messagererun-loader-rust-file examples/rust/external_data_loader/src/main.rs
successfully does nothingrerun-loader-rust-file examples/rust/external_data_loader/src/main.rs | rerun -
worksrerun-loader-python-file
just prints help messagererun-loader-python-file examples/python/external_data_loader/main.py
successfully does nothingrerun-loader-python-file examples/python/external_data_loader/main.py | rerun -
worksrerun-loader-cpp-file
just prints help messagererun-loader-cpp-file examples/cpp/external_data_loader/main.cpp
successfully does nothingrerun-loader-cpp-file examples/cpp/external_data_loader/main.cpp | rerun -
workscargo r -p rerun-cli --no-default-features --features native_viewer -- examples/assets
File > Open > examples/assets/
Drag-n-drop > examples/assets/
Part of a series of PRs to make it possible to load any file from the local filesystem, by any means, on web and native:
DataLoader
s 0: utility for hierarchicalEntityPath
from file path #4516DataLoader
s 1: introduce, and migrate to,DataLoader
s #4517DataLoader
s 2: add text-basedDataLoader
(.txt
,.md
) #4518DataLoader
s 3: add 3D point cloudDataLoader
(.ply
) #4519DataLoader
s 4: add generic folderDataLoader
#4520DataLoader
s 5: add support for external binaryDataLoader
s (PATH) #4521Checklist