Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More native Video support #7298

Open
Tracked by #7393
emilk opened this issue Aug 28, 2024 · 3 comments
Open
Tracked by #7393

More native Video support #7298

emilk opened this issue Aug 28, 2024 · 3 comments
Labels
🧑‍💻 dev experience developer experience (excluding CI) enhancement New feature or request 🎄 tracking issue issue that tracks a bunch of subissues 🎞️ video

Comments

@emilk
Copy link
Member

emilk commented Aug 28, 2024 •
edited
Loading

✅ Av1

We currently support AV1 on native, using a fast software decoder

We currently use re_rav1d for this, but if possible we want to change to:

🏗️ H.264

Work on H.264 support is underway in

An alternative we haven't explored is to use

❌ Other codecs

Should be doable with the same approach we're using for H.264:
@emilk emilk added the enhancement New feature or request label Aug 28, 2024
@adeebshihadeh adeebshihadeh mentioned this issue Sep 2, 2024
Open
2 tasks
@jprochazk
Copy link
Member

jprochazk commented Sep 4, 2024 •
edited
Loading

Current plan is to use https://github.com/rerun-io/rav1d-rs (a fork of dav1d-rs which replaces the underlying dav1d library with rav1d) to decode AV1 video.

We're uncertain about how to support other codecs, at least VP8/9 should be easily doable in a similar way as AV1, but H.264/265 is a bit of a special case.

@jprochazk jprochazk mentioned this issue Sep 4, 2024
Closed
3 tasks
@emilk emilk added this to the 0.19 - Dataframe and web video milestone Sep 10, 2024
@emilk emilk mentioned this issue Sep 10, 2024
Closed
5 tasks
@emilk emilk added the 🧑‍💻 dev experience developer experience (excluding CI) label Sep 10, 2024
@emilk emilk assigned emilk and Wumpf Oct 1, 2024
@emilk emilk mentioned this issue Oct 2, 2024
Merged
9 tasks
@jprochazk
Copy link
Member

(Note that I made the same mistake as in re_mp4 and uploaded a bunch of binary files to rav1d-rs, which need cleaning up via history rewriting and either be turned into LFS files or removed)

@emilk
Copy link
Member Author

emilk commented Oct 3, 2024 •
edited
Loading

@jprochazk I summarized my understanding of rav1d-rs in this issue - feel free to edit/comment:

@emilk emilk mentioned this issue Oct 3, 2024
Merged
6 tasks
Wumpf added a commit that referenced this issue Oct 3, 2024
@emilk @Wumpf
Update MSRV to Rust 1.79 (#7563)
86f461e 
### What
* Required for #7298

### TODO
* [x] Fix the debug-assertion
* [x] Fix all new clippy lints
* [x] Add more clippy lints thats been added in 1.77, 1.78, 1.79


### MiMalloc bug
Debug assertions are now part of stdlib:
https://blog.rust-lang.org/2024/05/02/Rust-1.78.0.html#asserting-unsafe-preconditions,
making us hit this bug in MiMalloc:

* purpleprotocol/mimalloc_rust#128

I also opened this issue on the standard library:
* rust-lang/rust#131189

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7563?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7563?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!
* [x] If have noted any breaking changes to the log API in
`CHANGELOG.md` and the migration guide

- [PR Build Summary](https://build.rerun.io/pr/7563)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.

---------

Co-authored-by: Andreas Reich <r_andreas2@web.de>
Wumpf pushed a commit that referenced this issue Oct 3, 2024
@emilk
Make VideoDecoder a trait (#7577)
e6f93d8 
### What
* Part of #7298 

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7577?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7577?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!
* [x] If have noted any breaking changes to the log API in
`CHANGELOG.md` and the migration guide

- [PR Build Summary](https://build.rerun.io/pr/7577)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.
This was referenced Oct 3, 2024
Merged
Closed
@emilk emilk unassigned emilk and Wumpf Oct 7, 2024
This was referenced Oct 7, 2024
Open
Open
emilk added a commit that referenced this issue Oct 7, 2024
@emilk @jprochazk @Wumpf
Native video support for AV1 (#7557)
2ce1c82 
### What
* Part of #7298


### What
Supports native decoding of AV1 videos.

Downsides: it is extremely slow in debug builds

In release builds it is ok, but there is still A LOT of performance on
the table.

### TODO before merging
* #7563
* [x] Re-add `nasm` to `pixi.toml`, because it is needed to compile
`rav1d`
* [x] Make the asm-features of `rav1d` opt-in so users don't need `nasm`
to compile `rerun`
* [x] Put it behind a feature flag, in case compiling `rav1d` is
difficult on some platforms
* [x] Fix error handling so we don't crash on bad videos
* ~Fix performance of debug builds, if possible~
  *  `dav1d` is always fast, but `rav1d` is super-slow in debug builds
* [x] Or show error message in debug builds
* [x] Review moved mp4 demux code to see nothing was lost

### Proof

https://github.com/user-attachments/assets/1b1a0a03-9fac-4fa2-bcae-7beb9591067c

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using examples from latest `main` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7557?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[rerun.io/viewer](https://rerun.io/viewer/pr/7557?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG
* [x] If applicable, add a new check to the [release
checklist](https://github.com/rerun-io/rerun/blob/main/tests/python/release_checklist)!
* [x] If have noted any breaking changes to the log API in
`CHANGELOG.md` and the migration guide

- [PR Build Summary](https://build.rerun.io/pr/7557)
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)

To run all checks from `main`, comment on the PR with `@rerun-bot
full-check`.

---------

Co-authored-by: jprochazk <pjanv42@gmail.com>
Co-authored-by: Andreas Reich <andreas@rerun.io>
@emilk emilk changed the title Native Video support More native Video support Oct 11, 2024
@emilk emilk added the 🎄 tracking issue issue that tracks a bunch of subissues label Oct 11, 2024
@emilk emilk removed this from the 0.19 - Dataframe and video milestone Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🧑‍💻 dev experience developer experience (excluding CI) enhancement New feature or request 🎄 tracking issue issue that tracks a bunch of subissues 🎞️ video
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@emilk @Wumpf @jprochazk