Driver/Input: Migrate audio backend to Symphonia #89
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Export is mainly for testing, but I think it'd also be a nice backport.
…rt. different input packet sizes.
Serenity's cache design has changed, this (finally) prevents the examples from failing to compile on CI.
Adds generics to any `Id` types on `Call`. Also includes the overlooked `Songbird::get_or_insert`. Closes serenity-rs#94. Tested using `cargo make ready`.
This matches a recent serenity change where `ClientBuilder` no longer has an explicit lifetime parameter. This was tested using `cargo make ready`.
MP3s now work great under the convert -> resample -> mix pipeline, even across unclean frame boundaries. Main issue seemed to be the number of internal subchunks in the resampler, which is still totally opaque... Oh well. Actual instantiation of Lazy->Wrapped and Wrapped->Parsed elements in the driver is not yet covered. Other formats are broken due to handling of the default track id, which is currently a) messy, and b) incorrect.
Oggs have some frames whose length is 0, yet must be decoded. I assume this covers the entire coder delay.
Tested with a URL extracted from bandcamp via youtube-dl, also includes the scaffolding necessary to have Reads/Seeks pass between sync/async boundaries. This format is MP3, streamed in over an HTTP request via reqwest. Seeks not tested with e.g. Async Files -- they are programmed, though.
|
Rough status update: we now have a decent mixing pipeline for audio files of different formats, samples rates, packet lengths etc. into a single buffer. This is horrifically messy. Opus packet passthrough isn't there yet, but should be possible with some care around codec resets. There are some issues around track data cleanup right now. We also have a wrapper for running seeks/reads over async audio sources (i.e., reqwest calls) and passing those bytes in via ring buffer, which seem to be working great for exposing some amount of flexibility around input sources. In tandem, mkv support in symphonia seems to be coming along well -- hopefully with that youtube-dl should integrate nicely to allow e.g. WebMs. I hope that will come down to simple parsing, with occasional renegotiation for a new DASH source if we don't want the stream to cut out halfway through. |
Moves all sync creation/parsing/seeking etc. over to an elastically-sized thread pool. Since basically everything is now a restartable, this means that the `ForwardOnly` distinction can be handled really cleanly in general and we can try to recreate sources as needed.
Asks for any streams without webm, since they're likely to be golden right now.
* Remove unnecessary poison messages * Re-add Poison for CoreMessage
This is most relevant for queue users -- an extra track would cause opus passthrough to end, even though only one track was being actively used in mixing.
Corollary: this same new clippy lint adds a ton of false positives which *will* fail to compile.
Will leave 're-computing filesize guesses' to a future issue.
|
I've put a fairly in-depth review and commenting round into the core driver changes, and I've had a more cursory look over the rest of the changes (inputs, adapters, tracks). I'll be merging shortly -- between this and the adventurous use in prod from a handful of users I think this is really quite stable now. Most importantly, it's fast and it's lightweight as a pure Rust implementation should be. One caveat is that we're waiting on one symphonia bug fix being merged and released before this can go to crates.io, at some point this might need to be pushed out with a disclaimer that the current symphonia fork should be Cargo patched in instead. Hopefully this won't be the case! |
FelixMcFelix
added a commit
that referenced
this pull request
Jul 26, 2022
This extensive PR rewrites the internal mixing logic of the driver to use symphonia for parsing and decoding audio data, and rubato to resample audio. Existing logic to decode DCA and Opus formats/data have been reworked as plugins for symphonia. The main benefit is that we no longer need to keep yt-dlp and ffmpeg processes alive, saving a lot of memory and CPU: all decoding can be done in Rust! In exchange, we now need to do a lot of the HTTP handling and resumption ourselves, but this is still a huge net positive. `Input`s have been completely reworked such that all default (non-cached) sources are lazy by default, and are no longer covered by a special-case `Restartable`. These now span a gamut from a `Compose` (lazy), to a live source, to a fully `Parsed` source. As mixing is still sync, this includes adapters for `AsyncRead`/`AsyncSeek`, and HTTP streams. `Track`s have been reworked so that they only contain initialisation state for each track. `TrackHandles` are only created once a `Track`/`Input` has been handed over to the driver, replacing `create_player` and related functions. `TrackHandle::action` now acts on a `View` of (im)mutable state, and can request seeks/readying via `Action`. Per-track event handling has also been improved -- we can now determine and propagate the reason behind individual track errors due to the new backend. Some `TrackHandle` commands (seek etc.) benefit from this, and now use internal callbacks to signal completion. Due to associated PRs on felixmcfelix/songbird from avid testers, this includes general clippy tweaks, API additions, and other repo-wide cleanup. Thanks go out to the below co-authors. Co-authored-by: Gnome! <45660393+GnomedDev@users.noreply.github.com> Co-authored-by: Alakh <36898190+alakhpc@users.noreply.github.com>
FelixMcFelix
added a commit
to FelixMcFelix/songbird
that referenced
this pull request
Nov 20, 2023
This extensive PR rewrites the internal mixing logic of the driver to use symphonia for parsing and decoding audio data, and rubato to resample audio. Existing logic to decode DCA and Opus formats/data have been reworked as plugins for symphonia. The main benefit is that we no longer need to keep yt-dlp and ffmpeg processes alive, saving a lot of memory and CPU: all decoding can be done in Rust! In exchange, we now need to do a lot of the HTTP handling and resumption ourselves, but this is still a huge net positive. `Input`s have been completely reworked such that all default (non-cached) sources are lazy by default, and are no longer covered by a special-case `Restartable`. These now span a gamut from a `Compose` (lazy), to a live source, to a fully `Parsed` source. As mixing is still sync, this includes adapters for `AsyncRead`/`AsyncSeek`, and HTTP streams. `Track`s have been reworked so that they only contain initialisation state for each track. `TrackHandles` are only created once a `Track`/`Input` has been handed over to the driver, replacing `create_player` and related functions. `TrackHandle::action` now acts on a `View` of (im)mutable state, and can request seeks/readying via `Action`. Per-track event handling has also been improved -- we can now determine and propagate the reason behind individual track errors due to the new backend. Some `TrackHandle` commands (seek etc.) benefit from this, and now use internal callbacks to signal completion. Due to associated PRs on felixmcfelix/songbird from avid testers, this includes general clippy tweaks, API additions, and other repo-wide cleanup. Thanks go out to the below co-authors. Co-authored-by: Gnome! <45660393+GnomedDev@users.noreply.github.com> Co-authored-by: Alakh <36898190+alakhpc@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This begins work on #67, which will be a huge step towards purely in-process audio processing (i.e., one day eliminating ffmpeg).
inputinterface.There will be some slight issues stemming from the fact that we will now be parsing metadata from bytestreams as they arrive:
spawn_blockingon the initialFormatparse phase? Pass messages out from the main mixer context?MediaSourceStreams,Reads, and fully processed(Format, Metadata, ...)sets -- due to the above, and as some extra metadata will only be exposed by youtube-dl, for instance.AsyncReads into the mixing system (as well as automating extension/MIME type extraction fromFiles or HTTP responses).