Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audio stretching implementation in playback #25

Open
goatchurchprime opened this issue Aug 27, 2024 · 3 comments
Open

Audio stretching implementation in playback #25

goatchurchprime opened this issue Aug 27, 2024 · 3 comments

Comments

@goatchurchprime
Copy link
Owner

The AudioStreamOpusChunked object is derived from AudioStreamPlaybackResampled so that the playback speed can be adjusted when the incoming buffer gets too full. This is likely to happen when there has been a delay in one of the incoming packets and the playback point gets shifted back in time. Without a means for playing the buffer back a little faster, it cannot catch up. While a slight resampling speedup works, the momentary higher pitch voice is distracting. It would be better to run it through an audio stretch library to speed it up without varying the pitch. An example of one is here: https://github.com/dbry/audio-stretch

Depending on how fast it can respond, the same library could slow down the speed of the speech when there is a holdup in the data stream beyond a single packet (the FEC Opus flag lets us skip only one packet), which would mean we could run the speech transmission with less delay and latency as long as the gaps are not noticeable. Before doing this, it would be worth gathering data about the transmission times of each packet, because if the variation is big (every so often there is a delay of one second) this technique isn't going to work if it does something like stretch out someone's vowel for a strangely long time.

This is the final component required for the total solution for VOIP

@ajlennon
Copy link
Collaborator

tbh I really don't like the audio stretching. I would like to have the option to turn it on and off

@ajlennon
Copy link
Collaborator

I'd also like to see some telemetry data on when there's this problem occurring so we can look at how to eliminate it. Like dropped or delayed packets or something

@goatchurchprime
Copy link
Owner Author

Having discovered that this feature is a implemented as a default setting in the HTML5 standard, I have submitted it as a proposal for the core of the Godot engine since it might have a much wider application: godotengine/godot-proposals#10574

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants