[discussion] Configurable audio behavior

[cobarx]

One of the most requested features is greater configurability on how the audio mixing should behave. This bug is intended to be a place to first discuss how the configuration should work, then track implementation.

There are a number of use cases people have requested including:

• Normal video playback (mutes all audio sources)
• Music playback mixed with other audio sources
• Video playback with audio muted
• Video playback and record audio from the same app

Here is an overview of the behavior on mobile platforms. We can ignore Windows since the audio channel is always shared.

iOS offers the following modes:

Category	Silenced by Silent switch & screen locking	Interrupts nonmixable app’s audio	Allows input (recording) & output (playback)
AVAudioSessionCategoryAmbient	Yes	No	Output only
AVAudioSessionCategorySoloAmbient (Default)	Yes	Yes	Output only
AVAudioSessionCategoryPlayback	No	Yes by default; no by using override switch	Output only
AVAudioSessionCategoryPlayAndRecord	No	Yes by default; no by using override switch	Input and output

We can ignore the following options:

Category	Silenced by the Ring/Silent switch and by screen locking see note	Interrupts nonmixable app’s audio	Allows audio input (recording) and output (playback)
AVAudioSessionCategoryRecord	No (recording continues with screen locked)	Yes	Input only
AVAudioSessionCategoryMultiRoute	No	Yes	Input and output

AVAudioSessionCategoryRecord: input only
AVAudioSessionCategoryMultiRoute: currently this project doesn't support multiple output devices

Sample configuration call:
[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryAmbient error:nil];

On Android, there is a concept of audio focus. Each type of audio stream (phone call, media, notifications) can have one app that is using that stream at the same time. Focus should be acquired when playback is about to start and released when media pauses.

Sample configuration call:
requestAudioFocus(listener, STREAM_MUSIC, AUDIOFOCUS_GAIN);

Ducking:
"ducking" is the behavior of the OS lowering the volume of an audio stream to allow other streams to play on top of it. A good example of this is allowing a song to continue playing while playing a text beep.

More to come...need to extensively test behavior when handling notifications like text and incoming phone calls

[cobarx]

I have been looking at the docs for how Expo handles audio mixing:
https://docs.expo.io/versions/v32.0.0/sdk/audio/

Based on this, I think we support 3 props:

ignoreSilentSwitch

Control whether to play audio when the silent switch is on. Already implemented, iOS only

audioRecording

Allow audio recording. iOS only

audioMode

• "Exclusive": default option. Interrupts audio from other sources
• "Mix": Allow other apps to play at the same time as our app. iOS only
• "Duck Others": Lower the volume of other apps

Some combinations of these props will be invalid and will throw an error.

I suspect we will want to make it so that muting or setting the volume to 0 ignores these modes.

I will need to do more testing to see how this working when there are multiple videos playing at the same time.

[evanjmg]

Any update on this? Does anyone have a fork?

[evanjmg]

I assume this needs work? #959

[camel113]

@evanjmg did you find something to manage sound from other apps?

[evanjmg]

@camel113 Unfortunately not, I've had to make my own native library for the feature I wanted - https://github.com/evanjmg/react-native-sound-detection

[CyxouD]

There is a fork of react-native-video branch (https://github.com/Stormotion-Mobile/react-native-video) which partly implements desired behavior both on Android and iOS, my colleague used it in his project and it worked good for him, he needed to switch between different audio modes by pressing button. Note, that most likely it has breaking changes.
Project's info:

react-native: 0.60.11
react-native-video: 4.3.1

It works by introducing audioFocusState prop, which can take 0, 1 , 2 value. You can play with them, but as far as I understand 0: Playback with mixing, 1 is Playback with ducking, 2: Playback.

For my use case, I needed to interrupt 3rd party audio and then resume it after my app's video finished. I used 2 to interrupt and 0 to resume. However, on Android it didn't work (audios were mixed instead), maybe I missed something. It could be broken because I updated my colleague's fork to react-native-video 5.1.0-alpha1, handling merge conflicts. If you need, you can use it here https://github.com/CyxouD/react-native-video, but it could really break something.

[ditorodev]

@CyxouD this fork works great did the same you did 0 when VideoPaused, 2 when not. However, I'm experiencing a bug here when trying to replay the video(seek to 0 and play again) the video gets paused every half second. Any idea on this?

[CyxouD]

@ditorojuan
yes, exactly the same audioFocusState={paused ? LOSE_FOCUS : GAIN_FOCUS} :)
Unfortunately, I don't know why your experienced this issue. By the way, it worked bad on Android (don't remember what issue was exactly)