Push-to-talk in video/voice conferencing

[BloodyIron]

Whether it's Freeswitch or Jitsi (AFAIK Freeswitch is on the way out), having Push-To-Talk for channel chat would be seriously appreciated! Right now if you have a channel with a whole lot of people wanting to talk, it can turn into a really big mess.

Also, this PTT feature would need to work when the "app" doesn't have focus. I'm thinking this may not work unless the "desktop" version is used, but I'm not 100% up to speed with browser features like this.

It might also be a good idea to get a keyboard bind to toggle self-muting of your own microphone. Let's say, if your fat cat knocks a glass of water all over your desk, oh noes!

[t3chguy]

Also, this PTT feature would need to work when the "app" doesn't have focus. I'm thinking this may not work unless the "desktop" version is used, but I'm not 100% up to speed with browser features like this.

yes this would not be possible in-browser, Discord has the same issue

[BloodyIron]

@t3chguy yeah, I wasn't sure if that particular detail was "solved" by browser devs since I last looked at it. ;)

[t3chguy]

as far as I'm aware its done by intent, so web applications can't behave like keyloggers

[BloodyIron]

@t3chguy that... is actually completely reasonable! I hadn't thought of that. Please don't keylog me Facebook! ;D

[turt2live]

ftr this is listed in the voip metabug: https://github.com/vector-im/riot-web/issues/3025#issuecomment-287929855

[BloodyIron]

@turt2live when I found that thread, I wasn't sure if the PTT facet would get lost in the size of that thread. :( So, I opened this one.

Along the lines of the reason why we have individual issues, instead of just one big meta issue for every single thing people want. ;P

[turt2live]

I'm not declaring a duplicate, just providing a reference so people can find it more easily.

[BloodyIron]

@turt2live you monster! :O

[josephtocci]

Nextel PTT (Always online, push notifications with audio, Example: Zello)
Remember this? I was too young to have a phone, but my parents each had one and the PTT was far above everything else at the time. There were no smartphones at the time. It was faster than calling, and usually the message got through. These days texting won out, partially because if you put your phone down and came back to it you could just read it. PTT you might miss. The other reason texting won out is because you don't necessarily want the person you are talking too IRL to hear what the person on the phone said. There are pros and cons to this method. There is a decent following on Zello which does this. In fact I know someone who uses Zello with all his camping buddies.

Mumble PTT (Must be running, no push notifications for audio, login/join/accept call to use and hear PTT)
This one is more what people will do now. Especially gamers. Even my friend on Zello would probably use this instead. The way he uses Zello is to schedule an hour with everybody, and everybody goes online at the same time and talks. There is no reason to use the always on Zello if you are just going to chatroom with voice. Basically you just mute the microphone, and have a keyboard button to hold down to unmute. (Headphone button on phone) If someone is trying to be quiet IRL and can't get around it, he can plug his headphones in to hear, and type in the room instead of talking. Optionally a text-to-speech thing would be cool, but that is a separate subject.

My thoughts
I think the Mumble type PTT is what people actually seem to need. You set up a room for the game, and everyone logs on and plays the game. (Not necessarily games but games is the common example) Text is king unless you are doing something with your hands, if you are doing something with your hands, set up a call. It's not complicated.

Forward
So I recommend a keyboard button for desktop and headphone button for phone would be excellent shortcuts to temporarily unmute the microphone while it is held down for both native calls and jitsi conferencing. I would recommend Riot to certain people if it had that.

[t3chguy]

headphone button for phone

/me points at vector-im/riot-ios and vector-im/riot-android

[josephtocci]

Android issue created:
element-hq/riot-android#1905

iPhone issue created:
element-hq/element-ios#1736

[josephtocci]

TL/DR, turn the camera on when you press the unmute button, unless you want an audio only conference call. Would reduce load

You could even make the same button to unmute the microphone to turn on the camera. I don't necessarily want my camera on all the time, but when I'm talking I want it on. So have the camera on but don't send anything to matrix unless I hold down the PTT button to unmute the microphone. You could make this the default for conference calls and not for 1-on-1. Does that solve the infrastructure problem? Maybe sometime in the future add the Nextel PTT, it is awesome, but if you say it's too much for now then ok.

It reduces load, and you can have an audio PTT only/no camera conference option for gamers, and a full conference call option for people that have 20 eyes to look at 20 different camera feeds for their friends. lol. Probably the best option considering infrastructure, and nothing has to change server side because load should go down even if you add a healthy amount of users.

[BloodyIron]

So any word when we can see push to talk for riot?

[BloodyIron]

Where is this on the radar? I know it's listed p3, but I'm not sure how that falls in terms of timeline of being looked at.

[t3chguy]

This would have to be implemented by Jitsi. (as Jitsi is what we use for Conferencing)

[BloodyIron]

Any chance the Riot team can co-ordinate that with Jitsi?

[BloodyIron]

Well, hoping I can appeal to them : jitsi/jitsi-meet#3464

If anyone wants to help, please post her and in the linked jitsi thread. This really needs to get implemented. Not being able to change the shortcut is pretty unacceptable.

[anoadragon453]

Just a heads up that I'm making major progress with this, and hope to have basic PTT functionality implemented very soon :)

[anoadragon453]

Got a very good chunk done tonight. Getting the functionality working required some extra PRs in other projects, but now I have everything I need to see it through.

Just working out the last fiddly bits, but I will do so tomorrow :)

[anoadragon453]

So I've managed to get proper functionality working! You can see a short and silly demo here: https://matrix.to/#/!DdJkzRliezrwpNebLk:matrix.org/$1541185902690rkqXW:amorgan.xyz

There's currently two problems however:

1. You're currently unable to edit the shortcut
2. The shortcut is registered when you add a Jitsi widget, not when you join in to the call, meaning it might not register in all cases

I plan to fix both of those (hopefully today) by integrating into Riot's settings, which I should be able to use to just register the shortcut on start-up and/or when the functionality is enabled.

Accomplishing all this required some PRs in an extra repo, notably https://github.com/WilixLead/iohook, and as such we're currently waiting for those to make it to master, which one already has.

So yes, functionality is nearing close, exciting!

[BloodyIron]

@anoadragon453 wait... you can link to Riot URLs, didn't even know that...

As for your work, woot! Awesome!

Does this work when the app is minimized/not in focus?

Also, for the shortcut change, are you going to push that upstream to jitsi as per your comment in : jitsi/jitsi-meet#3464 ?

Woot! Thanks for your hard work @anoadragon453 ! :DD

[anoadragon453]

Thank you!

Does this work when the app is minimized/not in focus?

Yep!

Also, for the shortcut change, are you going to push that upstream to jitsi as per your comment in : jitsi/jitsi-meet#3464 ?

So turns out the jitsi people were right and we can just use their iframe API to trigger the change. They do, unhelpfully, only have a function for toggling audio, so for muting we first have to query whether the audio is already muted, and if not then toggle.

So essentially what I'm doing here is useful for electron apps that want to embed Jitsi and have PTT functionality to control it, but I'm not actually changing anything in the jitsi project itself.

[kuipumu]

Is there any current library or client that is able to do this at the moment?

[BloodyIron]

With Element Call coming, will we finally get Push To Talk for VoIP???

[anoadragon453]

@BloodyIron Technically, likely not. The hard part of this feature is capturing the press and release of a keyboard shortcut globally even when Element is minimized. Sending the signal up to either a widget or an Element Call class is relatively easy.

However, with calls happening more inside Element as a result of integrating Element Call, this feature may become higher priority for the core team (though PRs always welcome!).

[BloodyIron]

But how can applications like Discord achieve that and Element cannot?

Man I have so much on my plate, and I am nowhere near up to speed with writing good code for something like this :/

[anoadragon453]

But how can applications like Discord achieve that and Element cannot?

@BloodyIron I'm not sure how Discord does it as it's proprietary software, but this feature is technically possible. See the comment history of this issue and my old PRs for a solution that at one point worked.

[t3chguy]

Discord has a much larger team and a lot more native code.

[anoadragon453]

Apparently there is a new native node module which looks far less fiddly than iohook: https://github.com/SnosMe/uiohook-napi

This may simplify the implementation, as getting iohook to build correctly everywhere was one of the big headaches with the original implementation.

[MikeBender222]

PTT functionality is a requirement for me to use a voice chat. I found a workaround solution that lets me use PTT, albeit with some extra steps.

This is a AutoHotKey script that mimics PTT usage. Hopefully this can help others in my position, and perhaps even lead to the implementation of PTT within Element some day.

The downsides:
-Requires AHK, and running an external script.
-This mutes and unmutes your microphone in Windows sound devices. Your microphone isn't just muted in Element, but in all other applications as well.
-Closing the script needs to be done with a hotkey. Otherwise simply closing the script will leave your microphone muted in windows and will need to be unmuted through sound devices settings.

If Element had the functionality, I can see it using AHK as a way of hooking into hotkeys system wide, and Element could have the function to mute/unmute microphone that AHK could trigger in some way.

The script should be saved as .AHK file and run with AutoHotKey (requires installation). Windows only ATM.
https://pastebin.com/raw/n0qfuigN

The script requires figuring out your microphone device ID, which you can do by uncommenting the 2nd line before running the script.

[shinysudo]

I'm going to bump this and second comments from @MikeBender222 on this one. Push-to-talk is an absolute requirement - not just for gamers, but professionals and really any voice or conference functionality. For voice and video conferencing push to talk is considered basic feature parity and is expected functionality.

The only existing workarounds involve AHK scripts some of which are / can be falsely detected as a cheat by Anti-Cheat systems.

Unfortunately this is a big miss and major blocker for adoption of Matrix or Element in general. 150+ comments over 4.5 years without a resolution is very disappointing... hoping this is one day so I can take advantage of what otherwise seems like an incredible platform!

[Heinz14]

To work around that issue and get global PTT, in Windows you can use AutoHotkey with this script:
[https://superuser.com/questions/1478630/turn-on-mic-only-when-key-is-pressed-like-a-global-push-to-talk-windows-10]

[JoshuaFern]

I'm really surprised this hasn't been implemented yet. Every time I look into using Element over Discord this issue has stopped me again and again. Are there any other Matrix clients that have implemented this?

[kongo09]

@BloodyIron asked me here #2320 (comment) on my view about PTT. We are aware of the use cases and also have a number of customers potentially interested in it.

However, due to focus on getting Element Call with full Matrix based e2ee implemented, we're deprioritizing other VoIP features like PTT. While Element Call is making great progress as you might have read here: https://element.io/blog/element-x-now-with-embedded-voip/ it is still far from where we would like it to be. Unfortunately, resources are limited and I fully understand the frustration, but right now this is not on the roadmap.

[BloodyIron]

@BloodyIron asked me here #2320 (comment) on my view about PTT. We are aware of the use cases and also have a number of customers potentially interested in it.

However, due to focus on getting Element Call with full Matrix based e2ee implemented, we're deprioritizing other VoIP features like PTT. While Element Call is making great progress as you might have read here: https://element.io/blog/element-x-now-with-embedded-voip/ it is still far from where we would like it to be. Unfortunately, resources are limited and I fully understand the frustration, but right now this is not on the roadmap.

Thanks for the insights! Still looking forward to PTT. :)

[samjco]

what about using this:
https://github.com/colyseus/demo-push-to-talk

[BloodyIron]

The thing is that @anoadragon453 already wrote a good chunk of PTT code for this multiple years ago, and it died on the vine (last I checked, sorry Anoa). So last I checked this is less about the method/library. But I likely have stale insights.

[iaxat]

Any update to bring discord like voice channel here?

[BloodyIron]

6 years later. I'm pretty sure the devs are ignoring this now, by choice.

[Thatoo]

No, it's most probably because they are busy to dev the next voice/video call system called Element Call that will be integrated to Element apps familly... And for what I recall this system does have a push-to-talk system

[Heinz14]

No, it's most probably because they are busy to dev the next voice/video call system called Element Call that will be integrated to Element apps familly... And for what I recall this system does have a push-to-talk system

That's a very bad excuse for refusing to implement a standard feature for 6 years.
Do you happen to have anything to do with that?

[Thatoo]

Nope, I'm just trying to figure out what can be priority of Element devs, knowing no humans can do everything at the same time.
I have no idea how long time dev PTT on jitsi widget in Element could take but certainly enough for them to need keeping this time to focus on other things like element call (which, again, is going to replace jitsi).

[anoadragon453]

All the VoIP dev is indeed on Element Call. Currently they're working on making it more reliable, see:

• Support MSC4140: Delayed events (Futures) synapse#17326 for informing the homeserver that a user has disconnected from the call, instead of users getting stuck.
• Support MSC3757: Restricting who can overwrite a state event synapse#17513 to allow users to join a call from multiple devices without race conditions.

Once the product is reliable and stable, they can focus on adding new features.

Element Web still lacks the infrastructure to read shortcuts from outside the application and call functions off the back of them. I did some work towards this in matrix-org/matrix-react-sdk#2280, #7709 but it's bit-rotted now.

If someone would like to take that on, that'd be extremely helpful for getting this feature finally landed. Then once Element Call is better integrated into Element Web, it can be hooked up to the shortcut feature.