-
Notifications
You must be signed in to change notification settings - Fork 89
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial support for Raise Hand feature #2542
base: livekit
Are you sure you want to change the base?
Conversation
Nice :-)
MatrixRTC is designed to not rely on the actual RTC backend (today it's LiveKit; tomorrow the-next-best-thing) it makes sense to implement those kinds of features with native Matrix primitives. Since the Raise-Hands feature would benefit a lot from being moderated we could start this journey first with a Reactions feature (👍️, 👏, 🤣, ❤️, ...). And here we could use the Matrix reactions feature or similar. |
Thanks for the feedback! I'm also interested in reactions but not message reactions like in MSC2677 - something more like what Jitsi and Teams consider to be reactions: playful animation overlays (+ sound?) that transmit a ephemeral feeling (indeed like 👍️, 👏, 🤣, ❤️ but for video). You mention moderation, which I understand could apply to stuff such as muting, spotlighting and lowering hands for other users - I assume getting people off the call can use existing kick / ban membership events as spec'ed. Under this light, here are a few proposals how this could be done: Option 1 - A new moderation state eventSince a raised hand is a binary state (either a call member has its hand raised or doesn't), we could introduce a room state event ( {
"type": "m.call.moderate",
"sender": "@alice:wonderland.org",
"content": {
"raised_hand": [
"@whiterabbit:wonderland.org",
"@redqueen:wonderland.org"
]
},
"state_key": "",
"event_id": "$143273582443PhrSn:example.org",
"origin_server_ts": 1432735824653,
"room_id": "!jEsUZKDJdhlrceRyVU:example.org",
"unsigned": {
"age": 1234
}
} Clients would add / remove their user's MXID to the list and use it to render accordingly. It could be further used to include other moderation features, like: {
"type": "m.call.moderate",
"sender": "@alice:wonderland.org",
"content": {
"raised_hand": [
"@whiterabbit:wonderland.org"
],
"hard_mute": [
"@whiterabbit:wonderland.org",
"@redqueen:wonderland.org",
"@madhatter:wonderland.org"
],
"spotlight": [
"@alice:wonderland.org"
]
},
"state_key": "",
"event_id": "$143273582443PhrSn:example.org",
"origin_server_ts": 1432735824653,
"room_id": "!jEsUZKDJdhlrceRyVU:example.org",
"unsigned": {
"age": 1234
}
} Having an empty state key, means any user with an adequate power level can update this state event. Room creation should make sure call members have the adequate power level to send Main issue with this approach is that anyone that is allowed to raise a hand is now also able to interfere with any of these moderation options, which is can be problematic. Option 2 - Reactions to membership eventsA raised hand could be an {
"type": "m.reaction",
"sender": "@alice:wonderland.org",
"content": {
"m.relates_to": {
"rel_type": "m.annotation",
"event_id": "$event-id-of-m.room.member-for-alice",
"key": "✋"
}
}
} Un-raising your hand would be achieved with a redaction. With this approach, only the sending user and room moderators would be able to redact the raised hands. Main issue with this approach is how to handle users that join the call after these reactions have been sent, potentially being unable to retrieve them from the room timeline? Option 3 - A new dedicated room state eventSimilar to the first option, but used only for raised hands: {
"type": "m.call.raised_hands",
"sender": "@alice:wonderland.org",
"content": {
"@whiterabbit:wonderland.org": {
"order": 1,
"raised_at_ts": 1432735824653
},
"@redqueen:wonderland.org": {
"order": 2,
"raised_at_ts": 1432736824653
}
},
"state_key": "",
"event_id": "$143273582443PhrSn:example.org",
"origin_server_ts": 1432735824653,
"room_id": "!jEsUZKDJdhlrceRyVU:example.org",
"unsigned": {
"age": 1234
}
} This has the same downside of Option 1 (anyone can add or remove himself and others from the list) but how critical is messing up the intended raised hands state by malicious actors? Such manipulation would be easily handled by kicking / banning them. Anyway, hope this can get the conversation one step further. |
Thank you for your work on this feature! I have a few design suggestions that could help simplify its implementation:
I believe these changes could help streamline the user interface by reducing the number of options available in the bottom action sheet. A quick competitive analysis also showed that similar products often combine these features. What do you think? Looking forward to your thoughts! |
@amshakal Great to hear your feedback. For the time being, my focus will be on the technical implementation of the feature and not worry much about the final UI/UX polish, as that is something that surely Element's Design team will have a strong opinon about. |
So I'm drafting an implementation using annotations and reactions (Option 2) and was wondering if the reaction should be to the Having it linked to the room member state event would open this feature up to non-call use cases - I've read about people being interested in having raised hands in text chat rooms... but maybe thats too much forward thinking? Either way, it's a minor implementation detail. |
|
488fd43
to
e84b0a4
Compare
Updating to a refactored version that uses reactions to call membership events. It works :) @amshakal I wasn't aware you were actually working for Element Design 🤦 I'm sure we can get something done along what you were suggesting. I won't be focusing yet on the reactions feature but we can look at the raise hand emoji suggestion you mentioned. I'm fine with replacing the current SVG with the hand emoji (✋). Let's try that. |
The relations api should solve this. joining a call gives you access to the room state and you can ask for the related raise hand events (even through the widget api) |
3eabce5
to
8d55f8b
Compare
@toger5 it works 👯 I've tested this in Widget mode inside Element Web and seems to work as well, although I'm wondering how as I don't see an implementation for Also, on mobile, it loads and shows reactions from other non-mobile clients but sending isn't working.. wondering how to debug that... |
That is surprising indeed. I also expected that to be a required change (to update the js-sdk matroska widget client to forward relations requests to the host client)
Probably you dont need to do too much debugging here. On mobile we are very strict with the permissions. The raise hand event most likely is just not part of that yet. I am wondering how you can receive them though. |
Yes, relations are queried only on the call's session first load.
🤔 |
4e13c66
to
21d6c8d
Compare
@@ -86,6 +88,22 @@ export const MediaView = forwardRef<HTMLDivElement, Props>( | |||
)} | |||
</div> | |||
<div className={styles.fg}> | |||
{raisedHand && ( | |||
<div className={styles.raisedHand}> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is ugly. Could use some hints on how to improve the emoji rendering style.
On the topic of reactions we might want to explore lottie files. That is probably very little effort and just showing the lottie file over the tile could already look really good. |
@fkwp and I crossed this PR in a meeting and concluded the following:
I will try looking into adjusting the js-sdk matroska mode but there is still the issue that the rust sdk is not handling this special case. |
This is not 100% correct. The js-sdk internally does convert events into redact PDU's internally and hence the widget api already gets the correct event for the So its basically a bug in the matroska mode that is easy enough to fix: The only missing bit now is support in the rust widget driver. Currently the rust sdk wont approve capabilities for the redactions and reactions. |
a5ff087
to
f6b811b
Compare
src/ClientContext.tsx
Outdated
setInitClientState({ | ||
widgetApi: initClientState.widgetApi, | ||
client: initClientState.client, | ||
passwordlessUser: false, | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we can use
setInitClientState({ | |
widgetApi: initClientState.widgetApi, | |
client: initClientState.client, | |
passwordlessUser: false, | |
}); | |
setInitClientState({ | |
...initClientState, | |
passwordlessUser: false, | |
}); |
here.
Signed-off-by: Milton Moura <miltonmoura@gmail.com>
Signed-off-by: Milton Moura <miltonmoura@gmail.com>
Signed-off-by: Milton Moura <miltonmoura@gmail.com>
Signed-off-by: Milton Moura <miltonmoura@gmail.com>
Element Call recently changed to AGPL-3.0
Signed-off-by: Milton Moura <miltonmoura@gmail.com>
… event id and refactors some async code Signed-off-by: Milton Moura <miltonmoura@gmail.com>
Signed-off-by: Milton Moura <miltonmoura@gmail.com>
f6b811b
to
69a50fb
Compare
Signed-off-by: Milton Moura <miltonmoura@gmail.com>
This allows the widget to send and receive event types used by the "raise hand" feature (element-hq/element-call#2542) without prompting the user to grant the capabilities to do so. * Test that call widgets get RoomCreate events
This allows the widget to send and receive event types used by the "raise hand" feature (element-hq/element-call#2542) without prompting the user to grant the capabilities to do so.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I gave this a quick test, and found two minor issues:
- If you join a running call with users who have their hands raised, you won't see their raised hands until they lower & reraise their hands.
- If you raise your hand, hang up, then rejoin a call, you won't see your hand as raised but other users will.
But overall this is very nice! Thanks again for your contribution!
Good catch, I think it's related to the room settings: access to history. If that is configured to "members since invite" we have a timeline access issue. |
* Grant Element Call widget caps for "raise hand" This allows the widget to send and receive event types used by the "raise hand" feature (element-hq/element-call#2542) without prompting the user to grant the capabilities to do so. * Lint
* Grant Element Call widget caps for "raise hand" This allows the widget to send and receive event types used by the "raise hand" feature (element-hq/element-call#2542) without prompting the user to grant the capabilities to do so. * Lint
It isn't, because I ran my experiments for two users (re)joining calls in a single room they are both members of, meaning they both had full access to history of newly-created calls & their reactions. |
Initial draft to implement #130
This implementation initially used LiveKit's API that allows for participants to publish data.
It is now using reactions to call membership events and redactions of said event to toggle raising of hands.
Planned features:
ec-raise-hand.mp4