Sync on Web, now and next of realtime media services on web

[KensakuKOMATSU]

Session description

Real time Web is improving, in this moment we have an ability to build realtime media services on web w/ WebRTC, WebCodecs, Web Audio ... etc.

IMO, one interesting movement for this would be synchronization, which means "time alignment". General case for "time alignment" would be lip-sync, but with Media over QUIC, mainly leveraging to WebCodes and WebTransport, we can develop more synchronization use cases with media streaming and arbitrary time-related data. Examples would be synchronization of realtime text, haptics, midi events in co-watching or stream-media play-out.

With this feature, we believe that the potential of web would be improving more and more, since we can build Real/Virtual orchestrated services. Use cases would be meta verse, live viewing with MIDI data, remote robot operation etc.

For these realtime use cases, clock accuracy and frequency is quite important. And we have several questions for this. For example, requestAnimationFrame() is enough or not for the future use cases? In this breakout session, we will discuss about these topic according to clock and sync on Web.

Session goal

figure out gap for sync on Web.

Additional session chairs (Optional)

No response

Who can attend

Anyone may attend (Default)

IRC channel (Optional)

#sync-on-web

Other sessions where we should avoid scheduling conflicts (Optional)

#62

Instructions for meeting planners (Optional)

No response

Agenda for the meeting.

1. Presentations and demo
2. Discussion
3. Wrap-up

Links to calendar

• 2024-09-25, 13:15 - 14:15

Meeting materials

• Slides
• Minutes

[tpac-breakout-bot]

Thank you for proposing a session!

You may update the session description as needed and at any time before the meeting, but please keep in mind that tooling relies on issue formatting: follow the instructions and leave all headings and other formatting intact in particular. Bots and W3C meeting organizers may also update the description, to fix formatting issues or add links and other relevant information. Please do not revert these changes. Feel free to use comments to raise questions.

Do not expect formal approval; W3C meeting organizers endeavor to schedule all proposed sessions that are in scope for a breakout. Actual scheduling should take place shortly before the meeting.

[aboba]

This session is very interesting. We will also be discussing "timing" in WEBRTC WG and MEDIA WG:

• WebRTC WG "Timing I" session
• WebRTC/MEDIA WG "Timing II" session

[KensakuKOMATSU]

Thanks @aboba !

I'll attend JOINT WG as well ;)

[toomim]

Can you verify that this session is about clock synchronization (e.g. NTP and PTP), not data synchronization (e.g. CRDT, OT, Google Docs, Dropbox)?

[aboba]

@toomim I didn't interpret the session blurb as indicating an interest in either topic.

Media synchronization use cases (e.g. "lip sync") often do not require a global clock, only alignment of media on the global wallclock of the sender. While the RTCP SR packet includes a timestamp with the NTP format, this does not imply a dependence on NTP.

Data mirroring doesn't require time alignment, so it seems quite different from what was described in the session blurb.

Potential "time alignment" use cases could include synchronization of text (e.g. realtime text),, haptics, midi events, co-watching or stream-media playout.

[toomim]

Ohh... this is for media sync, like lip-syncing video! Thank you for explaining! I apparently wasn't able to infer that from the description, although I tried to. It'd help to add that example (lip sync video to audio) in the blurb.

[KensakuKOMATSU]

@aboba and @toomim

Many thanks for supportive comments!! I updated description of this session according to your comments.

[KensakuKOMATSU]

Slide for this session : https://drive.google.com/file/d/1R5sI3gzVAJd2BZz2skB57OjE1l8HXLui/view?usp=drive_link

[fideltian]

About WebTransport, there is a limitation for application from our experiment. The lower end Quic has its congestion control algorithm, which limits the applications to do customize congestion control on Media transmission. We hope there is a customized congestion control inject into WebTransport.