WebRTC Server Asp.Net.Core

[steveo555]

I'm wondering if someone can point me in the right direction to replicate the following scenario.

ASP.Net.Core Web App .Net 5.0

Two pages Host and Client
Host creates a peer connection and sends webcam data server (hosted service) (while displaying it in video element as per usual)
Client connects and receives the video from the host in a video element as per usual

I've got no problems doing this directly (peer to peer) but when I try going via the server I'm getting not video (connection is good) I assume its ice related but I think I'm just not getting the concept of the server side peers.

My hosted service code is taken from the examples here.

 private static List<SDPAudioVideoMediaFormat> AudioOfferFormats = new List<SDPAudioVideoMediaFormat> {
            new SDPAudioVideoMediaFormat(SDPMediaTypesEnum.audio, 111, "OPUS", 48000, 2, "minptime=10;useinbandfec=1")
        };
        private static List<SDPAudioVideoMediaFormat> VideoOfferFormats = new List<SDPAudioVideoMediaFormat> {
            new SDPAudioVideoMediaFormat(SDPMediaTypesEnum.video, 98, "VP8", 90000)
        };

        private RTCConfiguration getConfig()
        {
            RTCConfiguration rTCConfiguration = new RTCConfiguration()
            {
                iceServers = new List<RTCIceServer> { new RTCIceServer { urls = "stun:stun.l.google.com:19302" },
                                                      new RTCIceServer { urls = "stun:stun1.l.google.com:19302" },
                                                      new RTCIceServer { urls = "stun:stun2.l.google.com:19302" },
                                                      new RTCIceServer { urls = "stun:stun3.l.google.com:19302" },
                                                      new RTCIceServer { urls = "stun:stun4.l.google.com:19302" }}
            };
            return rTCConfiguration;
        }

        private readonly ILogger<WebRTCHostedService> _logger;

        private ConcurrentDictionary<string, RTCPeerConnection> _peerConnections = new ConcurrentDictionary<string, RTCPeerConnection>();

        public WebRTCHostedService(ILogger<WebRTCHostedService> logger)
        {
            _logger = logger;
        }

This is my Host offer (it sends video to the client(s) when it gets a frame)

  public async Task<RTCSessionDescriptionInit> GetOffer(string id)
        {
            if (string.IsNullOrWhiteSpace(id))
            {
                throw new ArgumentNullException("id", "A unique ID parameter must be supplied when creating a new peer connection.");
            }
            else if (_peerConnections.ContainsKey(id))
            {
                throw new ArgumentNullException("id", "The specified peer connection ID is already in use.");
            }
            var peerConnection = new RTCPeerConnection(getConfig());
            MediaStreamTrack videoTrack = new MediaStreamTrack(SDPMediaTypesEnum.video, false, VideoOfferFormats, MediaStreamStatusEnum.RecvOnly);
            peerConnection.addTrack(videoTrack);


            peerConnection.OnVideoFrameReceived += (IPEndPoint remoteEP, uint timestamp, byte[] frame) =>
            {
                foreach (var rPC in _peerConnections)
                {
                    if (rPC.Key != id)
                    {
                        rPC.Value.SendVideo(3000, frame);
                    }
                }
            };

            peerConnection.OnTimeout += (mediaType) => _logger.LogWarning($"Timeout for {mediaType}.");
            peerConnection.onconnectionstatechange += (state) =>
            {
                _logger.LogDebug($"Peer connection {id} state changed to {state}.");

                if (state == RTCPeerConnectionState.closed || state == RTCPeerConnectionState.disconnected || state == RTCPeerConnectionState.failed)
                {
                    _peerConnections.TryRemove(id, out _);
                }
                else if (state == RTCPeerConnectionState.connected)
                {
                    _logger.LogDebug("Peer connection connected.");
                }
            };

            var offerSdp = peerConnection.createOffer(null);
            await peerConnection.setLocalDescription(offerSdp);

            _peerConnections.TryAdd(id, peerConnection);

            return offerSdp;
        }

and this is my client (receiving) offer

        public async Task<RTCSessionDescriptionInit> GetOfferRec(string id)
        {
            if (string.IsNullOrWhiteSpace(id))
            {
                throw new ArgumentNullException("id", "A unique ID parameter must be supplied when creating a new peer connection.");
            }
            else if (_peerConnections.ContainsKey(id))
            {
                throw new ArgumentNullException("id", "The specified peer connection ID is already in use.");
            }
            var peerConnection = new RTCPeerConnection(getConfig());
            MediaStreamTrack videoTrack = new MediaStreamTrack(SDPMediaTypesEnum.video, false, VideoOfferFormats, MediaStreamStatusEnum.SendOnly);
            peerConnection.addTrack(videoTrack);

            peerConnection.OnRtpPacketReceived += (IPEndPoint rep, SDPMediaTypesEnum media, RTPPacket rtpPkt) =>
            {
                _logger.LogDebug($"RTP {media} pkt received, SSRC {rtpPkt.Header.SyncSource}, SeqNum {rtpPkt.Header.SequenceNumber}.");

            };

            peerConnection.OnTimeout += (mediaType) => _logger.LogWarning($"Timeout for {mediaType}.");
            peerConnection.onconnectionstatechange += (state) =>
            {
                _logger.LogDebug($"Peer connection {id} state changed to {state}.");

                if (state == RTCPeerConnectionState.closed || state == RTCPeerConnectionState.disconnected || state == RTCPeerConnectionState.failed)
                {
                    _peerConnections.TryRemove(id, out _);
                }
                else if (state == RTCPeerConnectionState.connected)
                {
                    _logger.LogDebug("Peer connection connected.");
                }
            };

            var offerSdp = peerConnection.createOffer(null);
            await peerConnection.setLocalDescription(offerSdp);

            _peerConnections.TryAdd(id, peerConnection);

            return offerSdp;
        }

AddIceCandidate..

 public void AddIceCandidate(string id, RTCIceCandidateInit iceCandidate)
        {
            if (!_peerConnections.TryGetValue(id, out var pc))
            {
                throw new ApplicationException("No peer connection is available for the specified id.");
            }
            else
            {
                _logger.LogDebug("ICE Candidate: " + iceCandidate.candidate);
                pc.addIceCandidate(iceCandidate);
            }
        }

I'm sure what I'm trying to do is simple but I'm lost. Ultimately I'm trying to get multiple clients to view the live camera feed from the Host through the server rather than multiple peers connections from the host to each client.

Any help would be gratefully appreciated.

[sipsorcery]

Do you have a repository for your app? I'd be happy to give it a try but don't fancy manually cobbling it into an ASP.NET Core app.

Note that I've personally never tested OPUS with the library. It may just work but there also maybe some hidden gotchas.

In regards to your design am I right that's it:

1. Establish a WebRTC connection from a Source Peer to your ASP.NET Host app. The Source Peer is providing a sendonly audio and video stream to the Host.

2. Additional Destination Peers establish WebRTC connections to the Host. The Host forwards the audio and video stream from the Source Peer to the Destination Peers.

Is that correct?

That architecture is the same as a WebRTC Selective Forwarding Unit (SFU) albeit without the smarts of selecting amongst multiple streams.

[steveo555]

Thanks for getting back to me 👍

My repo is a bit of a bomper cos its part of a larger project but I can knock up a test scenario if you don't mind taking a look?

You're right about my scenario. I don't know if that the best way but I'm trying to replicate what I've already got peer to peer but introducing the SipSorcery server side peers to forward the data.

The background to my problem is that I have a meeting element to my system for teachers and pupils to interact. It works fine for 3 or 4 people but once you get more that 5 it glitches and slows down due to the payload of the uploads from all the peers to each other. My idea was that I was hoping I could use the SipSorcery server side peers to to the multiple forwarding the Clients only send their feed once and the server sends the video to all connected essentially becoming a media server? I don't know if this is stupid or not but in my head it makes sense?!

Am I right in thinking that when a Javascript peer connects to the asp.net peer it is essentially a peer to peer connection of a web client to the asp.net server? I

1. JS Client 1 connects (Teacher) to asp.net server side peer, exchanges ICE and starts sending AV
2. JS Client 2 connects (Pupil) to asp.net server side peer, exchanges ICE
3. server side code detects a pupil has connected and forwards the AV from Client 1 to Client 2
4. JS Client 3 connects (Pupil 2) to asp.net server side peer, exchanges ICE
5. server side code detects additional pupil and on receiving data from Client 1 forwards payload to Client 1 and Client 2

That's it in a nutshell. Is this possible or am I barking up the wrong tree?

Sorry for the waffle.

Cheers

Steve

[sipsorcery]

Yep happy to take a look and your approach sounds correct.

Am I right in thinking that when a Javascript peer connects to the asp.net peer it is essentially a peer to peer connection of a web client to the asp.net server?

Yes. Each javascript RTCPeerConnection will establish a peer-to-peer connection with a SIPSorcery.Net.RTCPeerConnection instance which is most likely running as a HostedService in your ASP.NET Core application.

In the diagram below you're switching from Mesh to SFU and potentially not dealing with streams from the Students' peers.

I'm sure you've researched this a bit but it's worth reiterating. The WebRTC portions of the sipsorcery library are still under heavy development. You can and should expect to encounter issues when using it. If you have an immediate need to implement a SFU in a production application there are mature options such as Janus, mediasoup and others.

[steveo555]

Yes that's exactly what I'm trying to achieve. I appreciate its all still under development (as is my project so no worries). A long time ago I was working on the TCP/IP coms over satellite so I'm broadly familiar with coms dev (albeit majorly out of practice!) so I can probably get to grips with what's going on inside.

I'll strip out the sections dealing with the WebRTC and pop it on here.

Thanks again for the assist. 👍

[steveo555]

In creating the test project for you I've solved my own problem! I was sending the keyframe to the wrong peer when connected! Now it works as suggested above. Great project. I'll keep going and post any issues on here.

Thanks again 👍👍👍

[sipsorcery]

Glad to hear it. Please do post back with how you get on. One other person asked about using the sipsorcery library to implement an SFU. It would be good to have an experience to point other developers to.

[steveo555]

Will do. I've got it working in basic manner on a local IIS, publishing to Azure has stopped the connection working. The java peer is connecting to the server but the ice fails. I'll keep working on it and once I've solved it I'll post it up here.

onicegatheringstatechange: gathering
jsDoStuff.js:159 oniceconnectionstatechange: checking
jsDoStuff.js:168 new-ice-candidate:
jsDoStuff.js:169 candidate:2072139682 1 udp 2122260223 192.168.1.59 50741 typ host generation 0 ufrag +/FQ network-id 1
jsDoStuff.js:170 RTCIceCandidate {candidate: "candidate:2072139682 1 udp 2122260223 192.168.1.59…741 typ host generation 0 ufrag +/FQ network-id 1", sdpMid: "0", sdpMLineIndex: 0, foundation: "2072139682", component: "rtp", …}address: "192.168.1.59"candidate: "candidate:2072139682 1 udp 2122260223 192.168.1.59 50741 typ host generation 0 ufrag +/FQ network-id 1"component: "rtp"foundation: "2072139682"port: 50741priority: 2122260223protocol: "udp"relatedAddress: nullrelatedPort: nullsdpMLineIndex: 0sdpMid: "0"tcpType: nulltype: "host"usernameFragment: "+/FQ"__proto__: RTCIceCandidate
jsDoStuff.js:168 new-ice-candidate:
jsDoStuff.js:169 candidate:889301842 1 tcp 1518280447 192.168.1.59 9 typ host tcptype active generation 0 ufrag +/FQ network-id 1
jsDoStuff.js:170 RTCIceCandidate {candidate: "candidate:889301842 1 tcp 1518280447 192.168.1.59 …ptype active generation 0 ufrag +/FQ network-id 1", sdpMid: "0", sdpMLineIndex: 0, foundation: "889301842", component: "rtp", …}address: "192.168.1.59"candidate: "candidate:889301842 1 tcp 1518280447 192.168.1.59 9 typ host tcptype active generation 0 ufrag +/FQ network-id 1"component: "rtp"foundation: "889301842"port: 9priority: 1518280447protocol: "tcp"relatedAddress: nullrelatedPort: nullsdpMLineIndex: 0sdpMid: "0"tcpType: "active"type: "host"usernameFragment: "+/FQ"__proto__: RTCIceCandidate
jsDoStuff.js:168 new-ice-candidate:
jsDoStuff.js:169 candidate:2375167606 1 udp 1686052607 82.15.160.222 50741 typ srflx raddr 192.168.1.59 rport 50741 generation 0 ufrag +/FQ network-id 1
jsDoStuff.js:170 RTCIceCandidate {candidate: "candidate:2375167606 1 udp 1686052607 82.15.160.22… rport 50741 generation 0 ufrag +/FQ network-id 1", sdpMid: "0", sdpMLineIndex: 0, foundation: "2375167606", component: "rtp", …}address: "82.15.160.222"candidate: "candidate:2375167606 1 udp 1686052607 82.15.160.222 50741 typ srflx raddr 192.168.1.59 rport 50741 generation 0 ufrag +/FQ network-id 1"component: "rtp"foundation: "2375167606"port: 50741priority: 1686052607protocol: "udp"relatedAddress: "192.168.1.59"relatedPort: 50741sdpMLineIndex: 0sdpMid: "0"tcpType: nulltype: "srflx"usernameFragment: "+/FQ"__proto__: RTCIceCandidate
jsDoStuff.js:155 onicegatheringstatechange: complete
jsDoStuff.js:159 oniceconnectionstatechange: disconnected

[sipsorcery]

Azure will make things more complicated because all the Vm's and Apps are put behind NAT. But ICE should be able to deal with that by using the STUN servers to get the srflx address as you are doing.

My next suspicion would be the RTP port getting blocked. Make sure your Azure Network Security Group accepts UDP 49000-65535. That's the ephemeral port range the OS should select the RTP port from.

[sipsorcery]

As a note for any interested parties this ASP.NET Core application, signalrtc, was developed to test out running some VoIP and WebRTC bits and pieces on Azure. It's currently running in demo mode at sipsorcery.cloud.

Fundamentally there's no problem running VoIP And WebRTC services on Azure, however, a big shortcoming is the Azure Load Balancer is not suitable for a SIP service that needs to load balance UDP traffic across multiple servers. Other than that it works very well and I currently manage to run two VM's with a bunch of different VoIP and WebRTC applications for about EUR30/month.