ScreenGear and WebGear_RTC not working when using monitor=2

[harshitsinghai77]

ScreenGear and WebGear_RTC does not work when using monitor=2

self.stram = ScreenGear(monitor=2, logging=True).start()

Description

# import necessary libs
import uvicorn
from av import VideoFrame
from aiortc import VideoStreamTrack
from vidgear.gears import ScreenGear
from aiortc.mediastreams import MediaStreamError
from vidgear.gears.asyncio import WebGear_RTC

# various performance tweaks
options = {
    "frame_size_reduction": 50,
    "enable_live_broadcast": True,
}   
# initialize WebGear_RTC app without any source
web = WebGear_RTC(logging=True, **options)

# create your own Bare-Minimum Custom Media Server
class Custom_RTCServer(VideoStreamTrack):
    """
    Custom Media Server using OpenCV, an inherit-class
    to aiortc's VideoStreamTrack.
    """

    def __init__(self, source=None):

        # don't forget this line!
        super().__init__()

        # initialize global params
        self.stream = ScreenGear(monitor=2, logging=True).start()

    async def recv(self):
        """
        A coroutine function that yields `av.frame.Frame`.
        """
        # don't forget this function!!!

        # get next timestamp
        pts, time_base = await self.next_timestamp()

        # read video frame
        frame = self.stream.read()

        # if NoneType
        if frame is None:
            return MediaStreamError

        # reducer frames size if you want more performance otherwise comment this line
        # frame = await reducer(frame, percentage=30)  # reduce frame by 30%

        # contruct `av.frame.Frame` from `numpy.nd.array`
        av_frame = VideoFrame.from_ndarray(frame, format="bgr24")
        av_frame.pts = pts
        av_frame.time_base = time_base

        # return `av.frame.Frame`
        return av_frame

    def terminate(self):
        """
        Gracefully terminates VideoGear stream
        """
        # don't forget this function!!!

        # terminate
        if not (self.stream is None):
            self.stream.stop()
            self.stream = None


# assign your custom media server to config with adequate source (for e.g. foo.mp4)
web.config["server"] = Custom_RTCServer()

# run this app on Uvicorn server at address http://localhost:8000/
uvicorn.run(web(), host="localhost", port=8000)

# close app safely
web.shutdown()

This doesn't work.

But using self.stream = ScreenGear(logging=True).start() works.

The problem is when using
But using self.stream = ScreenGear(monitor=2, logging=True).start()

Acknowledgment

• [x ] I have searched the issues for my issue and found nothing related or helpful.
• [ x] I have read the Documentation.
• [ x] I have read the Issue Guidelines.

Environment

• VidGear version: 0.2.2
• Branch: master
• Python version: 3.9.5
• PiP version: 20.3.4
• Operating System and version: Linux

Expected Behavior

Only monitor 2 screen should be shared.

[abhiTronix]

@harshitsinghai77 Is logging stuck at WebGear_RTC :: DEBUG :: ICE connection state is completed ?

[harshitsinghai77]

Yes.

[abhiTronix]

@harshitsinghai77 Ok I've found the bug. And you're right about problem being with av_frame = VideoFrame.from_ndarray(frame, format="bgr24") line.

Bug:

Turned out ScreenGear was outputting BGRA(4 channel frame) format instead of default BGR(3 channel frame), when monitor value is defined (!=None). Hence, VideoFrame.from_ndarray function cannot work with "bgr24" colorspace since it requires 3 channels and not 4-channel frame at its input and this resut in AssertionError:

aiortc.rtcrtpsender :: WARNING :: RTCRtpsender(video) Traceback (most recent call last):
  File "C:\Users\foo\AppData\Roaming\Python\Python38\site-packages\aiortc\rtcrtpsender.py", line 295, in _run_rtp
    payloads, timestamp = await self._next_encoded_frame(codec)
  File "C:\Users\foo\AppData\Roaming\Python\Python38\site-packages\aiortc\rtcrtpsender.py", line 248, in _next_encoded_frame
    frame = await self.__track.recv()
  File "p.py", line 56, in recv
    av_frame = VideoFrame.from_ndarray(frame, format="bgr24")
  File "av\video\frame.pyx", line 337, in av.video.frame.VideoFrame.from_ndarray
AssertionError

Solution:

The solution is just to change colorspace based on number of channel in frame. Hence following code will work for you:

# import necessary libs
import uvicorn, asyncio, cv2
from av import VideoFrame
from aiortc import VideoStreamTrack
from aiortc.mediastreams import MediaStreamError
from vidgear.gears.asyncio import WebGear_RTC
from vidgear.gears import ScreenGear
from vidgear.gears.asyncio.helper import reducer

# various performance tweaks
options = {
    "frame_size_reduction": 25,
}   
# initialize WebGear_RTC app without any source
web = WebGear_RTC(logging=True, **options)

# create your own Bare-Minimum Custom Media Server
class Custom_RTCServer(VideoStreamTrack):
    """
    Custom Media Server using OpenCV, an inherit-class
    to aiortc's VideoStreamTrack.
    """

    def __init__(self):

        # don't forget this line!
        super().__init__()

        # initialize global params
        self.stream = ScreenGear(monitor=2, logging=True).start()
        self.stream.read()

    async def recv(self):
        """
        A coroutine function that yields `av.frame.Frame`.
        """
        # don't forget this function!!!

        # get next timestamp
        pts, time_base = await self.next_timestamp()

        # read video frame
        frame = self.stream.read()

        # if NoneType
        if frame is None:
            return MediaStreamError

        # reducer frames size if you want more performance otherwise comment this line
        frame = await reducer(frame, percentage=10)  # reduce frame by 10%

        # contruct `av.frame.Frame` from `numpy.nd.array`
        if frame.shape[-1] == 4:
            av_frame = VideoFrame.from_ndarray(frame, format="bgra")
        else:
            av_frame = VideoFrame.from_ndarray(frame, format="bgr24")
        av_frame.pts = pts
        av_frame.time_base = time_base

        # return `av.frame.Frame`
        return av_frame

    def terminate(self):
        """
        Gracefully terminates VideoGear stream
        """
        # don't forget this function!!!

        # terminate
        if not (self.stream is None):
            self.stream.stop()
            self.stream = None


# assign your custom media server to config with adequate source (for e.g. foo.mp4)
web.config["server"] = Custom_RTCServer()

# run this app on Uvicorn server at address http://localhost:8000/
uvicorn.run(web(), host="localhost", port=8000)

# close app safely
web.shutdown()

[abhiTronix]

Successfully resolved and merged in commit 7cbc1bd

ScreenGear with WebGear_RTC Example is available here: https://abhitronix.github.io/vidgear/v0.2.3-dev/help/screengear_ex/#using-screengear-with-webgear_rtc

[harshitsinghai77]

Hey, Thanks for this. I was caught up in some work. This looks good. Thanks again for fixing it so soon. Really great work.. Keep up the good work. I really like the project. 🚀🚀🚀