MJPEG multipart/x-mixed-replace image stream flashes/flickers

[jameshilliard]

When using wpewebkit 2.22.3 with the cog launcher with the following options the mjpg-streamer jpeg image stream flashes rapidly as new images are received:

/bin/cog --platform=fdo http://127.0.0.1:8100/stream.html

I tried a number of options and for some reason using this prevents the flashing and shows the proper stream without flickering(there are grid overlay lines present when doing so however):

/bin/cog --platform=fdo --draw-compositing-indicators=true http://127.0.0.1:8100/stream.html

Any idea why setting --draw-compositing-indicators=true prevents the stream from flickering rapidly and how I might get it to stop flickering without the compositing indicator grid?

[aperezdc]

Thanks for pointing out the setting for the draw-compositing-indicators setting, which helps narrowing down the cause. Most likely the additional painting done to have the compositing indicators drawn (inside WebKit) is doing something that somehow better synchronizes the frames, or maybe it prevents a blank frame from being painted. I think @magomez and/or @zdobersek may have some idea on how to further investigate the issue.

@jameshilliard Could you please indicate which options are you using when running mjpg-streamer? I would like to use the same settings as you to check whether the issue is also present in upstream WebKit.

[jameshilliard]

Could you please indicate which options are you using when running mjpg-streamer?

I'm running mjpg-streamer with these options:

/bin/mjpg_streamer -i "/usr/lib/mjpg-streamer/input_uvc.so -n -r 1280x720 -d /dev/ffloopcam" -o "/usr/lib/mjpg-streamer/output_http.so -w /usr/share/mjpg-streamer/www -p 8100 -l 0.0.0.0"

The bug happens with both the <img src="./?action=stream" /> multipart/x-mixed-replace mjpeg stream and the javascript version that works by replacing the image individually from the /?action=snapshot endpoint.

I would like to use the same settings as you to check whether the issue is also present in upstream WebKit.

I did check using the latest safari, chrome and firefox and none had this bug.

[jameshilliard]

So I did manage to fix the flicker for the javascript stream with this change. This doesn't seem to effect the multipart/x-mixed-replace flicker bug however.
I'm launching wpewebkit with this option for the javascript stream:

/bin/cog --platform=fdo http://127.0.0.1:8100/javascript.html

My guess would be that this bug has something to do with the images from mjpeg-streamer coming in faster than wpewebkit is capable of rendering them which is causing the background to flash white.

[jameshilliard]

I came up with another hack to workaround the bug using HTML5 canvas with the multipart/x-mixed-replace stream. Hopefully this helps narrow down the root cause.

import React, { Component } from "react";
import { FFCAM_SERVER_URL, IDCAM_SERVER_URL } from "../constants";

class FFCamera extends Component {
    constructor(props){
        super(props)
        this.state = {
            stream: FFCAM_SERVER_URL + '/?action=stream',
            loaded: FFCAM_SERVER_URL + '/?action=stream'
        }
    }
    componentDidMount() {
        const canvas = this.refs.canvas
        const ctx = canvas.getContext("2d")
        const img = this.refs.image
        const drawInterval = 30;
        function draw() {
            ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
        }
        img.onload = () => {
            draw();
            setInterval(draw, drawInterval);
        }
    }
    reloadStream = () => {
        this.setState({loaded: ''})
        setTimeout(() => {
            this.setState({loaded: this.state.stream})
        }, 0)
    }
    render() {
        return (
            <div>
                <canvas ref="canvas" width={640} height={425} />
                <img ref="image" onError={this.reloadStream} src={this.state.loaded} style={{visibility: 'hidden'}} />
            </div>
        );
    }
}

export default FFCamera;

[tobias-grasse]

I can confirm the flickering issue running WPE WebKit v2.24.3 with wpebackend-fdo 1.2.3 and libwpe 1.2.x. I can also confirm both workarounds that @jameshilliard posted do resolve the issue (though the draw-compositing-indicators isn't really an option for production).

I don't have a local MJPEG stream, but testing with this public cam showed the same behavior:
http://217.7.233.140/cgi-bin/faststream.jpg?stream=full
It also has the multipart/x-mixed-replace content type.

Another cam at http://194.103.218.15/mjpg/video.mjpg which has just image/jpeg also flickered on WPE, which was resolved with @jameshilliard 's canvas workaround.

Will update to the latest stable WPE + libs next week and check if the issue persists.

[Edited since I referenced the wrong WPE versions initially]

[tobias-grasse]

I just tested with the latest WPE 2.26.x | libwpe 1.4.x | backend-fdo 1.4.x | cog 0.4.x and the flicker issue still occurs across all tested MJPEG streams. Appreciate any pointer where to start debugging @magomez / @zdobersek 😊

[urothis]

@tobias-grasse did you make any progress on this issue? I've ran into the same issue of wpe not being able to handle the amount of frames I'm sending from the rtmp stream.

[tobias-grasse]

did you make any progress on this issue?

@urothis unfortunately not; but most streams are working flicker-free with the canvas workaround. It's significant overhead compared to just <img src="">, but my use case requires some JS logic to control the stream anyway. Note I'm still testing against WPE 2.26.4 though; 2.28.x fails to build on armhf architecture snapcraft builders. Will update here if that version bump changes anything w.r.t. this issue.

[tobias-grasse]

Unfortunately, this is still an issue with WPE 2.30.5 / cog 0.8.1 and libwpe/wpebackend-fdo 1.8.0.

[gruvin]

Easy workaround; Turning off Chrome's Settings => Advanced => System => "Use hardware acceleration when available" fixed this for me. ymmv

[kamk]

I have tried to continuously reload image frame by frame with this code:

<!DOCTYPE html>
<html>
  <head>
    <title>Frame by frame test</title>
    <script>
      function imgLoaded(image) {
        setTimeout(() => {
          image.src = "http://localhost:3000/camera-frame/0?ts=" + Date.now();
        }, 40);
      }
    </script>
  </head>

  <body>
    <img onload="imgLoaded(this)" src="http://localhost:3000/camera-frame/0" />
  </body>
</html>

This doesn't resolve the problem with flickering so it can be something with <img> tag rendering.

Snaps used:

Name                  Version         Rev    Tracking       Publisher   Notes
...
ubuntu-frame          90-mir2.10.0    4455   latest/stable  canonical✓  -
wpe-webkit-mir-kiosk  2.38.2          86     latest/stable  glancr      -
...