Camera stream example

[emilk]

We should have an example where we stream data from a web-camera, and doing some simple things to it (e.g. edge detection).

This would demonstrate and test our memory pruning.

Part of this issue is also making memory pruning of such streaming ergonomic, both over network (--connect) and with spawn.

Related:

• Crash on memory pruning #1414

[Erol444]

Hi @emilk,
I just tried this out, looks like rerun quickly eats up whole RAM, especially when using 1080P or 4k. Looks like rerun is holding all previous frames in RAM, is there a way to either a) save them to disk instead or b) discard old frames?
Thanks, Erik

import cv2
import rerun as rr
rr.init("Webcam streaming test", spawn=True, default_enabled=False)
cap = cv2.VideoCapture(0)

# read frames from webcam
while True:
    ok, frame = cap.read()
    if not ok:
        raise Exception('No frame')
    # OpenCV default is BGR, rerun expects RGB
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    rr.log_image("color", image=frame)

[emilk]

Hi @Erol444!

There is a way to discard old frames, but it's a bit of a hidden feature. This whole area is something we plan on improving very soon (probably next week).

To test this at the moment you'll need to change your code to:

rr.init("Webcam streaming test", spawn=False, default_enabled=False)
rr.connect()

and then start the rerun viewer separately with:

python -m rerun --memory-limit=8GB --drop-at-latency=200ms

See:

• Documentation on Memory Usage / out-of-memory behavior #1138
• Document how to connect remotely #1342
• Crash on memory pruning #1414

[Erol444]

Hi @emilk ,
Thanks for the quick reply, I tried it out, memory purger is quite ambitious when trying to purge memory (eg 10.1 GiB) but its performance isn't that great (170 MiB). It should be put on a PIP :)
Looking forward to updates coming next week!
Thanks, Erik

 python -m rerun --memory-limit=4GB --drop-at-latency=300ms
2023-02-28T16:43:53.158219Z  INFO re_viewer::app: Attempting to purge 25.0% of used RAM (958 MiB)…2023-02-28T16:43:53.158819Z  INFO re_arrow_store::store_gc: starting GC kind="gc" id=1 target=DropAtLeast(25%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="488" initial_size_bytes="1.4 GiB" target_size_bytes="1.1 GiB" drop_at_least_size_bytes="360 MiB"
2023-02-28T16:43:53.161863Z  INFO re_arrow_store::store_gc: GC done kind="gc" id=1 target=DropAtLeast(25%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="488" initial_size_bytes="1.4 GiB" new_nb_rows="488" new_size_bytes="1.4 GiB"
2023-02-28T16:43:53.165048Z  INFO re_viewer::app: Freed up 894 MiB (23.3%)
2023-02-28T16:44:03.173977Z  INFO re_viewer::app: Attempting to purge 57.8% of used RAM (3.5 GiB)…2023-02-28T16:44:03.174956Z  INFO re_arrow_store::store_gc: starting GC kind="gc" id=2 target=DropAtLeast(57.78%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="990" initial_size_bytes="2.9 GiB" target_size_bytes="1.2 GiB" drop_at_least_size_bytes="1.7 GiB"
2023-02-28T16:44:03.177168Z  INFO re_arrow_store::store_gc: GC done kind="gc" id=2 target=DropAtLeast(57.78%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="990" initial_size_bytes="2.9 GiB" new_nb_rows="990" new_size_bytes="2.9 GiB"
2023-02-28T16:44:03.180858Z  INFO re_viewer::app: Freed up 208 MiB (3.4%)
2023-02-28T16:44:13.249289Z  INFO re_viewer::app: Attempting to purge 77.8% of used RAM (6.9 GiB)…2023-02-28T16:44:13.253350Z  INFO re_arrow_store::store_gc: starting GC kind="gc" id=3 target=DropAtLeast(77.80%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="1.5k" initial_size_bytes="4.3 GiB" target_size_bytes="973 MiB" drop_at_least_size_bytes="3.3 GiB"
2023-02-28T16:44:13.257424Z  INFO re_arrow_store::store_gc: GC done kind="gc" id=3 target=DropAtLeast(77.80%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="1.5k" initial_size_bytes="4.3 GiB" new_nb_rows="1.5k" new_size_bytes="4.3 GiB"
2023-02-28T16:44:13.260014Z  INFO re_viewer::app: Freed up 202 MiB (2.2%)
2023-02-28T16:44:23.288231Z  INFO re_viewer::app: Attempting to purge 82.0% of used RAM (8.0 GiB)…2023-02-28T16:44:23.289351Z  INFO re_arrow_store::store_gc: starting GC kind="gc" id=4 target=DropAtLeast(81.97%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="1.7k" initial_size_bytes="4.8 GiB" target_size_bytes="893 MiB" drop_at_least_size_bytes="4.0 GiB"
2023-02-28T16:44:23.291480Z  INFO re_arrow_store::store_gc: GC done kind="gc" id=4 target=DropAtLeast(81.97%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="1.7k" initial_size_bytes="4.8 GiB" new_nb_rows="1.7k" new_size_bytes="4.8 GiB"
2023-02-28T16:44:23.293181Z  INFO re_viewer::app: Freed up 59.3 MiB (0.6%)
2023-02-28T16:44:33.585555Z  INFO re_viewer::app: Attempting to purge 86.9% of used RAM (9.8 GiB)…2023-02-28T16:44:33.602243Z  INFO re_arrow_store::store_gc: starting GC kind="gc" id=5 target=DropAtLeast(86.88%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="1.9k" initial_size_bytes="5.4 GiB" target_size_bytes="722 MiB" drop_at_least_size_bytes="4.7 GiB"
2023-02-28T16:44:33.640933Z  INFO re_arrow_store::store_gc: GC done kind="gc" id=5 target=DropAtLeast(86.88%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="1.9k" initial_size_bytes="5.4 GiB" new_nb_rows="1.9k" new_size_bytes="5.4 GiB"
2023-02-28T16:44:33.645960Z  INFO re_viewer::app: Freed up 374 MiB (3.2%)
2023-02-28T16:44:33.926329Z  WARN re_sdk_comms::server: Input latency is over the max (0.3 s) - dropping packets.
2023-02-28T16:44:43.666333Z  INFO re_viewer::app: Attempting to purge 87.7% of used RAM (10.1 GiB)…
2023-02-28T16:44:43.691789Z  INFO re_arrow_store::store_gc: starting GC kind="gc" id=6 target=DropAtLeast(87.71%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="2.0k" initial_size_bytes="5.6 GiB" target_size_bytes="711 MiB" drop_at_least_size_bytes="5.0 GiB"
2023-02-28T16:44:43.694857Z  INFO re_arrow_store::store_gc: GC done kind="gc" id=6 target=DropAtLeast(87.71%) timeline=log_time primary_component=rerun.msg_id initial_nb_rows="2.0k" initial_size_bytes="5.6 GiB" new_nb_rows="2.0k" new_size_bytes="5.6 GiB"
2023-02-28T16:44:43.697855Z  INFO re_viewer::app: Freed up 172 MiB (1.5%)

[teh-cmc]

There's definitely something very wrong with the GC at the moment; it'll receive some love during this week.

[jleibs]

Resolved in: #1502

[themarpe]

Hi @teh-cmc

Looks like rerun is holding all previous frames in RAM, is there a way to either a) save them to disk instead or b) discard old frames?

Any news on this note specifically?

[teh-cmc]

Hey @themarpe, the --memory-limit flag should do the right thing once again since #1560 landed.

I don't think it shipped in any release yet, though it will ship as part of today's release (yes, there's a release happening today, coincidentally!).

[themarpe]

Thanks @teh-cmc !

Was thinking of a more native "streaming but not saving" way. I know it's not in the main alley of "rerun" hah, but would be interesting feature for such cases. Perhaps more specifically, double/triple buffered per "name of stream" and that would be all in this mode (without having to wonder what memory limit to put)