4.13.1 (90346) Docker crash when running container with selenium and chrome

[trevesz]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID: 5b5ed26e-0021-4de6-8651-9305b4039bb3/20221021175526

Expected behavior

Single run python script executes and uses selenium and chrome to scrape target website then done. Been doing this for almost 3 years daily.

Actual behavior

After upgrading to Docker Desktop for Mac 4.13.0, the docker engine crashes with this message in the terminal that launched the container with docker run command. I thought the issue might have been fixed in 4.13.1 but it is still occurring, though less frequently/reliably than before.

ERRO[0007] error waiting for container: invalid character 'c' looking for beginning of value

The menu bar icon shows the engine stopping...the dashboard shows engine stopped but trying to restart from the docker menu fails with a fatal error. Screenshot attached. You have to exit/quit docker and restart it for the engine to become operational again.

Information

• macOS Version: Monterey 12.6
• Intel chip or Apple chip: Intel
• Docker Desktop Version: 4.13.0 & 4.13.1

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

This output is before running the container and causing the crash. If I run this after the crash, of course it generates a bunch of errors about not being able to reach the docker services. I can post that as well but figured you cared more to know that before each run, the base docker installation is healthy.

Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0013: is the $PATH ok?
[PASS] DD0003: is the Docker CLI working?
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[PASS] DD0012: is the VM networking working?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0019: is the com.docker.vmnetd process responding?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[PASS] DD0004: is the Docker engine running?
[PASS] DD0015: are the binary symlinks installed?
[PASS] DD0031: does the Docker API work?
[PASS] DD0032: do Docker networks overlap with host IPs?
No fatal errors detected.

Steps to reproduce the behavior

1. Dockerfile

# For more information, please refer to https://aka.ms/vscode-docker-python
FROM python:3.10-slim

# Install packages we are going to need for setting up Chrome
RUN apt-get update && apt-get install -y \
    gnupg \
    wget \
    curl \
    unzip \
    && rm -rf /var/lib/apt/lists/*

# Install Google Chrome
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add -
RUN sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google-chrome.list'
RUN apt-get update && apt-get install -y \
    google-chrome-stable \
    && rm -rf /var/lib/apt/lists/*

# Install chromedriver
RUN wget -O /tmp/chromedriver.zip http://chromedriver.storage.googleapis.com/`curl -sS chromedriver.storage.googleapis.com/LATEST_RELEASE`/chromedriver_linux64.zip
RUN unzip /tmp/chromedriver.zip chromedriver -d /usr/local/bin/ && rm /tmp/chromedriver.zip

# Set display port to avoid crashing Chrome
ENV DISPLAY=:99

# Keeps Python from generating .pyc files in the container
ENV PYTHONDONTWRITEBYTECODE=1

# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1

# Install pip requirements
ADD requirements.txt .
RUN python -m pip install -r requirements.txt

WORKDIR /app
ADD ./src /app

# Switching to a non-root user, please refer to https://aka.ms/vscode-docker-python-user-rights
RUN useradd appuser && chown -R appuser /app
USER appuser

# During debugging, this entry point will be overridden. For more information, please refer to https://aka.ms/vscode-docker-python-debug
CMD ["python", "app.py"]

2. Run this app.py to cause the fault, you may have to run it a few times before the crash will occur but it only takes 1-3 tries reliably on my machine.

from selenium import webdriver
from selenium.webdriver.common.by import By
from time import sleep

class Scraper:
    def __init__(self, delay):
        # Let's grab our default settings for Chromium instance
        chrome_options = self.__build_chrome_options()
        self.delay = delay

        try:
            # Let's instantiate our scraper
            self.driver = webdriver.Chrome(options=chrome_options)
        except Exception as e:
            raise e

    def __build_chrome_options(self):
        chrome_options = webdriver.ChromeOptions()
        chrome_options.add_argument('--window-size=1920,1200')
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--single-process')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-gpu')
        chrome_options.add_argument('--enable-logging')
        chrome_options.add_argument('--user-data-dir=/tmp/user1')
        chrome_options.add_argument('--v=1')
        chrome_options.add_argument('--ignore-certificate-errors')
        chrome_options.add_argument('--disable-dev-shm-usage')

        return chrome_options

    def scroll_down(self, delay):
        while True:
            # Get scroll height.
            lastHeight = self.driver.execute_script("return document.body.scrollHeight")

            # Scroll down to the bottom. 
            print(f"Last height: [{lastHeight}] - Scrolling...") # uncomment for debugging scrolling
            self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

            # Wait to load the page
            sleep(delay/2)

            # Calculate new scroll height and compare with last scroll height
            newHeight = self.driver.execute_script("return document.body.scrollHeight")

            # If the browser hasn’t scrolled any more (i.e. it’s reached the end) then stop
            if newHeight == lastHeight:
                print("Reached bottom") # uncomment for debugging scrolling
                break

    def get_with_delay(self, request_url, delay):
        self.driver.get(request_url)
        sleep(delay)
        self.scroll_down(delay)
    
    def __del__(self):
        self.driver.quit()

# use this for testing locally
if __name__ == "__main__":
    scraper = Scraper(3)

    url = "https://abcnews.go.com/"

    try:
        print("Getting url: ", url)
        scraper.get_with_delay(url, 3)
    except Exception as e:
        print(f'Error {e} while scraping url: {url}')

You should just need to add selenium to a requirements.txt file and/or install it on running container to get the test code to work on top of the base python image. I tried to see if the issue was python, base linux in container and/or selenium version rooted but same result with python 3.8, 3.9 and 3.10, selenium 4.2, 4.3, 4.4, 4.5 and both buster and bullseye distros. Hope this is helpful in fixing for future Docker release.

[trevesz]

I should also point out that I tried this with both chrome_options.add_argument('--disable-dev-shm-usage') option, which is how I've run for years, as well as without that option and added --shm-size=2g to docker run as well. These are related to memory handling issues and crashes in the past with selenium and chrome but issue occurs with either approach in this case.

[joaopbnogueira]

Similar crash over here with a M1 Pro

[boxshadow]

same issue

[handelaar2]

Same issue. Docker desktop 4.13.0 (89412) crashes:
ERRO[0014] error waiting for container: invalid character 'c' looking for beginning of value
Also own container, not the one mentioned in the description.
(MacBook Pro 2019, Monterey 12.6)

[E-G-C]

equivalent error here, Mac M1 Pro message: invalid character 's' looking for beginning of value
(I'm running my own container, not the one mentioned on the description)
Update: Reverted back to version 4.12.0 and it's working fine.

[henriquebremenkanp]

I'm also experiencing this problem after the latest update to 4.13.0. Tried factory reset many times, but always get the same error after some time of usage.
I guess it was after running docker-compose up --build, but I'm not sure. Have no idea on how to debug it...

As it was mentioned above, 4.12.0 works fine. I'm also on Macbook Pro M1 Pro, Monterey 12.6.1

[jyoti-c]

Same issue with Monterey 12.6.1.

[oliveremberton]

Same issue here with Monterey 12.6.1, confirmed for three separate developers.

[StefanIliev545]

Exactly the same issue using different images. M1 Pro, Ventura

[hopeseekr]

I can confirm experiencing the same issue with custom built docker image containing both Chromium and Selenium. Version 4.13.0 (89412) on Mac OSX (Intel and M1).

Everything works fine in Docker Desktop v4.12.0.

We experience it even after rebuilding the images, fwiw.

[riabiy-denis]

I can confirm experiencing the same issue with custom built docker image containing both Chromium and Selenium. Version 4.13.0 (89412) on Mac OSX (Intel and M1).

Everything works fine in Docker Desktop v4.12.0.

We experience it even after rebuilding the images, fwiw.

Thank you for giving an advice on what to downgrade :D
For some reason I was playing with images, but seems like it's bout a new docker release.

[apotrebin]

Seems like fixed on Version 4.13.1
ERRO[0019] error waiting for container: invalid character 'c' looking for beginning of value
I don't have this issue anymore with mcr.microsoft.com/playwright image
-macOS Ventura 13.0 (Intel M1)

[trevesz]

It would appear so! Issue not happening with 4.13.1 for my original containers either. :)

[trevesz]

Unfortunately, it happened again this morning.

ERRO[0115] error waiting for container: invalid character 'c' looking for beginning of value

All the same exact code. Appears to still be present, though happens less frequently than before.

[dimagoltsman]

many random crashes, this version is unstable, went back to 4.12 and everything works

[riabiy-denis]

many random crashes, this version is unstable, went back to 4.12 and everything works

Same.

[mmikhalko]

I have the same bug!

[bluehz]

Same bug here Mac OS 12.4 / Docker 4.13.1 - it starts and runs for a while, then silently crashes and freezes the app.

[mattdoran]

Me too. Mac OS 12.5.1 (Intel) / Docker 4.13.1- crashes after running for less than a day.

[vtsimbaluk]

Same issue, OS 13.0 (Intel)

[smailpouri]

Similar issue here with 4.13 up to 4.14.
4.14 is more stable for me.

This issue does NOT exist on the latest 4.12.x

Apple M1 on macOS Ventura 13.0.1

[mmikhalko]

Issue still in 4.14 version

[helloguille]

Docker freezing since my last update, randomly.

I cannot restart it, the contenerized app (apache and php) stops working (freezes and stops responding to requests)

MacOSX Monterrey (M1)
4.13.1

[mmikhalko]

Issue still in 4.14.1 version

[helloguille]

How could I go back to 4.12.1?

[xjose97x]

Still crashing for me every now and then. I have a M1 Pro processor.
Running Docker 4.15.0 (93002)

[kennethredler]

@trevesz thanks for the update! It's good to know we're on the right track...

I've got another test build which should have both TCP and UDP working if you'd like to try it (without the QUIC workaround). Let me know what you think:

• amd64
• arm64

Build is working great @djs55 ‼️

No longer seeing Docker "going out to lunch". 👍👏

When can we expect this fix in a production release? 🙏

[djs55]

@kennethredler I'm hoping to get this fix into 4.16, due in January. It's merged already but I'm on the look out for possible regressions. Speaking of which:

@ylor could you describe your setup a little more? Is it TCP or UDP? Are you using the virtualisation.framework (or Intel hyperkit or Apple Silicon qemu)? Virtualization framework with recent macOS versions should be fast. Could you share a benchmark result (e.g. with iperf?) Thanks!

[ylor]

Hey @djs55 👋, no benchmarks unfortunately -- downgraded back to 4.12. I can try and get some put together but in the meantime, my setup looks like:

• macOS 13.1
• Virtualization Framework + VirtioFS
• Large file transfers (TCP mostly) of Linux ISOs tunneled through a Wireguard (UDP only) container

[kennethredler]

@ kennethredler I'm hoping to get this fix into 4.16, due in January. It's merged already but I'm on the look out for possible regressions. Speaking of which:

@ ylor could you describe your setup a little more? Is it TCP or UDP? Are you using the virtualisation.framework (or Intel hyperkit or Apple Silicon qemu)? Virtualization framework with recent macOS versions should be fast. Could you share a benchmark result (e.g. with iperf?) Thanks!

Thanks for the info and update @djs55 . Regrettably, the real issue I'm seeing may be #6472 ("Docker is Stopping") and this build doesn't seem to resolve that. Will be looking for help over there.

[samsawyer]

@djs55, unfortunately I can confirm @ylor's regression. The set up is using docker compose to run a Wireguard VPN container (so all UDP) and funneling network traffic through that. When I do that and test using fast.com in a browser, I get 93 Mbps from fast.com under 4.12 and only 6.8 Mbps under your new 4.16 build.

To confirm, I also set up the same VPN connection using a different Wireguard container and ran iperf3 tests — here's what that looks like:

Under 4.12:

iperf 3.12
Linux 2514c6071d1d 5.10.124-linuxkit #1 SMP Thu Jun 30 08:19:10 UTC 2022 x86_64
Control connection MSS 1368
Time: Sat, 31 Dec 2022 16:05:52 GMT
Connecting to host 94.154.159.137, port 5204
      Cookie: ujfqmc7cdnvgwhbzv6emfk5enyw3drqeyhx2
      TCP MSS: 1368 (default)
[  5] local 10.16.253.3 port 47290 connected to 94.154.159.137 port 5204
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  12.6 MBytes   105 Mbits/sec    0    334 KBytes       
[  5]   1.00-2.00   sec  12.9 MBytes   108 Mbits/sec    1    367 KBytes       
[  5]   2.00-3.00   sec  12.8 MBytes   107 Mbits/sec    0    393 KBytes       
[  5]   3.00-4.00   sec  13.7 MBytes   115 Mbits/sec    0    417 KBytes       
[  5]   4.00-5.00   sec  12.8 MBytes   107 Mbits/sec    0    438 KBytes       
[  5]   5.00-6.00   sec  10.1 MBytes  84.4 Mbits/sec    0    454 KBytes       
[  5]   6.00-7.00   sec  12.8 MBytes   107 Mbits/sec    0    474 KBytes       
[  5]   7.00-8.00   sec  12.3 MBytes   103 Mbits/sec    0    508 KBytes       
[  5]   8.00-9.00   sec  13.6 MBytes   114 Mbits/sec    0    590 KBytes       
[  5]   9.00-10.00  sec  13.6 MBytes   114 Mbits/sec    0    695 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   127 MBytes   107 Mbits/sec    1             sender
[  5]   0.00-10.00  sec   124 MBytes   104 Mbits/sec                  receiver
CPU Utilization: local/sender 1.1% (0.2%u/0.9%s), remote/receiver 3.2% (0.6%u/2.6%s)

and under your 4.16 build:

iperf 3.12
Linux b2cef2755e74 5.15.49-linuxkit #1 SMP Tue Sep 13 07:51:46 UTC 2022 x86_64
Control connection MSS 1368
Time: Sat, 31 Dec 2022 15:47:33 GMT
Connecting to host 94.154.159.137, port 5204
      Cookie: a3au2ke5xuodtpy5fuajmsfu3ciebo23rdpt
      TCP MSS: 1368 (default)
[  5] local 10.15.173.84 port 49974 connected to 94.154.159.137 port 5204
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.52 MBytes  12.8 Mbits/sec    0    108 KBytes       
[  5]   1.00-2.00   sec  1005 KBytes  8.23 Mbits/sec   11   93.5 KBytes       
[  5]   2.00-3.00   sec   753 KBytes  6.17 Mbits/sec    7   74.8 KBytes       
[  5]   3.00-4.00   sec  1005 KBytes  8.23 Mbits/sec    0   86.8 KBytes       
[  5]   4.00-5.00   sec   753 KBytes  6.18 Mbits/sec    0   90.8 KBytes       
[  5]   5.00-6.01   sec  1005 KBytes  8.18 Mbits/sec    0   96.2 KBytes       
[  5]   6.01-7.01   sec  1.04 MBytes  8.75 Mbits/sec    5   70.8 KBytes       
[  5]   7.01-8.01   sec   753 KBytes  6.18 Mbits/sec    1   78.8 KBytes       
[  5]   8.01-9.00   sec  1005 KBytes  8.27 Mbits/sec    0   86.8 KBytes       
[  5]   9.00-10.00  sec  1.23 MBytes  10.3 Mbits/sec    0   97.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.92 MBytes  8.32 Mbits/sec   24             sender
[  5]   0.00-10.00  sec  9.56 MBytes  8.02 Mbits/sec                  receiver
CPU Utilization: local/sender 0.5% (0.1%u/0.4%s), remote/receiver 0.1% (0.1%u/0.0%s)

In both 4.12 and 4.16, I'm using the new Virtualization framework and running macOS Ventura 13.1 on an Intel chip.

If I can provide any additional details that would be helpful in tracking this down, happy to do what I can.

[bewareofgeek]

I tried the 4.16 test build as well on mac m1. It resolved the issue of docker going down completely when under network load, but now my total throughput from the containers is stuck at about 5MB/s. I've tested this repeatedly by having downloads in both transmission and sabnzbd. They'll each get a part of the 5MB/s and if I pause one, the other gets it's speed. I'm wondering, did you just put a limiter. Is the original issue caused by higher throughput? In any case, anxious to get a fix for this. It's sad my Pi4 is more efficient at this and it has to copy the completed downloads over the network to my media server.

[djs55]

Thanks for the info on UDP performance. I found a logging problem in that development build -- it was logging once per packet (!) so I've removed that and tidied the network logs generally. If you'd like to try a new M1 build I have one here: https://desktop-stage.docker.com/mac/main/arm64/94771/Docker.dmg This also has a fix for a bug which affected multiple concurrent TCP connections.

Could you try it and let me know what happens? For reference the logs are in ~/Library/Containers/com.docker.docker/Data/log/vm/nat.log -- take a look to check you're not seeing excessive logging in there.

[samsawyer]

@djs55 if you have an Intel build that fixes the logging issue for UDP performance, I'd be happy to test it and report back iperf numbers again.

[bewareofgeek]

@djs55 I know I jumped in on this, but I wanted to let you know I just tried that latest build you provided. The docker crashing and bandwidth issues I was having both appear to be resolved. You're a lifesaver. I was pulling my hair out trying to figure this out. First time using docker and just needed something better than the Pi4 I was using and couldn't figure out why I was the only one seeming to have issues on such a basic setup.

[djs55]

@samsawyer I've got an Intel build here: https://desktop-stage.docker.com/mac/main/amd64/94771/Docker.dmg -- let me know what you observe!

[Neurone]

The test build solved this issue on my machine!

[samsawyer]

@djs55, nice work. Just tested with the new Intel build, and here are some iperf results:

iperf 3.12
Linux f66c6ab4be84 5.15.49-linuxkit #1 SMP Tue Sep 13 07:51:46 UTC 2022 x86_64
Control connection MSS 1368
Time: Wed, 04 Jan 2023 01:44:27 GMT
Connecting to host 94.154.159.137, port 5204
      Cookie: j2ouplljljrbdybtsmsonnooniqhbzxudj4d
      TCP MSS: 1368 (default)
[  5] local 10.6.156.58 port 53438 connected to 94.154.159.137 port 5204
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 seconds, 10 second test, tos 0
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.8 MBytes  99.1 Mbits/sec   13    172 KBytes       
[  5]   1.00-2.00   sec  8.58 MBytes  72.0 Mbits/sec   19    124 KBytes       
[  5]   2.00-3.00   sec  9.99 MBytes  83.8 Mbits/sec    0    172 KBytes       
[  5]   3.00-4.00   sec  10.4 MBytes  87.4 Mbits/sec   11    158 KBytes       
[  5]   4.00-5.01   sec  9.87 MBytes  82.4 Mbits/sec    9    104 KBytes       
[  5]   5.01-6.00   sec  8.58 MBytes  72.4 Mbits/sec    0    152 KBytes       
[  5]   6.00-7.00   sec  11.0 MBytes  92.6 Mbits/sec    1    140 KBytes       
[  5]   7.00-8.00   sec  10.4 MBytes  87.5 Mbits/sec   19    139 KBytes       
[  5]   8.00-9.00   sec  9.93 MBytes  83.3 Mbits/sec   11    135 KBytes       
[  5]   9.00-10.00  sec  11.0 MBytes  92.6 Mbits/sec    0    186 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   102 MBytes  85.3 Mbits/sec   83             sender
[  5]   0.00-10.00  sec   100 MBytes  84.2 Mbits/sec                  receiver
CPU Utilization: local/sender 1.1% (0.2%u/0.9%s), remote/receiver 2.3% (0.4%u/1.8%s)

So back to near-parity with 4.12 in terms of performance; whew.

(For the sake of completeness: I also ended up trying colima as a result of all this — and in colima, the same iperf test run with the same docker-compose.yml gives me more than double the performance, clocking in at 222 Mbps.)

But in any case, I think you've solved the speed regression from 4.12 to what's due to be merged into 4.16, and you'd already solved the crash-under-network-load issue. So thanks for tackling both of those.

[sellersj]

@djs55 thank you for your work.

4.16 is released. Does this mean all the changes are part of that release?

[djs55]

@sellersj yes, all those changes should be in 4.16. Let me know if you see anything suspicious!

[trevesz]

Awesome news. Thank you as well for the great follow up and support @djs55! Closing out the issue as resolved.

-tomas

[mat007]

A fix has been released in Docker Desktop 4.16. See the release notes for more details.

[ylor]

Thank you @samsawyer for following up with numbers and @djs55 for the hard work!

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

/lifecycle locked