Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BlockingIOError: [Errno 35] Resource temporarily unavailable #1185

Open
moreati opened this issue Nov 6, 2024 · 8 comments
Open

BlockingIOError: [Errno 35] Resource temporarily unavailable #1185

moreati opened this issue Nov 6, 2024 · 8 comments

Comments

@moreati
Copy link
Member

moreati commented Nov 6, 2024

Seen in CI on macOS 13 https://github.com/mitogen-hq/mitogen/actions/runs/11669080981/job/32490230504?pr=1176

TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Monday 04 November 2024  17:08:28 +0000 (0:00:00.007)       0:00:26.162 ******* 
changed: [target]
TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Monday 04 November 2024  17:08:30 +0000 (0:00:01.338)       0:00:27.501 ******* 
ERROR! [mux  5271] 17:08:37.240824 E mitogen: <Stream unix_client.5683 #1dd0> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

Originally posted by @moreati in #1133 (comment)
@moreati
Copy link
Member Author

moreati commented Nov 6, 2024 •
edited
Loading

Again https://github.com/mitogen-hq/mitogen/actions/runs/11700584431/job/32584889627

PLAY [regression/issue_615_streaming_transfer.yml] *****************************
TASK [include_tasks _raw_params=_mitogen_only.yml] *****************************
Wednesday 06 November 2024  09:20:58 +0000 (0:00:00.120)       0:00:26.685 **** 
included: /Users/runner/work/mitogen/mitogen/tests/ansible/regression/_mitogen_only.yml for target
TASK [meta _raw_params=end_play] ***********************************************
Wednesday 06 November 2024  09:20:58 +0000 (0:00:00.039)       0:00:26.725 **** 
skipping: [target]
TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Wednesday 06 November 2024  09:20:58 +0000 (0:00:00.014)       0:00:26.739 **** 
changed: [target]
TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Wednesday 06 November 2024  09:21:00 +0000 (0:00:01.685)       0:00:28.424 **** 
ERROR! [mux  5601] 09:21:05.103152 E mitogen: <Stream unix_client.5949 #8cd0> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

@moreati
Copy link
Member Author

moreati commented Nov 7, 2024

Again, macos (Loc_313_10, py313-mode_localhost-ansible10)

PLAY [regression/issue_615_streaming_transfer.yml] *****************************
TASK [include_tasks _raw_params=_mitogen_only.yml] *****************************
Thursday 07 November 2024  14:22:43 +0000 (0:00:00.113)       0:00:25.291 ***** 
included: /Users/runner/work/mitogen/mitogen/tests/ansible/regression/_mitogen_only.yml for target
TASK [meta _raw_params=end_play] ***********************************************
Thursday 07 November 2024  14:22:43 +0000 (0:00:00.037)       0:00:25.328 ***** 
skipping: [target]
TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Thursday 07 November 2024  14:22:43 +0000 (0:00:00.010)       0:00:25.339 ***** 
changed: [target]
TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Thursday 07 November 2024  14:22:44 +0000 (0:00:01.361)       0:00:26.700 ***** 
ERROR! [mux  3519] 14:22:49.843910 E mitogen: <Stream unix_client.3873 #db50> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

@moreati moreati mentioned this issue Nov 7, 2024
Open
@moreati
Copy link
Member Author

moreati commented Nov 7, 2024

#1194 has hit this 3 times in a row

@moreati
Copy link
Member Author

moreati commented Nov 27, 2024

#1177 twice in a row (job 394, 2 attempts), once it occurs the build hangs until it is concelled or times out (currently 2 hours)

TASK [Create /tmp/512mb.zero creates=/tmp/512mb.zero, _raw_params=dd if=/dev/zero of=/tmp/512mb.zero bs=1048576 count=512;
chmod go= /tmp/512mb.zero
] ***
Wednesday 27 November 2024  10:45:54 +0000 (0:00:00.011)       0:00:28.576 **** 
changed: [target]

TASK [Fetch /tmp/512mb.zero src=/tmp/512mb.zero, dest=/tmp/fetch-{{ inventory_hostname }}-512mb.zero, flat=True] ***
Wednesday 27 November 2024  10:45:55 +0000 (0:00:01.644)       0:00:30.220 **** 
ERROR! [mux  6790] 10:46:04.059200 E mitogen: <Stream unix_client.7491 #c550> crashed
Traceback (most recent call last):
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 3677, in _call
    func(self)
    ~~~~^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 1923, in on_transmit
    self.protocol.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2371, in on_transmit
    self._writer.on_transmit(broker)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2111, in on_transmit
    written = self._protocol.stream.transmit_side.write(buf)
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 2237, in write
    written, disconnected = io_op(os.write, self.fd, s)
                            ~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/mitogen/mitogen/mitogen/core.py", line 595, in io_op
    return func(*args), None
           ~~~~^^^^^^^
BlockingIOError: [Errno 35] Resource temporarily unavailable

@moreati
Copy link
Member Author

moreati commented Dec 2, 2024

This stopped happening in CI on Nov 29. I now cannot reproduce

@moreati
Copy link
Member Author

moreati commented Dec 2, 2024

Spoke too soon, It's back https://github.com/mitogen-hq/mitogen/actions/runs/12116635990/job/33777406222

@moreati
Copy link
Member Author

moreati commented Dec 2, 2024

Judging by https://github.com/mitogen-hq/mitogen/actions/runs/12117305356/job/33779477126 I don't think there's a pattern wrt to Ansible versions or Python versions. So it's probably something non-deterministic such as a race condition.

@moreati
Copy link
Member Author

moreati commented Dec 9, 2024

It's baaaaaack https://github.com/mitogen-hq/mitogen/actions/runs/12237211642/job/34139694093?pr=1200

moreati added a commit to moreati/mitogen that referenced this issue Dec 10, 2024
@moreati
build: Reduce macOS job timeout to mitigate BlockingIOError
941da31 
This will mitigate the impact of mitogen-hq#1185 a little, which causes the job to
continue running without making progress, until it hits this timeout.
Successful jobs are typically completing in 8 - 12 minutes.

refs mitogen-hq#1185
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@moreati