/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/geoapis/vector.py:270: ShapelyDeprecationWarning: The 'GeometryType()' method is deprecated, and will be removed in the future. You can use the 'geom_type' attribute instead. shapely_geometry.geometryType() == "MultiPolygon" 0GB [00:00, ?GB/s] 0GB [00:00, ?GB/s] /nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/node.py:182: UserWarning: Port 8787 is already in use. Perhaps you already have a cluster running? Hosting the HTTP server on port 40248 instead warnings.warn( WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 13:19:59,313 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:34403 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for return await fut ^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect stream = await self.client.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect af, addr, stream = await connector.start(connect_timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect comm = await wait_for( ^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for async with asyncio.timeout(timeout): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker comm = await rpc.connect(worker) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect return connect_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect comm = await connect( ^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect raise OSError( OSError: Timed out trying to connect to tcp://127.0.0.1:34403 after 30 s 2023-10-13 13:21:32,899 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:40162 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for return await fut ^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect stream = await self.client.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect af, addr, stream = await connector.start(connect_timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect comm = await wait_for( ^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for async with asyncio.timeout(timeout): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker comm = await rpc.connect(worker) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect return connect_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect comm = await connect( ^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect raise OSError( OSError: Timed out trying to connect to tcp://127.0.0.1:40162 after 30 s 2023-10-13 13:22:09,334 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:44227 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for return await fut ^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect stream = await self.client.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect af, addr, stream = await connector.start(connect_timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect comm = await wait_for( ^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for async with asyncio.timeout(timeout): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker comm = await rpc.connect(worker) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect return connect_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect comm = await connect( ^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect raise OSError( OSError: Timed out trying to connect to tcp://127.0.0.1:44227 after 30 s 2023-10-13 13:34:26,705 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:41447 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for return await fut ^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect stream = await self.client.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect af, addr, stream = await connector.start(connect_timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect comm = await wait_for( ^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for async with asyncio.timeout(timeout): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2863, in get_data_from_worker comm = await rpc.connect(worker) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect return connect_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect comm = await connect( ^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect raise OSError( OSError: Timed out trying to connect to tcp://127.0.0.1:41447 after 30 s WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. /nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/client.py:3160: UserWarning: Sending large graph of size 18.20 MiB. This may cause some slowdown. Consider scattering data ahead of time and using futures. warnings.warn( WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 14:02:01,556 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:42381'. 2023-10-13 14:02:01,591 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError 2023-10-13 14:02:01,592 - distributed.worker - ERROR - Scheduler was unaware of this worker 'tcp://127.0.0.1:42381'. Shutting down. 2023-10-13 14:02:04,707 - distributed.nanny - WARNING - Worker process still alive after 3.1999975585937506 seconds, killing WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 14:02:34,162 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:42381 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 234, in read n = await stream.read_into(chunk) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2866, in get_data_from_worker response = await send_recv( ^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in Worker for gather local=tcp://127.0.0.1:51832 remote=tcp://127.0.0.1:42381>: Stream is closed WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. /nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/client.py:3160: UserWarning: Sending large graph of size 19.37 MiB. This may cause some slowdown. Consider scattering data ahead of time and using futures. warnings.warn( WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 14:37:29,516 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:37233'. 2023-10-13 14:37:31,516 - distributed.worker - ERROR - Scheduler was unaware of this worker 'tcp://127.0.0.1:37233'. Shutting down. 2023-10-13 14:37:37,370 - distributed.nanny - WARNING - Worker process still alive after 3.1999978637695317 seconds, killing WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 14:37:42,947 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:34678 -> tcp://127.0.0.1:37233 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1800, in get_data response = await comm.read(deserializers=serializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 14:58:33,177 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:34403 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2866, in get_data_from_worker response = await send_recv( ^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in Worker for gather local=tcp://127.0.0.1:48964 remote=tcp://127.0.0.1:34403>: Stream is closed 2023-10-13 14:58:33,178 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:34403 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2866, in get_data_from_worker response = await send_recv( ^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in Worker for gather local=tcp://127.0.0.1:47458 remote=tcp://127.0.0.1:34403>: Stream is closed 2023-10-13 14:58:33,318 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:34403 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2866, in get_data_from_worker response = await send_recv( ^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in Worker for gather local=tcp://127.0.0.1:48674 remote=tcp://127.0.0.1:34403>: Stream is closed 2023-10-13 14:58:38,908 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:34403 -> tcp://127.0.0.1:36606 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1800, in get_data response = await comm.read(deserializers=serializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2023-10-13 14:58:38,913 - distributed.scheduler - WARNING - Received heartbeat from unregistered worker 'tcp://127.0.0.1:34403'. 2023-10-13 14:58:38,926 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:34403 -> tcp://127.0.0.1:35037 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1800, in get_data response = await comm.read(deserializers=serializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2023-10-13 14:58:38,928 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:34403 -> tcp://127.0.0.1:38212 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1800, in get_data response = await comm.read(deserializers=serializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2023-10-13 14:58:38,930 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:34403 -> tcp://127.0.0.1:40162 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1800, in get_data response = await comm.read(deserializers=serializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2023-10-13 14:58:38,932 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:34403 -> tcp://127.0.0.1:34678 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1800, in get_data response = await comm.read(deserializers=serializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2023-10-13 14:58:38,935 - distributed.worker - ERROR - Scheduler was unaware of this worker 'tcp://127.0.0.1:34403'. Shutting down. 2023-10-13 14:58:38,944 - distributed.worker.state_machine - WARNING - Async instruction for > ended with CancelledError 2023-10-13 14:58:33,180 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:34403 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2866, in get_data_from_worker response = await send_recv( ^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in Worker for gather local=tcp://127.0.0.1:48396 remote=tcp://127.0.0.1:34403>: Stream is closed 2023-10-13 14:58:51,415 - distributed.nanny - WARNING - Worker process still alive after 3.199997711181641 seconds, killing 2023-10-13 14:59:00,212 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:34403 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2866, in get_data_from_worker response = await send_recv( ^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in Worker for gather local=tcp://127.0.0.1:51132 remote=tcp://127.0.0.1:34403>: Stream is closed WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 15:13:48,858 - distributed.worker - ERROR - failed during get data with tcp://127.0.0.1:37759 -> tcp://127.0.0.1:45100 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/iostream.py", line 861, in _read_to_buffer bytes_read = self.read_from_fd(buf) ^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/iostream.py", line 1116, in read_from_fd return self.socket.recv_into(buf, len(buf)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TimeoutError: [Errno 110] Connection timed out The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1800, in get_data response = await comm.read(deserializers=serializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 141, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in : TimeoutError: [Errno 110] Connection timed out WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 15:13:58,548 - distributed.worker - ERROR - Worker stream died during communication: tcp://127.0.0.1:37759 Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/iostream.py", line 861, in _read_to_buffer bytes_read = self.read_from_fd(buf) ^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/iostream.py", line 1116, in read_from_fd return self.socket.recv_into(buf, len(buf)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ConnectionResetError: [Errno 104] Connection reset by peer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2073, in gather_dep response = await get_data_from_worker( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 2866, in get_data_from_worker response = await send_recv( ^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 141, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc.__class__.__name__}: {exc}") from exc distributed.comm.core.CommClosedError: in Worker for gather local=tcp://127.0.0.1:47617 remote=tcp://127.0.0.1:37759>: ConnectionResetError: [Errno 104] Connection reset by peer WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. /nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/client.py:3160: UserWarning: Sending large graph of size 25.29 MiB. This may cause some slowdown. Consider scattering data ahead of time and using futures. warnings.warn( WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 15:22:27,719 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for return await fut ^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 490, in connect stream = await self.client.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/tcpclient.py", line 279, in connect af, addr, stream = await connector.start(connect_timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 342, in connect comm = await wait_for( ^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for async with asyncio.timeout(timeout): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__ raise TimeoutError from exc_val TimeoutError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1253, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils_comm.py", line 454, in retry_operation return await retry( ^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils_comm.py", line 433, in retry return await coro() ^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1353, in send_recv_from_rpc comm = await self.pool.connect(self.addr) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1600, in connect return connect_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1526, in _connect comm = await connect( ^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/core.py", line 368, in connect raise OSError( OSError: Timed out trying to connect to tcp://127.0.0.1:40561 after 30 s WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. WARNING:root:In dem.elevation_over_chunk the latest chunk has no data and is being ignored. 2023-10-13 15:56:50,591 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat. Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 224, in read frames_nbytes = await stream.read_bytes(fmt_size) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ tornado.iostream.StreamClosedError: Stream is closed The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/worker.py", line 1253, in heartbeat response = await retry_operation( ^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils_comm.py", line 454, in retry_operation return await retry( ^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils_comm.py", line 433, in retry return await coro() ^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1356, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/core.py", line 1115, in send_recv response = await comm.read(deserializers=deserializers) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 240, in read convert_stream_closed_error(self, e) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/comm/tcp.py", line 143, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in : Stream is closed 2023-10-13 15:56:53,655 - distributed.nanny - WARNING - Worker process still alive after 3.199977264404297 seconds, killing 2023-10-13 15:56:53,890 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing 2023-10-13 15:56:53,892 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2023-10-13 15:56:53,894 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2023-10-13 15:56:53,894 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2023-10-13 15:56:53,894 - distributed.nanny - WARNING - Worker process still alive after 3.1999987792968754 seconds, killing 2023-10-13 15:56:53,898 - distributed.nanny - WARNING - Worker process still alive after 3.1999980163574224 seconds, killing 2023-10-13 15:56:53,909 - distributed.nanny - WARNING - Worker process still alive after 3.199999694824219 seconds, killing 2023-10-13 15:56:53,909 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing 2023-10-13 15:56:53,914 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2023-10-13 15:56:53,916 - distributed.nanny - WARNING - Worker process still alive after 3.1999989318847657 seconds, killing 2023-10-13 15:56:53,916 - distributed.nanny - WARNING - Worker process still alive after 3.1999989318847657 seconds, killing 2023-10-13 15:56:53,917 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2023-10-13 15:56:53,944 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2023-10-13 15:56:53,946 - distributed.nanny - WARNING - Worker process still alive after 3.1999980163574224 seconds, killing 2023-10-13 15:56:53,959 - distributed.nanny - WARNING - Worker process still alive after 3.1999986267089846 seconds, killing 2023-10-13 15:56:53,978 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2023-10-13 15:56:53,980 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2023-10-13 15:56:53,981 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2023-10-13 15:56:54,008 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2023-10-13 15:56:54,009 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2023-10-13 15:56:54,017 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing 2023-10-13 15:56:54,018 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2023-10-13 15:56:54,438 - tornado.application - ERROR - Exception in callback functools.partial(>, exception=TimeoutError()>) Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for return await fut ^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/ioloop.py", line 738, in _run_callback ret = callback() ^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/ioloop.py", line 762, in _discard_future_result future.result() TimeoutError Traceback (most recent call last): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1922, in wait_for return await fut ^^^^^^^^^ asyncio.exceptions.CancelledError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "/home/pearsonra/cylc-run/cylc-geofabrics/run5/lib/python/lidar.py", line 61, in main runner.run() File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/geofabrics/processor.py", line 882, in run with cluster, distributed.Client(cluster) as client: File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/cluster.py", line 540, in __exit__ aw = self.close() ^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/spec.py", line 293, in close aw = super().close(timeout) ^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/cluster.py", line 226, in close return self.sync(self._close, callback_timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 359, in sync return sync( ^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 426, in sync raise exc.with_traceback(tb) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 399, in f result = yield future ^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/tornado/gen.py", line 767, in run value = future.result() ^^^^^^^^^^^^^^^ File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/spec.py", line 446, in _close await self._correct_state() File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/deploy/spec.py", line 359, in _correct_state_internal await asyncio.gather(*tasks) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/nanny.py", line 595, in close await self.kill(timeout=timeout, reason=reason) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/nanny.py", line 380, in kill await self.process.kill(reason=reason, timeout=0.8 * (deadline - time())) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/nanny.py", line 843, in kill await process.join(max(0, deadline - time())) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/process.py", line 330, in join await wait_for(asyncio.shield(self._exit_future), timeout) File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/site-packages/distributed/utils.py", line 1921, in wait_for async with asyncio.timeout(timeout): File "/nesi/project/niwa03440/conda/envs/geofabrics/lib/python3.11/asyncio/timeouts.py", line 111, in __aexit__ raise TimeoutError from exc_val TimeoutError 2023-10-13T15:57:08+13:00 CRITICAL - failed/ERR