Python RuntimeError when trying to scan LiDAR point cloud data

[danielrode]

I have successfully initialized a new TileDB database and am now trying to move onto the next step: scanning. Python throws a RuntimeError when I try to use the scan function of SilviMetric. This happened with my own data, but I can reproduce the error with test data from the SilviMetric GitHub repository as well.

System: OpenSUSE Tumbleweed (Linux)
Python version: 3.11

Here is my workflow (in Bash):

# Setup Python virtual environment
mkdir ~/working
cd ~/working
python3 -m venv .venv  # Create virtual environment
source .venv/bin/activate  # Activate virtual environment

# Install SilviMetric requirements
python3 -m pip install click dask 'dask[distributed]' pyproj numpy tiledb \
     scipy dill pdal gdal==3.8.4 pandas 'dask[dataframe]'

# Install SilviMetric into virtual environment
git clone https://github.com/hobuinc/silvimetric
pip install ./silvimetric

# Initialize SilviMetric database
# TileBD expects lower left coordinate pair first, then upper right
bounds='[495579, 4780091, 695301, 4978956]'
   silvimetric --database database.tdb initialize \
     --bounds "$bounds" --crs "EPSG:6342"
fi

# Scan?
path="$HOME"/working/silvimetric/tests/data/autzen-small.copc.laz 
silvimetric --database database.tdb scan "$path"

When I run that last command, I get the error:

2024-04-10 09:59:10,013 - silvimetric - INFO - extent_handle:67 - Chunking 1 tiles at depth 0
2024-04-10 09:59:10,013 - silvimetric - INFO - extent_handle:67 - Chunking 1 tiles at depth 0
2024-04-10 09:59:10,013 - silvimetric - INFO - extent_handle:67 - Chunking 1 tiles at depth 0
2024-04-10 09:59:10,013 - silvimetric - INFO - extent_handle:67 - Chunking 1 tiles at depth 0
2024-04-10 09:59:10,048 - distributed.worker - WARNING - Compute Failed
Key:       ('tile_info-bag-from-delayed-1852343071d0de96aa6fe6c175b4f3aa', 0)
Function:  execute_task
args:      ((<function reify at 0x7f7e7f262fc0>, (<function tile_info at 0x7f7e62d06840>, ([635559.00,639009.00],[848856.00,853566.00]), <silvimetric.resources.data.Data object at 0x7f7e62d22a90>, 100.0, 600000, 6)))
kwargs:    {}
Exception: 'RuntimeError("No opening \'(\'.")'

Traceback (most recent call last):
  File "/home/daniel/working/.venv/bin/silvimetric", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/silvimetric/cli/cli.py", line 90, in scan_cmd
    return scan.scan(app.tdb_dir, pointcloud, bounds, point_count, resolution,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/silvimetric/commands/scan.py", line 24, in scan
    cell_counts = extent_handle(extents, data, resolution, point_count,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/silvimetric/commands/scan.py", line 68, in extent_handle
    n = curr.compute()
        ^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/dask/base.py", line 375, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/dask/base.py", line 661, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/silvimetric/commands/scan.py", line 85, in tile_info
    pc = data.estimate_count(extent.bounds)
      ^^^^^^^^^^^^^^^^^
  File "/home/daniel/working/.venv/lib64/python3.11/site-packages/silvimetric/resources/data.py", line 153, in estimate_count
    qi = pipeline.quickinfo[reader.type]
  ^^^^^^^^^^^^^^^^^
RuntimeError: No opening '('.

[kylemann16]

Hey @danielrode thanks for posting! I'm taking a look at this, but currently unable to replicate from the main branch of silvimetric and copying your environment process.

Would you mind providing environment versions with

pip list
which pdal
pdal --version

Thank you!

[danielrode]

pip list:

Package            Version
------------------ -----------
certifi            2024.2.2
click              8.1.7
cloudpickle        3.0.0
dask               2024.4.1
dask-expr          1.0.11
dill               0.3.8
distributed        2024.4.1
fsspec             2024.3.1
GDAL               3.8.4
importlib_metadata 7.1.0
Jinja2             3.1.3
locket             1.0.0
MarkupSafe         2.1.5
msgpack            1.0.8
numpy              1.26.4
packaging          24.0
pandas             2.2.1
partd              1.4.1
pdal               3.3.0
pip                24.0
psutil             5.9.8
pyarrow            15.0.2
pyproj             3.6.1
python-dateutil    2.9.0.post0
pytz               2024.1
PyYAML             6.0.1
scipy              1.13.0
setuptools         65.5.0
silvimetric        1.0.0
six                1.16.0
sortedcontainers   2.4.0
tblib              3.0.0
tiledb             0.27.1
toolz              0.12.1
tornado            6.4
tzdata             2024.1
urllib3            2.2.1
zict               3.0.0
zipp               3.18.1

which pdal --> /usr/bin/pdal

pdal --version --> pdal 2.5.6 (git-version: Release)

[danielrode]

Update: I get the same No opening '(' RuntimeError when running shatter as well:

silvimetric --log-level DEBUG --database autzen-smdb.tdb shatter https://s3.amazonaws.com/hobu-lidar/autzen-classified.copc.laz --date 2001-01-01:

2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,167 - silvimetric - INFO - chunk:71 - Filtering 1 tiles at depth 0
2024-04-10 16:43:34,203 - distributed.worker - WARNING - Compute Failed
Key:       ('filter-bag-from-delayed-f8b79650d1280add110b061dd0bb38fa', 0)
Function:  execute_task
args:      ((<function reify at 0x7f7c2fe35440>, (<function Extents.filter at 0x7f7c13917600>, ([635549.20,639029.20],[848856.21,853566.21]), <silvimetric.resources.data.Data object at 0x7f7c13937050>, 100, 600000, 6)))
kwargs:    {}
Exception: 'RuntimeError("No opening \'(\'.")'

Traceback (most recent call last):
  File "/home/daniel/bh/.venv/bin/silvimetric", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/click/decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/cli/cli.py", line 161, in shatter_cmd
    shatter.shatter(config)
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/commands/shatter.py", line 179, in shatter
    leaves = extents.chunk(data, 100)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/resources/extents.py", line 72, in chunk
    n = curr.compute()
        ^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/dask/base.py", line 375, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/dask/base.py", line 661, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/resources/extents.py", line 106, in filter
    pc = data.estimate_count(self.bounds)
  ^^^^^^^^^^^^^^^^^
  File "/home/daniel/bh/.venv/lib64/python3.11/site-packages/silvimetric/resources/data.py", line 153, in estimate_count
    qi = pipeline.quickinfo[reader.type]
  ^^^^^^^^^^^^^^^^^
RuntimeError: No opening '('.

[kylemann16]

The error you're seeing is a PDAL error message referring to the bounds created, and I don't think this exists in the latest version of PDAL.

I can't replicate your error, and I'm fairly certain it's because of the mixing of environments. The latest version of python-pdal that you have installed should be using pdal version 2.7.1 or newer, but this error is coming from the pdal installed in your /usr/bin.

Relevant issue in pdal python: PDAL/python#164

If you have conda available to you, the easiest solution would be to use that to manage your environment with conda env create -f environment.yml && conda activate silvimetric from the base directory of the project. If you're stuck with source installs and pip, then I would update the pdal you have installed in /usr/bin and see if that fixes it.

This error can be replicated through pdal directly:

(a) ~/code/silvimetric % m list pdal
# packages in environment at /Users/kmann/mambaforge/envs/a:
#
# Name                    Version                   Build  Channel
pdal                      2.5.6                hb8312d3_4    conda-forge
(a) ~/code/silvimetric % pdal info tests/data/test_data.copc.laz --readers.copc.bounds '[495579, 4780091, 695301, 4978956]'
PDAL: No opening '('.
(a) ~/code/silvimetric % m upgrade pdal
...
(a) ~/code/silvimetric % m list pdal                                                                                       
# packages in environment at /Users/kmann/mambaforge/envs/a:
#
# Name                    Version                   Build  Channel
pdal                      2.7.1                hff7fe72_4    conda-forge
(a) ~/code/silvimetric % pdal info tests/data/test_data.copc.laz --readers.copc.bounds '[495579, 4780091, 695301, 4978956]'
{
  "file_size": 399984,
  "filename": "tests/data/test_data.copc.laz",
  "now": "2024-04-11T10:19:45-0500",
  "pdal_version": "2.7.1 (git-version: Release)",
  "reader": "readers.copc",

[danielrode]

I tried the Conda approach shown in the SilviMetric QuickStart doc. However, that resulted in a libglog.so.2: cannot open shared object file error on my machine. I may open a separate issue about that later.

I got things working using just the native OpenSUSE package manager and venv/pip. Looks like older versions of PDAL expect the ([minx, maxx], [miny, maxy]) bounds format, but SilviMetric uses [minx, miny, maxx, maxy] format internally when passing the bounds object to PDAL. I patched that so the code now works with my older version of PDAL (which is currently the latest version of PDAL available in OpenSUSE repos). Here's the modification I made: 1b6e973

If you are interested in adding this to the main repo, let me know and I can make a PR. Otherwise, feel free to close this issue.

Specifying a minimum expected PDAL version number in the guide/docs is an alternative solution, but I don't know when PDAL switched from expecting that specific bounds format to the several currently supported in the latest version.