Difficulties with script that uses Terracotta package

[mkleinha]

I am trying to make a portable program that starts a local raster tile server. I am using the terracotta package and cx_freeze to generate an executable. The script runs fine in python, but I get these errors when I run the executable:

Traceback (most recent call last):
  File "{virtual_environment_path}\Lib\site-packages\cx_Freeze\initscripts\__startup__.py", line 141, in run
    module_init.run(name + "__main__")
  File "{virtual_environment_path}\Lib\site-packages\cx_Freeze\initscripts\console.py", line 25, in run
    exec(code, main_globals)
  File "test_freeze_tc.py", line 38, in <module>
  File "test_freeze_tc.py", line 32, in start_server
  File "{virtual_environment_path}\Lib\site-packages\flask\app.py", line 625, in run
    run_simple(t.cast(str, host), port, self, **options)
  File "{virtual_environment_path}\Lib\site-packages\werkzeug\serving.py", line 1084, in run_simple
    srv = make_server(
          ^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\werkzeug\serving.py", line 937, in make_server
    return BaseWSGIServer(
           ^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\werkzeug\serving.py", line 783, in __init__
    self.socket = socket.fromfd(fd, address_family, socket.SOCK_STREAM)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\{USERNAME}\AppData\Local\Programs\Python\Python312\Lib\socket.py", line 548, in fromfd
    nfd = dup(fd)
          ^^^^^^^
OSError: [WinError 10038] An operation was attempted on something that is not a socket
127.0.0.1 - - [15/Jul/2024 11:52:18] "GET /singleband/randomExample/preview.png?tile_size=[128,128] HTTP/1.1" 500 -
Traceback (most recent call last):
  File "{virtual_environment_path}\Lib\site-packages\flask\app.py", line 1498, in __call__
    return self.wsgi_app(environ, start_response)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\flask\app.py", line 1476, in wsgi_app
    response = self.handle_exception(e)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\flask\app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\flask\app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\flask\app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\flask\app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\terracotta\server\singleband.py", line 181, in get_singleband_preview
    return _get_singleband_image(keys)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\terracotta\server\singleband.py", line 197, in _get_singleband_image
    image = singleband(parsed_keys, tile_xyz=tile_xyz, **options)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\{USERNAME}\AppData\Local\Programs\Python\Python312\Lib\contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\terracotta\handlers\singleband.py", line 50, in singleband
    tile_data = xyz.get_tile_data(
                ^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\terracotta\xyz.py", line 28, in get_tile_data
    return driver.get_raster_tile(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\terracotta\drivers\terracotta_driver.py", line 285, in get_raster_tile
    return self.raster_store.get_raster_tile(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{virtual_environment_path}\Lib\site-packages\terracotta\drivers\geotiff_raster_store.py", line 173, in get_raster_tile
    result = future.result()
             ^^^^^^^^^^^^^^^
  File "C:\Users\{USERNAME}\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\{USERNAME}\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

Where the path to the virtual environment has been replaced with {virtual_environment_path} and my username has been replaced with {USERNAME}.

This is a python script that reproduces these errors:

import terracotta
from terracotta.server import create_app

database = 'layers.sqlite'
port = 5050

def start_server(
    db_path: str = None,
    port: int = 5050):
        
        host = "localhost"
        debug = True
        profile = False
        database_provider = None
        
        terracotta.update_settings(
            DRIVER_PATH=database,
            DRIVER_PROVIDER=database_provider,
            DEBUG=debug,
            FLASK_PROFILE=profile,
        )

        driver = terracotta.get_driver(database, provider=database_provider)
        
        with driver.connect():
            pass

        server_app = create_app(debug=debug, profile=profile)

        server_app.run(port=port, 
                       host=host, 
                       threaded=False,
                      )

if __name__ == '__main__':
    start_server(database, port)

The setup script I have been using is as follows:

from cx_Freeze import setup, Executable

build_exe_options = {
    "packages": ["sqlalchemy.dialects.sqlite"],
	"include_files": [("layers.sqlite")],
}

setup(
    name="Start Terracotta Tileserver",
    version="0.1",
    description="Start Terracotta Tileserver",
    options={"build_exe": build_exe_options},
    executables=[Executable(script="test_freeze_tc.py", base="console",)],
)

The script depends on the attached file randomExample_layer.tif.
example_raster.zip

I create a virtual environment and install the terracotta(0.8.3) and cx_freeze(7.2.0.dev48) packages using pip. In order to create a SQLite database for the server to use to interact with the cloud optimized raster, I run this command:

terracotta ingest {data}_layer.tif -o layers.sqlite

I was able to solve one error by commenting out some of the source code in the terracotta package that was unnecessary for my application. I replaced lines 15-21 in terracotta/cmaps/get_cmaps.py with just line 21.

try:
    PACKAGE_DIR = resource_filename(
        Requirement.parse("terracotta"), "terracotta/cmaps/data"
    )
except DistributionNotFound:
    # terracotta was not installed, fall back to file system
    PACKAGE_DIR = os.path.join(os.path.dirname(__file__), "data")

was replaced with

PACKAGE_DIR = os.path.join(os.path.dirname(__file__), "data")

The errors shown at the beginning of the post are not thrown immediately after the executable is run; a separate process must try to interact with the server. After the server is running, I run this command in another command prompt instance to test whether the server is serving tiles properly:

terracotta connect localhost:5050

I’m using Python 3.12.4 on Windows 11. Please let me know if there is any other information that would be helpful in diagnosing this issue. I have dug around in the source code for the different packages involved in the error and have been unsuccessful in isolating the issue. Any help is appreciated.

[mkleinha]

I spent some more time digging into this and found a solution. I guess the concurrent.futures module used by terracotta has similar requirements to the multiprocessing module in that
multiprocessing.freeze_support()
must be included in the main module as described here:
https://cx-freeze.readthedocs.io/en/stable/faq.html#multiprocessing-support