Prisma Python is missing binaries when frozen in CI but not locally

[ntindle]

When I freeze the Prisma python package as part of my CI using GitHub Actions, it has an error when run:

PS C:\Users\nicka> agpt_server.exe
Traceback (most recent call last):
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\src\cx_Freeze\cx_Freeze\initscripts\__startup__.py", line 141, in run
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\src\cx_Freeze\cx_Freeze\initscripts\console.py", line 25, in run
  File "autogpt_server\app.py", line 46, in <module>
  File "autogpt_server\app.py", line 34, in main
  File "autogpt_server\app.py", line 26, in run_processes
  File "autogpt_server\app.py", line 22, in run_processes
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\util\process.py", line 50, in start
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\util\process.py", line 28, in execute_run_command
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\util\service.py", line 64, in run
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\asyncio\base_events.py", line 649, in run_until_complete
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\data\db.py", line 10, in connect
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\_base_client.py", line 430, in connect
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\engine\_query.py", line 349, in connect
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\engine\_query.py", line 58, in _ensure_file
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\engine\utils.py", line 111, in ensure
prisma.engine.errors.BinaryNotFoundError: Expected C:\Users\nicka\prisma-query-engine-windows.exe, C:\Users\nicka\.cache\prisma-python\binaries\5.11.0\efd2449663b3d73d637ea1fd226bafbcf45b3102\prisma-query-engine-windows.exe or C:\Users\runneradmin\.cache\prisma-python\binaries\5.11.0\efd2449663b3d73d637ea1fd226bafbcf45b3102\node_modules\prisma\query-engine-windows.exe to exist but none were found or could not be executed.
Try running prisma py fetch

Pipeline: https://github.com/Significant-Gravitas/AutoGPT/actions/runs/9950216596?pr=7406

I think what is happening is query-engine-windows.exe needs to be available when it's not. Not super sure how to debug this. I've added the prisma py fetch to the CI pipeline but that didn't fix. It may be some config with Prisma via https://github.com/RobertCraigie/prisma-client-py/blob/main/src/prisma/_config.py#L49 is needed, but I'm not sure yet.

Our setup.py is below, but all the code is open source and available in this branch: Significant-Gravitas/AutoGPT#7406

import platform
from pathlib import Path
from pkgutil import iter_modules
from typing import Union
import shutil
import os

from cx_Freeze import Executable, setup  # type: ignore

packages = [
    m.name
    for m in iter_modules()
    if m.ispkg and m.module_finder and "poetry" in m.module_finder.path  # type: ignore
]
packages.append("collections")
packages.append("autogpt_server.util.service")
packages.append("autogpt_server.executor.manager")
packages.append("autogpt_server.util.service")

# set the icon based on the platform
icon = "../../assets/gpt_dark_RGB.ico"
if platform.system() == "Darwin":
    icon = "../../assets/gpt_dark_RGB.icns"
elif platform.system() == "Linux":
    icon = "../../assets/gpt_dark_RGB.png"


def txt_to_rtf(input_file: Union[str, Path], output_file: Union[str, Path]) -> None:
    """
    Convert a text file to RTF format.

    Args:
    input_file (Union[str, Path]): Path to the input text file.
    output_file (Union[str, Path]): Path to the output RTF file.

    Returns:
    None
    """
    input_path = Path(input_file)
    output_path = Path(output_file)

    with input_path.open("r", encoding="utf-8") as txt_file:
        content = txt_file.read()

    # RTF header
    rtf = r"{\rtf1\ansi\deff0 {\fonttbl {\f0 Times New Roman;}}\f0\fs24 "

    # Replace newlines with RTF newline
    rtf += content.replace("\n", "\\par ")

    # Close RTF document
    rtf += "}"

    with output_path.open("w", encoding="utf-8") as rtf_file:
        rtf_file.write(rtf)


# Convert LICENSE to LICENSE.rtf
license_file = "LICENSE.rtf"
txt_to_rtf("../../LICENSE", license_file)

# call npm run build in ../autogpt_builder


os.system("npm run build --prefix ../autogpt_builder")

# copy the ../autogpt_builder/out to frontend


shutil.rmtree("../frontend", ignore_errors=True)
shutil.copytree("../autogpt_builder/out", "../frontend")

setup(
    name="AutoGPT Server",
    url="https://agpt.co",
    # The entry points of the application
    executables=[
        Executable(
            "autogpt_server/app.py",
            target_name="agpt_server",
            base="console",
            icon=icon,
        ),
        Executable(
            "autogpt_server/cli.py",
            target_name="agpt_server_cli",
            base="console",
            icon=icon,
        ),
    ],
    options={
        # Options for building all the executables
        "build_exe": {
            "packages": packages,
            "includes": [
                "autogpt_server",
                "prisma",
            ],
            # Exclude the two module from readability.compat as it causes issues
            "excludes": ["readability.compat.two"],
            "include_files": [
                # source, destination in the bundle
                # (../frontend, frontend) would also work but you'd need to load the frontend differently in the data.py to correctly get the path when frozen
                ("../frontend", "frontend"),
                ("./secrets", "secrets"),
            ],
        },
        # Mac .app specific options
        "bdist_mac": {
            "bundle_name": "AutoGPT",
            "iconfile": "../../assets/gpt_dark_RGB.icns",
        },
        # Mac .dmg specific options
        "bdist_dmg": {
            "applications_shortcut": True,
            "volume_label": "AutoGPTServer",
            "background": "builtin-arrow",
            "license": {
                "default-language": "en_US",
                "licenses": {"en_US": license_file},
                "buttons": {
                    "en_US": [
                        "English",
                        "Agree",
                        "Disagree",
                        "Print",
                        "Save",
                        "If you agree, click Agree to continue the installation. If you do not agree, click Disagree to cancel the installation.",
                    ]
                },
            },
        },
        # Windows .msi specific options
        "bdist_msi": {
            "target_name": "AutoGPTServer",
            "add_to_path": True,
            "install_icon": "../../assets/gpt_dark_RGB.ico",
            "license_file": license_file,
        },
        # Linux .appimage specific options
        "bdist_appimage": {},
        # Linux rpm specific options
        "bdist_rpm": {
            "name": "AutoGPTServer",
            "description": "AutoGPT Server",
            "version": "0.1",
            "license": "UNKNOWNORPROPRIETARY",
            "url": "https://agpt.co",
            "long_description": "AutoGPT Server",
        },
    },
)

[ntindle]

Setting these environment variables seem to do nothing in the frozen environment, but does work in the non frozen

[ntindle]

Copying the exe into place manually at run time seems to still fail:

{"is_panic":false,"message":"Error querying the database: Error code 14: Unable to open the database file","backtrace":" 0: \n 1: \n 2: \n 3: \n 4: \n 5: \n 6: \n 7: \n 8: BaseThreadInitThunk\n 9: RtlUserThreadStart\n"}
{"is_panic":false,"message":"Error querying the database: Error code 14: Unable to open the database file","backtrace":" 0: \n 1: \n 2: \n 3: \n 4: \n 5: \n 6: \n 7: \n 8: BaseThreadInitThunk\n 9: RtlUserThreadStart\n"}

[ntindle]

Will work on making a minimal repro. The biggest problem with testing this is prisma installs stuff outside its site-packages and in doing so, basically means you can't test it on the machine you build on

[ntindle]

Repro here: https://github.com/ntindle/cx_freeze_prisma_reproduction
It is worth pointing out you MUST build in ci (or another PC), as your paths will be valid from the poetry install in the repo once you run cx_freeze.

[marcelotduarte]

#2509