Support for ARM?

[richin13]

Bug description

Hello, I'm trying to run the prisma commands and to use the client on a raspberry pi that uses ARM64, I noticed the binaries that get downloaded are built for x86. Is there any way I can generate those binaries myself? Or download the appropriate version for my system?

How to reproduce

Run an application on an ARM64 machine, running any command or attempting to use the client will result int:

$ python3 -m prisma db push
Traceback (most recent call last):
  File "/home/ubuntu/.asdf/installs/python/3.9.6/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/ubuntu/.asdf/installs/python/3.9.6/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/apps/bsc-tools/venv/lib/python3.9/site-packages/prisma/__main__.py", line 6, in <module>
    cli.main()
  File "/home/ubuntu/apps/bsc-tools/venv/lib/python3.9/site-packages/prisma/cli/cli.py", line 39, in main
    sys.exit(prisma.run(args[1:]))
  File "/home/ubuntu/apps/bsc-tools/venv/lib/python3.9/site-packages/prisma/cli/prisma.py", line 44, in run
    process = subprocess.run(
  File "/home/ubuntu/.asdf/installs/python/3.9.6/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/home/ubuntu/.asdf/installs/python/3.9.6/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/ubuntu/.asdf/installs/python/3.9.6/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/tmp/prisma/binaries/engines/1c9fdaa9e2319b814822d6dbfd0a69e1fcc13a85/prisma-cli-linux'

Downloaded binary file result:

$ file /tmp/prisma/binaries/engines/1c9fdaa9e2319b814822d6dbfd0a69e1fcc13a85/prisma-cli-linux
/tmp/prisma/binaries/engines/1c9fdaa9e2319b814822d6dbfd0a69e1fcc13a85/prisma-cli-linux: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.18, BuildID[sha1]=2106a959eb86690fe4628f369dd4b325b515ed02, with debug_info, not stripped

Expected behavior

Ideally, to download the appropriate version for my system arch but pointing to instructions on how to build these binaries myself can suffice.

Prisma information

Worth noting that npx prisma@3.4.0 db push worked fine but the issue appears again when I try to run my python script where I use the Client from prisma.

Environment & setup

• OS: Linux ubuntu 5.13.0-1011-raspi #13-Ubuntu SMP PREEMPT Fri Nov 19 18:40:23 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
• Database: sqlite
• Python version: 3.9.6
• Prisma version: 0.4.2

prisma               : 3.4.0
prisma client python : 0.4.2
platform             : debian-openssl-1.1.x
engines              : 1c9fdaa9e2319b814822d6dbfd0a69e1fcc13a85
install path         : /home/ubuntu/apps/bsc-tools/venv/lib/python3.9/site-packages/prisma
installed extras     : []

[RobertCraigie]

It would be difficult for us to provide binaries for the raspberry pi architecture as our current setup relies completely on the precompiled binaries that Prisma provides, of which the raspberry pi architecture is not included (prisma/prisma#5379 (comment)). However we can provide a better error message for this case, I will create a separate issue to track this.

However you can build the binaries yourself by following these steps:

1. Clone the prisma-engines repository at the current version that the python client supports:

   git clone https://github.com/prisma/prisma-engines --branch=3.4.0

2. Build the binaries following the steps at https://github.com/prisma/prisma-engines#building-prisma-engines
3. Make sure all 4 binaries are executable using chmod +x <binary path>
4. Set the following environment variables:

   PRISMA_QUERY_ENGINE_BINARY=/path/to/query-engine
   PRISMA_MIGRATION_ENGINE_BINARY=/path/to/migration-engine
   PRISMA_INTROSPECTION_ENGINE_BINARY=/path/to/introspection-engine
   PRISMA_FMT_BINARY=/path/to/prisma-fmt
   

It should be noted that I have not tested these steps so please let me know if you encounter any issues or have any other questions!

[RobertCraigie]

I have just realised I have misread your original issue, the error is due to the CLI binary, not the engine binaries. The CLI binary path is actually not configurable, I should fix this.

Here's the bash script used to build the CLI:

#!/bin/sh

set -eux

v="3.4.0"

mkdir -p build
cd build
npm init --yes
npm i "pkg" --dev
npm i "prisma@$v" --dev
npm i "@prisma/client@$v"
npx prisma version

mkdir -p node_modules/prisma/node_modules/@prisma/engines
cp -R node_modules/@prisma/engines/* node_modules/prisma/node_modules/@prisma/engines

npx pkg -t node12-linux node_modules/prisma

However, I am doubtful that this will work due to the explicit target passed to pkg, see https://github.com/vercel/pkg#targets.

Hopefully you can get this to work for you and then you can replace the CLI binary that is already downloaded with the one you just built.

I do not know if this is something that can be easily solved by us, I will look into it further.

[richin13]

@RobertCraigie thanks for the help here, I ended up doing both suggestions: building the engine binaries myself (and setting the corresponding env vars) and building the cli using node12-linux-arm64 as the pkg -t.

Also I came across this comment (prisma/prisma#861 (comment)) but I wasn't able to find the pre-built binaries for the engines in https://binaries.prisma.sh/...

My issue seems resolved, I'm able to run my scripts now in the RPI so we can close this but I agree it could be helpful to have a clearer error message if possible.

Thank you!

[RobertCraigie]

@richin13 I'm glad you could get it to work!

Could you please let me know what the following script outputs? I don't have access to an arm64 machine which makes testing difficult.

import platform
print(platform.architecture())
print(platform.system())
print(platform.machine()) 

[richin13]

@RobertCraigie Sure, here's the output:

>>> import platform
>>> print(platform.architecture())
('64bit', 'ELF')
>>> print(platform.system())
Linux
>>> print(platform.machine())
aarch64
>>> print(platform.processor())
aarch64

[RobertCraigie]

Thank you!

[a7ul]

I was trying to run prisma python on a docker container in M1 mac (arm64 container by default)
Was facing the same issue where prisma python by default downloads the x86_64 binary for cli

I ended up with this solution (posting here for anyone who is in the same situation)

setup-prisma-arm64.sh

#!/bin/sh
set -eux

export PRISMA_VERSION="3.13.0" # Set it to the prisma client version used by prisma python
export TARGE_DIRECTORY="$PRISMA_BINARY_CACHE_DIR"

function install_dependencies() {
    apt update && apt install -y curl
    curl -fsSL https://deb.nodesource.com/setup_16.x | bash -
    apt install -y nodejs
}

function build_prisma_cli() {
    # Package CLI: https://github.com/RobertCraigie/prisma-client-py/issues/195#issuecomment-1001287195
    export PRISMA_CLI_QUERY_ENGINE_TYPE=binary
    
    mkdir -p /tmp/prisma_cli
    cd /tmp/prisma_cli
    npm init --yes
    npm i "pkg" --dev
    npm i "prisma@$PRISMA_VERSION" --dev
    npm i "@prisma/client@$PRISMA_VERSION"
    npx prisma version
    
    # Build cli executable
    npx pkg -t node16-linux node_modules/prisma -o prisma-cli-linux
    
    # Copy to correct path where prisma expects the binaries to be
    mkdir -p $TARGE_DIRECTORY
    mv prisma-cli-linux $TARGE_DIRECTORY/prisma-cli-linux
    mv node_modules/@prisma/engines/introspection-engine-linux-arm64-openssl-1.1.x $TARGE_DIRECTORY/prisma-introspection-engine-debian-openssl-1.1.x
    mv node_modules/@prisma/engines/query-engine-linux-arm64-openssl-1.1.x $TARGE_DIRECTORY/prisma-query-engine-debian-openssl-1.1.x
    mv node_modules/@prisma/engines/migration-engine-linux-arm64-openssl-1.1.x $TARGE_DIRECTORY/prisma-migration-engine-debian-openssl-1.1.x
    mv node_modules/@prisma/engines/prisma-fmt-linux-arm64-openssl-1.1.x $TARGE_DIRECTORY/prisma-prisma-fmt-debian-openssl-1.1.x
    
    # Cleanup
    rm -rf /tmp/prisma_cli
}

function install_prisma_cli() {
    # Only run the script in arm64 arch since prisma cli by default doesnt package binaries in arm64
    CURRENT_ARCH=$(dpkg --print-architecture)
    if [[ "$CURRENT_ARCH" != "arm64" ]]; then
        return
    fi
    
    install_dependencies
    build_prisma_cli
}

install_prisma_cli

Dockerfile

FROM python:3.11-slim

ENV PRISMA_BINARY_CACHE_DIR="/tmp/.prisma-cache"

COPY ./setup-prisma-arm64.sh /setup-prisma-arm64.sh
RUN bash /setup-prisma-arm64.sh

...
...
...

Essentially, the custom script would only run in cases where the architecture is arm64 and it would download the binaries to the folder where prisma client python is expecting the binaries to be.

For x86_64 things will run as usual

[RobertCraigie]

Hey @a7ul! Sorry I should've commented on this issue more recently....

Support for ARM is now in main and will be included in the next release. This works by directly using the Node CLI instead of the packaged CLI.

Sorry to have caused you more work than was necessary.

[a7ul]

Wow thats amazing @RobertCraigie 🙌🏽
Even better 😄

[pimmesz]

Any clue when this would be released (or which version it would be?)? This would add support for running it on a raspberry pi?

[RobertCraigie]

Hey @pimmesz, this feature was included in the latest release.

However, I'm unsure of the support for raspberry pi. There is this open issue which references this workaround which has now been archived stating that it's recommended to use a 64 bit pi, would that be possible for you to use?

[pimmesz]

Hi @RobertCraigie. I've opened an issue at the Prisma repo as well. I am currently running 64 bit pi. If I run uname -m it returns aarch64 so I don't really understand where this goes wrong?

[RobertCraigie]

Hey @pimmesz, sorry I missed your latest comment but it looks like the folks over at Prisma managed to resolve the issue for you! Let me know if you have any more questions :)