Delegate binary downloading to prisma

[RobertCraigie]

Problem

With many possible configurations and settings, binary downloading is complicated, as such we should see if it is possible to delegate this task to the prisma as they already handle all of it for us.

This is an issue as we currently don't respect any binary targets options or http proxy options like prisma does.

We would still have to download the CLI ourselves at the very least first.

[janpio]

The generator process of Prisma CLI prisma generate can also take care of downloading and supplying the Query Engine to a generator artifact. The API is mostly untested (and I would actually have to dig it up), but if you want to work on that, ping me on our public Slack and I can def make something happen.

Maybe it would also help to describe how you currently handle the engines here (and the CLI as well probably).

[RobertCraigie]

@janpio Thank you! I will definitely make use of that if I start working on this.

prisma generate can also take care of downloading and supplying the Query Engine to a generator artifact

I did notice that while reading through some prisma code and added it but don't make use of it:

prisma-client-py/src/prisma/jsonrpc.py

Line 35 in 33dc2c0

requiresEngines: Optional[List[str]]

Do you know if prisma passes the resolved engine binary to the generator?

[janpio]

My current understanding would be that it should pass a path to the engine, yes. But I am not aware of anyone using that, so might also be utterly broken. A quick read through the generator code on Prisma side might give clarity if this can even work.

[RobertCraigie]

You are correct it does pass the engine path to the generator.

However even though the native binaries are successfully downloaded, the following error is thrown when running prisma generate with and without binary targets.

  prisma:loadEnv skipping package.json at /package.json +0ms
  prisma:tryLoadEnv Environment variables not found at null +0ms
  prisma:tryLoadEnv Environment variables not found at undefined +1ms
  prisma:tryLoadEnv No Environment variables loaded +0ms
  prisma:engines binaries to download query-engine, migration-engine, introspection-engine, prisma-fmt +0ms
Prisma schema loaded from tests/data/schema.prisma
  plusX Execution permissions of /private/var/folders/ql/0v8h20s972s6zz4t_3qc49bc0000gp/T/prisma-binaries/query-engine-darwin are fine +0ms
  prisma:getConfig Using Query Engine Binary at: /private/var/folders/ql/0v8h20s972s6zz4t_3qc49bc0000gp/T/prisma-binaries/query-engine-darwin +0ms
  plusX Execution permissions of /private/var/folders/ql/0v8h20s972s6zz4t_3qc49bc0000gp/T/prisma-binaries/query-engine-darwin are fine +505ms
  prisma:getDMMF Using Query Engine Binary at: /private/var/folders/ql/0v8h20s972s6zz4t_3qc49bc0000gp/T/prisma-binaries/query-engine-darwin +0ms
  prisma:getGenerators neededVersions {
  "60b19f4a1de4fe95741da371b4c44a92f4d1adcb": {
    "engines": [
      "queryEngine"
    ],
    "binaryTargets": [
      {
        "fromEnvVar": null,
        "value": "darwin"
      },
      {
        "fromEnvVar": null,
        "value": "rhel-openssl-1.0.x"
      }
    ]
  }
} +0ms
  prisma:download file /snapshot/node_modules/prisma/query-engine-darwin does not exist and must be downloaded +0ms
  prisma:download file /snapshot/node_modules/prisma/query-engine-rhel-openssl-1.0.x does not exist and must be downloaded +0ms
Error: Error: File '/**/node_modules/prisma/query-engine-darwin' was not included into executable at compilation stage. Please recompile adding it as asset or script.
    at error_ENOENT (pkg/prelude/bootstrap.js:446:17)
    at readFileFromSnapshot (pkg/prelude/bootstrap.js:729:29)
    at readFile (pkg/prelude/bootstrap.js:796:5)
    at internal/util.js:297:30
    at new Promise (<anonymous>)
    at readFile (internal/util.js:296:12)
    at maybeCopyToTmp (/snapshot/node_modules/prisma/build/index.js:26765:26)
    at async download (/snapshot/node_modules/prisma/build/index.js:26562:41)
    at async getBinaryPathsByVersion (/snapshot/node_modules/prisma/build/index.js:36480:43)
    at async Object.getGenerators (/snapshot/node_modules/prisma/build/index.js:36403:36)

I can provide a full repro if that would help.

[janpio]

That would be neat.

Are you using pkg to package the CLI? That is often a pain... (at least it is for our Go Client which does similar things) But of course there is no other real alternative.

[RobertCraigie]

Yes, I have used pkg before and it was not fun to say the least.

This actually uses the exact same CLI binary as the Go Client does and the engine binaries are handled in a similar way (I essentially just rewrote the Go Client's solution in python).

Here's a reproduction for the issue, it assumes you have python3 installed.
https://github.com/RobertCraigie/prisma-reproductions/tree/main/py-22