Improve dependencies management

[Tansito]

What is the expected enhancement?

During our short lifetime we found several problems with pip installation, like: #900

From the different proposals that I found implementing a lock file similar to other languages could solve our problem:

• https://pipenv.pypa.io/en/latest/pipfile/

I'm not the most expert in Python so if someone knows a better library I'm 100% open to it. Special cc to @IceKhan13 here, due to his knowledge in this field.

[psschwei]

re: the PR linked above -- we're setting different dependency versions in the dockerfiles vs. requirements.txt

[psschwei]

I wonder if part of the issue is the backlog of dependabot updates: #76

[Tansito]

At least my intention with the issue is different. My idea is to provide a lock where all the dependencies are specified included dependencies of dependencies. Something similar to what NPM has with the package-lock: https://docs.npmjs.com/cli/v10/configuring-npm/package-lock-json

[Tansito]

I found this answer really interesting commenting pro's and con's of the different approaches to implement dependencies lock files:

• https://stackoverflow.com/a/52665767

Looking in depth pip-tools because is less opinionated or invasive than poetry and pipenv in the sense that it continues letting you use venv, conda or whatever environment you have installed. Very promising.

[josephtedds]

When you come back to this, you might want to take a look at uv. It can act as a drop-in replacement for both pip and pip-tools, but is much, much faster.

I ran it on the dependencies from 0.9.0 + a few extras, and it took the resolution time down from hours to ~15 seconds. Even without implementing lock files etc., it should still save significant amounts of time for you in github actions.

[Tansito]

Wow, thank you for the recommendation @josephtedds . We will definitely take a look 😄

[djmcgreal-cc]

May I ask also what the intention is around the execution environment's base environment and how it supports a general development workflow?

Including a base environment in the image seems like a good idea to simplify execution but doesn't it also place restrictions on what code (i.e. dependencies that can be compatible with its version choices) can be successfully executed?

Right now we're considering building our own execution image which will add to developer workflow complexity - is that sensible/the intention? Is the contract of such an image stable/known?

It sounds like #1257, if picked up would help.

Thanks (including for your patience, I'm going to this from a place of minor understanding!).

[Tansito]

May I ask also what the intention is around the execution environment's base environment and how it supports a general development workflow?

Of course you can and it's a really good question, in fact. The project is evolving really fast and we hope to bring new features in the next versions, by now for 0.9 and 0.10 the way that we are designing the development workflow would be something similar to:

• You start with the creation of your Python code locally
• To create your Qiskit Pattern and run it you should identify: dependencies, environment variables and arguments
• Modify your Python code to accept environment variables and arguments to create the Qiskit Pattern and specify the required dependencies
• Once time you have it you are ready to upload and run it

Including a base environment in the image seems like a good idea to simplify execution but doesn't it also place restrictions on what code (i.e. dependencies that can be compatible with its version choices) can be successfully executed?

Yep, totally. That's one of the reasons because we offer different images with different python versions, for example. In the near future, we will add security features like the possibility to configure dependencies whitelists. It will add more restrictions on what code can be executed but it's important from a security point of view. Pros and cons I suppose.

Right now we're considering building our own execution image which will add to developer workflow complexity - is that sensible/the intention? Is the contract of such an image stable/known?

Well, I don't know your use-case so maybe is a good idea, maybe not. But I personally would go 100% of the times for the simplest solution for the developer.

Thanks (including for your patience, I'm going to this from a place of minor understanding!).

No problem! Happy to answer any question!