Auto-GPT Packaging

[collijk]

Background

Auto-GPT is not a proper python package. It depends on custom and pretty complicated installation steps.

Related Issues

There are many, many, many issues related to installation and setup. Here is a sampling:

• cant find API key although it is in the .env file #4990
• clean docker installation: ModuleNotFoundError: No module named 'autogpt.memory_item' #4989
• can't install packages  #4978
• ImportError: Bad git executable. #4939
• Base url not working #4869
• TypeError: 'numpy._DTypeMeta' object is not subscriptable #4850
• AutoGPT writing to multiple workspaces depending on how it is ran #4821
• When having files in the plugin directory, I get this error (even if the configuration is set to false for all the plugins). #4817
• No such file or directory source issue #4814
• While performing installation encounter -> python3.11-minimal package post-installation script subprocess returned error exit status 1 #4805
• Run Auto-GPT Error  #4800
• run.sh fails due to python issue #4768
• prompt_toolkit.output.win32.NoConsoleScreenBufferError #4753
• import requests ModuleNotFoundError: No module named 'requests'. Infinite loop #4731
• installation problem #4792
• So many more

Related PRs

There are fewer open issues, but there have been a few attempts to clean up installation and move us to modern packaging.

• Removed docker-compose.yaml example #4970
• Adding plugins_config.yaml.template #4849
• Make Auto-GPT pypi or testpypi installable #4833
• [UX] Magic CLI with docker integration and bootstrapping capability #4668
• startup menu for ai_settings.yaml to support multiple configurations #4646
• Create Publish Package to TestPyPi #2506
• Drop requirements.txt in favor of pyproject.toml and add autogpt entrypoint #1955
• Migrate to poetry #1102

The Plan:

• We need to find out the platform-specific, non-python dependencies for AutoGPT. We'll need to list these in the install instructions. I'm running into a host of these on linux for the TTS stuff. Install instructions will have some brew/apt-get/whatever windows uses instructions to install those things.
• Do some code motion. There are a number of issues with the repo layout right now.
  • Swap to a src layout to ensure we're running and testing installed versions of the code.
  • Clean up a handful of cruft files that are leftover from earlier implementations.
  • Move settings files out of the repository and into a user directory (as the user may never clone the repo).
• Detailed file movement plan:
  • /autogpt -> /src/autogpt/
  • /data/CURRENT_BULLETIN.md -> {USER_AGPT_DIRECTORY}/CURRENT_BULLETIN.md
  • /logs/ -> {USER_AGPT_DIRECTORY}/agents/{AGENT_NAME}/logs
  • /plugins/ -> {USER_AGPT_DIRECTORY}/plugins
  • /scripts/ -> /src/autogpt/scripts/
  • .env.template -> /dev/null
  • .env -> {USER_AGPT_DIRECTORY}/.env
  • ai_settings.yaml -> {USER_AGPT_DIRECTORY}/agents/{AGENT_NAME}/ai_settings.yaml
  • azure.yaml.template -> {USER_AGPT_DIRECTORY}/azure.yaml
  • benchmarks.py -> /agbenchmark/benchmarks.py
  • data_ingestion.py -> /dev/null
  • main.py -> /dev/null (never used?)
  • plugins_config.yaml -> {USER_AGPT_DIRECTORY}/agents/{AGENT_NAME}/plugins_config.yaml
  • prompt_settings.yaml -> {USER_AGPT_DIRECTORY}/prompt_settings.yaml
  • run* -> /dev/null
  • tests.py -> /dev/null
• Add a autogpt setup command that can make the USER_AGPT_DIRECTORY and fill it with all the necessaries.
• Use poetry to configure our builds and setup deployment to PyPI (pending some testing from Luke)
• Integrate poetry into CI
• Update docs

The Caveats

• This changes the user workflow. They'll no longer need to clone the repo, but things will no longer be in the places they're used to. That's pretty annoying.
• This changes the developer workflow (though not in any significant way).

The Benefits

• We look like a normal python package on PyPI (expanding our audience)
• We can interactively guide the user through setup, instead of making them configure from docs.
• We reduce maintenance overhead by using a standard layout and standard tools.
• Deploying docker images without issues becomes a lot easier/more stable.
• We enable other people to build apps using autogpt as a lib.
• Users will get the often-requested feature of being able to maintain multiple agents at once.

The Timeline

Release 0.5.0 (before the end of July).

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.