Support downloading archive which do not contain executable

[iilyak]

Feature Overview

Add ability to use aquaproj to download/verify/extract archive into specified folder

If this can be somehow achieved already extend documentation to cover this use case.

Why is the feature needed?

I wanted to use aqua to build a tool from source. In particular I am interesting in building lua. The main distribution method for lua is a source tarball. I want aqua to act as generic downloader of archive source artifacts with ability to specify version and validate checksums before I would attempt to build the source.

I think this feature could be implemented via introduction of an additional flag in package definition for http package type.

  - name: lua/lua
    type: http
    repo_owner: Lua.org
    repo_name: lua
    url: https://www.lua.org/ftp/lua-{{trimV .Version}}.tar.gz
    description: Lua is a lightweight, high-level, multi-paradigm programming language
    enable_exe_validation: false # this is a new flag which is true by default

Other naming ideas:

• enable_exe_validation
• check_executable_presence
• update_exe_permission
• ensure_executable_exists

Workaround

No response

Example Code

bash-3.2$  AQUA_LOG_LEVEL=debug aqua install lua/lua
DEBU[0000] install the proxy                             aqua_version=2.8.0 env=darwin/arm64 package_name=aqua-proxy package_version=v1.2.0 program=aqua registry=
DEBU[0000] check if aqua-proxy is already installed      aqua_version=2.8.0 env=darwin/arm64 package_name=aqua-proxy package_version=v1.2.0 program=aqua registry=
DEBU[0000] no version_constraint                         aqua_version=2.8.0 env=darwin/arm64 package_name=lua/lua package_version=5.4.4 program=aqua registry=local
DEBU[0000] install the package                           aqua_version=2.8.0 env=darwin/arm64 package_name=lua/lua package_version=5.4.4 program=aqua registry=local
DEBU[0000] check if the package is already installed     aqua_version=2.8.0 env=darwin/arm64 package_name=lua/lua package_version=5.4.4 program=aqua registry=local
INFO[0000] download and unarchive the package            aqua_version=2.8.0 env=darwin/arm64 package_name=lua/lua package_version=5.4.4 program=aqua registry=local
ERRO[0002] check file_src is correct                     aqua_version=2.8.0 env=darwin/arm64 error="check file_src is correct: exe_path isn't found: stat /Users/iilyak/.local/share/aquaproj-aqua/pkgs/http/www.lua.org/ftp/lua-5.4.4.tar.gz/lua: no such file or directory" exe_path=/Users/iilyak/.local/share/aquaproj-aqua/pkgs/http/www.lua.org/ftp/lua-5.4.4.tar.gz/lua file_name=lua package_name=lua/lua package_version=5.4.4 program=aqua registry=local
ERRO[0002] executable files aren't found
Files in the unarchived package (Only 30 files are shown):
lua-5.4.4/Makefile
lua-5.4.4/doc/contents.html
lua-5.4.4/doc/index.css
lua-5.4.4/doc/logo.gif
lua-5.4.4/doc/lua.1
lua-5.4.4/doc/lua.css
lua-5.4.4/doc/luac.1
lua-5.4.4/doc/manual.css
lua-5.4.4/doc/manual.html
lua-5.4.4/doc/osi-certified-72x60.png
lua-5.4.4/doc/readme.html
lua-5.4.4/src/Makefile
lua-5.4.4/src/lapi.c
lua-5.4.4/src/lapi.h
lua-5.4.4/src/lauxlib.c
lua-5.4.4/src/lauxlib.h
lua-5.4.4/src/lbaselib.c
lua-5.4.4/src/lcode.c
lua-5.4.4/src/lcode.h
lua-5.4.4/src/lcorolib.c
lua-5.4.4/src/lctype.c
lua-5.4.4/src/lctype.h
lua-5.4.4/src/ldblib.c
lua-5.4.4/src/ldebug.c
lua-5.4.4/src/ldebug.h
lua-5.4.4/src/ldo.c
lua-5.4.4/src/ldo.h
lua-5.4.4/src/ldump.c
lua-5.4.4/src/lfunc.c
lua-5.4.4/src/lfunc.h
   aqua_version=2.8.0 env=darwin/arm64 package_name=lua/lua package_version=5.4.4 program=aqua registry=local
ERRO[0002] install the package                           aqua_version=2.8.0 env=darwin/arm64 error="check file_src is correct" package_name=lua/lua package_version=5.4.4 program=aqua registry=local
FATA[0002] aqua failed                                   aqua_version=2.8.0 env=darwin/arm64 error="it failed to install some packages" program=aqua
bash-3.2$

Configuration

cat aqua.yaml
# aqua - Declarative CLI Version Manager
# https://aquaproj.github.io/
registries:
  - type: standard
    ref: v4.139.0 # renovate: depName=aquaproj/aqua-registry
  - name: local
    type: local
    path: registry.yaml
packages:
  - name: lua/lua@5.4.4
    registry: local


cat registry.yaml
packages:
  - name: lua/lua
    type: http
    repo_owner: Lua.org
    repo_name: lua
    url: https://www.lua.org/ftp/lua-{{trimV .Version}}.tar.gz
    description: Lua is a lightweight, high-level, multi-paradigm programming language

cat aqua-checksums.json
{
    "checksums": [
      {
        "id": "https://www.lua.org/ftp/lua-5.4.4.tar.gz",
        "checksum": "164c7849653b80ae67bec4b7473b884bf5cc8d2dca05653475ec2ed27b9ebf61",
        "algorithm": "sha256"
      }
    ]
}

cat aqua-policy.yaml
---
# aqua Policy
# https://aquaproj.github.io/
registries:
  - type: standard
    ref: semver(">= 3.0.0")
  - name: local
    type: local
    path: registry.yaml

packages:
# Example
# - registry: local # allow all packages in the Registry
# - name: cli/cli # allow only a specific package. The default value of registry is "standard"
# - name: cli/cli
#   version: semver(">= 2.0.0") # version is optional
  - registry: local

Note

No response

[iilyak]

Ability to specify destination would also be helpful.

  - name: lua/lua
    type: http
    repo_owner: Lua.org
    repo_name: lua
    url: https://www.lua.org/ftp/lua-{{trimV .Version}}.tar.gz
    description: Lua is a lightweight, high-level, multi-paradigm programming language
    dest_dir: .cache/lua-src

[iilyak]

Another possibility is to extend the https://aquaproj.github.io/docs/reference/registry-config/files with convention

packages:
  - name: lua/lua
    type: http
    repo_owner: Lua.org
    repo_name: lua
    url: https://www.lua.org/ftp/lua-{{trimV .Version}}.tar.gz
    description: Lua is a lightweight, high-level, multi-paradigm programming language
    files:
       - name: lua
          src: /
          dir: lua-src

The presence of / at the end of a string in src would indicate that the content of archive doesn't contain executable.

[iilyak]

dir shouldn't start with / or contain series of ../ to escape from project directory.

[suzuki-shunsuke]

Thank you for your proposal.

This is related to the issue. #1899 (comment)

As I mentioned in #1899 (comment) ,
aqua doesn't support executing any external commands to install tools.

There are some exceptional package types.

• go_build
• go_install
• cargo

We allow these package types because what they can do is limited and secure.

BTW, in case of lua, how does aqua know how to build it?
In your example there is no instruction about the build.

If lua provides any standard and secure commands like go install, go build, and cargo install, perhaps we can support it as new package types.

I'm not familiar with Lua at all.

[iilyak]

I am not asking to run the command. I need to

• download archive
• verify hash
• extract the files into specified directory

nothing more than that. the building would be done by other tools. I provided config for lua just to simplify reproduction of the problem.

[suzuki-shunsuke]

Oh, I see. Thank you for your explanation. I misunderstood.

The idea is interesting, but I don't think this is the feature aqua should support.

[iilyak]

Turned out it is not difficult to implement. Here is the result of my experiment https://github.com/aquaproj/aqua/compare/main...iilyak:aqua:add-support-for-directories?expand=1. I am not opening the PR since it is not the feature aqua wants. Please let me know if you think it would benefit aqua.