Foundryup failure due to GitHub API rate limiting

[frangio]

Getting intermittent failures from foundryup in slither-action as seen here. The error doesn't seem abort the script and foundryup reports that it finished installing.

[-] Foundry target detected, installing foundry nightly
curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)
foundryup: downloading latest forge and cast

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: downloading manpages

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: line 128: /opt/foundry/bin/forge: No such file or directory
foundryup: installed - 
foundryup: line 129: /opt/foundry/bin/cast: No such file or directory
foundryup: installed - 
foundryup: done

In the first two lines we see curl returns 403 status and the next line shows nightly-.

curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)

What seems to be failing is this request:

foundry/foundryup/foundryup

Lines 79 to 84 in f10df79

	SHA=$(ensure curl -sSf https://api.github.com/repos/${FOUNDRYUP_REPO}/git/refs/tags/nightly \
	| grep -Eo '"sha"[^,]*' \
	| grep -Eo '[^:]*$' \
	| tr -d '"' \
	| tr -d ' ')
	FOUNDRYUP_TAG="nightly-${SHA}"

Unauthenticated requests to the GitHub API are heavily rate limited:

For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the person making requests.

This seems to make foundryup not well suited for CI environments in its current form. Perhaps it should use the nightly tag without a commit hash?

Consider using set -o pipefail in addition to set -e. I think this is why the error doesn't abort the script.

[mattsse]

my bashscript skills are pretty limited, but we def should handle this.

https://gist.github.com/mohanpedala/1e2ff5661761d3abd0385e8223e16425#set--o-pipefail

from what I understand adding set -o pipefail would be useful.

[frangio]

Yeah that should make the script at least catch this failure.

[frangio]

Hm, it seems like ensure should be doing that but it doesn't seem to be working as intended.

foundry/foundryup/foundryup

Lines 239 to 244 in b78509f

	# Run a command that should never fail. If the command fails execution
	# will immediately terminate with an error showing the failing
	# command.
	ensure() {
	if ! "$@"; then err "command failed: $*"; fi
	}

[sambacha]

i tried to fix this but was closed #2221

I actually have the fix here: https://github.com/foundry-rs/foundry/pull/3907/files#diff-79a844de4d6869f243ecd1e7ca4d9532d9a4987f85ca63d9f4af85aa227ecb38R22-R32

[sambacha]

Getting intermittent failures from foundryup in slither-action as seen here. The error doesn't seem abort the script and foundryup reports that it finished installing.

[-] Foundry target detected, installing foundry nightly
curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)
foundryup: downloading latest forge and cast

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: downloading manpages

######################################################################## 100.0%

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
foundryup: line 128: /opt/foundry/bin/forge: No such file or directory
foundryup: installed - 
foundryup: line 129: /opt/foundry/bin/cast: No such file or directory
foundryup: installed - 
foundryup: done

In the first two lines we see curl returns 403 status and the next line shows nightly-.

curl: (22) The requested URL returned error: 403 
foundryup: installing foundry (version nightly, tag nightly-)

What seems to be failing is this request:

foundry/foundryup/foundryup

Lines 79 to 84 in f10df79

	SHA=$(ensure curl -sSf https://api.github.com/repos/${FOUNDRYUP_REPO}/git/refs/tags/nightly \
	| grep -Eo '"sha"[^,]*' \
	| grep -Eo '[^:]*$' \
	| tr -d '"' \
	| tr -d ' ')
	FOUNDRYUP_TAG="nightly-${SHA}"

Unauthenticated requests to the GitHub API are heavily rate limited:

For unauthenticated requests, the rate limit allows for up to 60 requests per hour. Unauthenticated requests are associated with the originating IP address, and not the person making requests.

This seems to make foundryup not well suited for CI environments in its current form. Perhaps it should use the nightly tag without a commit hash?

Consider using set -o pipefail in addition to set -e. I think this is why the error doesn't abort the script.

The real issue is that GitHub has no direct way to directly link to the latest build from GitHub actions of a given repository.

Even if you do have a link to an artifact, using it requires the visitor to be logged into the GitHub website.¹

Because GitHub doesn't provide any permanent and public links to an artifact (this being the 'latest'), you should use a > service redirect to time-limited links that GitHub can give to the application -- only on behalf of an authenticated user that has access to the repository. So, whenever someone downloads an artifact from a repository that you had added, it would use a service token that is associated with your installation of the GitHub App. -- https://nightly.link/

Also, these are pre-releases so a service like https://nightly.link/ wont actually work

Here is a script we use to download the latest releases in json file, https://github.com/sambacha/foundry-release/blob/master/get-release.sh

just parse that to get your download link

Footnotes

1. see https://github.com/actions/upload-artifact/issues/51 ↩

[collide143]

forge init hello_foundry && cd hello_foundry

[grandizzy]

the change was added in #6352 PR which addressed similar issue #6329, please reopen if still an issue. thank you!

[sambacha]

This only correctly fixes the foundryup issue for exiting. The issue for rate limiting is still here. The correct way to solve the rate limiting issue would be the following:

Conditional API Request for ETag version check

Making a conditional request does not count against your primary rate limit if a 304 response is returned.
source, GitHub REST API Documentation

• function that handles both SHA and ETag validation
• Uses GitHub's API version specification and SHA-specific accept header
• store the ETags and SHAs in a cache directory for persistence

Conditional ETag check for downloading binary example

the GITHUB_API_SHA should be stored or checked against local installation.

download() {
  local url=$1
  local output=$2

  if [ -z "$GITHUB_API_SHA" ]; then
    err "GITHUB_API_SHA is missing. Set the SHA for comparison."
  fi

  local etag_header=""
  if [ -n "$GITHUB_API_ETAG" ]; then
    etag_header="--header 'If-None-Match: \"$GITHUB_API_ETAG\"'"
  fi

  if check_cmd curl; then
    response=$(curl -#L -w "%{http_code}" -o "$output" $etag_header \
      --header "X-GitHub-Api-Version:2022-11-28" \
      --header "Accept: application/vnd.github.sha" \
      -D - "$url")

    new_etag=$(echo "$response" | grep -i '^ETag:' | awk '{print $2}' | tr -d '\r')

    if [ "$response" = "304" ]; then
      say "No update needed, file is current"
      return 0
    else
      GITHUB_API_ETAG="$new_etag"
    fi

  else
    response=$(wget --header "If-None-Match: \"$GITHUB_API_ETAG\"" \
      --header "X-GitHub-Api-Version:2022-11-28" \
      --header "Accept: application/vnd.github.sha" \
      -O "$output" "$url" -S --quiet 2>&1)

    new_etag=$(echo "$response" | grep -i '^ETag:' | awk '{print $2}' | tr -d '\r')

    if [[ "$response" != *"304 Not Modified"* ]]; then
      GITHUB_API_ETAG="$new_etag"
    fi
  fi

  computed_sha=$(sha256sum "$output" | awk '{print $1}')
  if [ "$computed_sha" != "$GITHUB_API_SHA" ]; then
    say "SHA mismatch! Expected: $GITHUB_API_SHA, Got: $computed_sha"
    return 1
  else
    say "SHA matches the expected value."
  fi
}

[frangio]

This issue seems to have been fixed by #6155 which removed API requests from the script.