Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Github action for pulling in nightly builds from CircleCI #29

Closed
wants to merge 1 commit into from

Conversation

cameel
Copy link
Member

@cameel cameel commented Jul 2, 2020

Part of ethereum/solidity#9258.

What works

  1. Cloning solc-bin repo without checking out all the 10 GB of data. This makes it run in under a minute (compared to ~5 minutes with full checkout).
    • A partial checkout (using git clone --filter blob:none) makes it almost instant but then git downloads all the blobs when you commit anyway. Maybe I could get it to stop doing that somehow but 1 minute is already good enough, even for debugging.
    • Theoretically a shallow clone (git clone --depth 1) should make it download less data but that doesn't seem to be the case and even makes the whole operation run slower (~30 seconds longer).
  2. Getting job information and downloading binaries from CircleCI without any tokens.
  3. Running the update script.
  4. Committing changes and pushing them back to the repo.

It's already running in my fork of solc-bin.

What's still in progress

  • For now I'm taking binaries from any recent b_ems job, even if it's still a PR. I need to switch to getting them only from develop.
    • The issue here is that the API endpoint gives me at most 100 recent jobs and jobs from develop run only once per day so they can be several pages down the list. I need to start using the offset query parameter for pagination and do multiple requests.
  • CircleCI API does not provide information about tags other than those directly on the current commit so I can't use Solidity version in the file name yet.
    • Maybe I can get it using Github API. In the worst case I'll have to clone the Solidity repo just to get the tag.
  • Binaries from tagged commits are not distinguished from ordinary nightlies yet.
  • I'm not adding symlinks to wasm/ directory yet.
  • The update script needs to be modified to just append the nightly instead of rewriting the whole list.
    • I'm not checking out the old binaries so currently it removes them all from the lists.
  • Add an extra validation that the CircleCI job produced exactly one artifact.
    • Maybe also a check to ensure that the files are not truncated or damaged in some way?
  • Don't try to commit and push if there were no changes.

Other

  • I need an e-mail address to put in the committer field (currently it's pull-soljson action <solidity@example.com>).
  • I have created a master branch in the repo and set it as the branch where the action adds new commits. Since there are external tools that rely on binaries hosted on GH pages, I think that it would be best to consider gh-pages frozen until we get S3 ready to serve the files instead and to work on master in the meantime.

GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

steps:
- name: Clone the repository without checking out a working copy
Copy link
Member

@ekpyron ekpyron Jul 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will run once every night without oversight - are 5 minutes for a full checkout really worth avoiding?

.github/workflows/pull-soljson.yml Show resolved Hide resolved
@cameel
Copy link
Member Author

cameel commented Jul 3, 2020

After getting some feedback, the approach changed radically so I decided to close this PR and open a new one (#30) instead.

@cameel cameel closed this Jul 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants