Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace dedicated host_tools CI job with superset build_all. #10195

Merged
merged 6 commits into from
Aug 25, 2022
Merged

Replace dedicated host_tools CI job with superset build_all. #10195

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
d962d47
Replace dedicated host_tools CI job with superset build_all.
ScottTodd Aug 24, 2022
f1a6925
Swap ordering in `needs:` to see if that affects grouping.
ScottTodd Aug 24, 2022
8d2de4f
Merge remote-tracking branch 'upstream/main' into ci-host-tools-merge
ScottTodd Aug 25, 2022
ebb075b
Extract just the relevant directory
ScottTodd Aug 25, 2022
049f14a
Extract just the install folder in more places.
ScottTodd Aug 25, 2022
9ceae9e
Revert riscv64 build dir extract change (it uses both).
ScottTodd Aug 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 58 additions & 86 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ jobs:
# dependent jobs.
build-dir: ${{ env.BUILD_DIR }}
build-dir-archive: ${{ steps.archive.outputs.build-dir-archive }}
gcs-artifact: ${{ steps.upload.outputs.gcs-artifact }}
build-dir-gcs-artifact: ${{ steps.upload.outputs.build-dir-gcs-artifact }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was keeping this terse with the rationale that which gcs-artifact it is is already implied by the job name and this can only be referenced relative to the job name. I guess the same could be said for "build-dir-archive" here, which could just be "archive". Not sure that "dir" really stands on its own though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about generating build-dir-archive and install-dir-archive, at which point gcs-artifact would be ambiguous. I ended up just using the single archive, but then figured the naming clarification / consistency would be useful regardless.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I don't have super strong opinions. I think there's some value to brevity when it's already namespaced, but 🤷

steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
Expand Down Expand Up @@ -138,11 +138,11 @@ jobs:
- name: "Uploading build dir archive"
id: upload
env:
GCS_ARTIFACT: ${{ env.GCS_DIR }}/${{ steps.archive.outputs.build-dir-archive }}
BUILD_DIR_ARCHIVE: ${{ steps.archive.outputs.build-dir-archive }}
BUILD_DIR_GCS_ARTIFACT: ${{ env.GCS_DIR }}/${{ steps.archive.outputs.build-dir-archive }}
run: |
gcloud alpha storage cp "${BUILD_DIR_ARCHIVE}" "${GCS_ARTIFACT}"
echo "::set-output name=gcs-artifact::${GCS_ARTIFACT}"
gcloud alpha storage cp "${BUILD_DIR_ARCHIVE}" "${BUILD_DIR_GCS_ARTIFACT}"
echo "::set-output name=build-dir-gcs-artifact::${BUILD_DIR_GCS_ARTIFACT}"

build_test_all_bazel:
needs: should_run
Expand Down Expand Up @@ -183,15 +183,15 @@ jobs:
env:
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
GCS_ARTIFACT: ${{ needs.build_all.outputs.gcs-artifact }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
with:
submodules: true
- name: "Downloading build dir archive"
run: gcloud alpha storage cp "${GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting archive"
run: gcloud alpha storage cp "${BUILD_DIR_GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting build dir archive"
run: tar -xf "${BUILD_DIR_ARCHIVE}"
- name: "Testing all"
run: |
Expand All @@ -213,7 +213,7 @@ jobs:
env:
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
GCS_ARTIFACT: ${{ needs.build_all.outputs.gcs-artifact }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
Expand All @@ -224,8 +224,8 @@ jobs:
./build_tools/scripts/check_cuda.sh
./build_tools/scripts/check_vulkan.sh
- name: "Downloading build dir archive"
run: gcloud alpha storage cp "${GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting archive"
run: gcloud alpha storage cp "${BUILD_DIR_GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting build dir archive"
run: tar -xf "${BUILD_DIR_ARCHIVE}"
- name: "Testing with GPU"
run: |
Expand Down Expand Up @@ -287,11 +287,11 @@ jobs:
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
with:
submodules: true
- name: "Downloading runtime build directory"
- name: "Downloading runtime build dir archive"
uses: actions/download-artifact@fb598a63ae348fa914e94cd0ff38f362e927b741 # v3.0.0
with:
name: "${{ env.BUILD_DIR }}.tar"
- name: "Extracting archive"
- name: "Extracting runtime build dir archive"
run: tar -xf ${BUILD_DIR}.tar
- name: "Testing runtime"
run: |
Expand All @@ -301,14 +301,6 @@ jobs:
./build_tools/cmake/ctest_all.sh \
"${BUILD_DIR}"

host_tools_assertions:
needs: should_run
if: needs.should_run.outputs.should-run == 'true'
uses: ./.github/workflows/host_tools.yml
with:
host-binary-root: host-tools-assertions
enable-assertions: "ON"

################################# Tensorflow #################################
# Jobs that build the IREE-Tensorflow integrations
##############################################################################
Expand All @@ -324,7 +316,7 @@ jobs:
outputs:
binaries-dir: ${{ steps.build.outputs.binaries-dir }}
binaries-archive: ${{ steps.archive.outputs.binaries-archive }}
gcs-artifact: ${{ steps.upload.outputs.gcs-artifact }}
binaries-gcs-artifact: ${{ steps.upload.outputs.binaries-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
Expand Down Expand Up @@ -353,14 +345,14 @@ jobs:
- name: "Uploading binaries archive"
id: upload
env:
GCS_ARTIFACT: ${{ env.GCS_DIR }}/${{ steps.archive.outputs.binaries-archive }}
BINARIES_ARCHIVE: ${{ steps.archive.outputs.binaries-archive }}
BINARIES_GCS_ARTIFACT: ${{ env.GCS_DIR }}/${{ steps.archive.outputs.binaries-archive }}
run: |
gcloud alpha storage cp "${BINARIES_ARCHIVE}" "${GCS_ARTIFACT}"
echo "::set-output name=gcs-artifact::${GCS_ARTIFACT}"
gcloud alpha storage cp "${BINARIES_ARCHIVE}" "${BINARIES_GCS_ARTIFACT}"
echo "::set-output name=binaries-gcs-artifact::${BINARIES_GCS_ARTIFACT}"

test_tf_integrations:
needs: [should_run, build_tf_integrations, build_all]
needs: [should_run, build_all, build_tf_integrations]
if: needs.should_run.outputs.should-run == 'true'
runs-on:
# Pseudo-ternary hack and order matters. See comment at top of file.
Expand All @@ -371,10 +363,10 @@ jobs:
env:
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.gcs-artifact }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
TF_BINARIES_DIR: ${{ needs.build_tf_integrations.outputs.binaries-dir }}
TF_BINARIES_ARCHIVE: ${{ needs.build_tf_integrations.outputs.binaries-archive }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.gcs-artifact }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.binaries-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
Expand All @@ -399,7 +391,7 @@ jobs:
"${BUILD_DIR}"

test_tf_integrations_gpu:
needs: [should_run, build_tf_integrations, build_all]
needs: [should_run, build_all, build_tf_integrations]
if: needs.should_run.outputs.should-run == 'true'
runs-on:
# Pseudo-ternary hack and order matters. See comment at top of file.
Expand All @@ -410,10 +402,10 @@ jobs:
env:
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.gcs-artifact }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
TF_BINARIES_DIR: ${{ needs.build_tf_integrations.outputs.binaries-dir }}
TF_BINARIES_ARCHIVE: ${{ needs.build_tf_integrations.outputs.binaries-archive }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.gcs-artifact }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.binaries-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
Expand Down Expand Up @@ -486,7 +478,7 @@ jobs:
./build_tools/cmake/build_and_test_tsan.sh

build_benchmarks:
needs: [should_run, host_tools_assertions, build_tf_integrations]
needs: [should_run, build_all, build_tf_integrations]
if: needs.should_run.outputs.should-run == 'true'
runs-on:
# Hacks, and order matters. See the comment at the top of the file.
Expand All @@ -495,23 +487,21 @@ jobs:
- cpu
- os-family=Linux
env:
HOST_BINARY_ROOT: ${{ needs.host_tools_assertions.outputs.host-binary-root }}
HOST_BINARY_ARCHIVE: ${{ needs.host_tools_assertions.outputs.host-binary-root }}.tar
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
TF_BINARIES_DIR: ${{ needs.build_tf_integrations.outputs.binaries-dir }}
TF_BINARIES_ARCHIVE: ${{ needs.build_tf_integrations.outputs.binaries-archive }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.gcs-artifact }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.binaries-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
with:
submodules: true
- name: "Downloading host tools"
uses: actions/download-artifact@fb598a63ae348fa914e94cd0ff38f362e927b741 # v3.0.0
with:
name: "${{ env.HOST_BINARY_ARCHIVE }}"
- name: "Extracting host tools archive"
run: |
tar -xvf ${HOST_BINARY_ARCHIVE}
- name: "Downloading build dir archive"
run: gcloud alpha storage cp "${BUILD_DIR_GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting install from build dir archive"
run: tar -xf "${BUILD_DIR_ARCHIVE}" "${BUILD_DIR}/install"
ScottTodd marked this conversation as resolved.
Show resolved Hide resolved
- name: "Downloading TF binaries archive"
run: gcloud alpha storage cp "${TF_BINARIES_GCS_ARTIFACT}" "${TF_BINARIES_ARCHIVE}"
- name: "Extracting TF binaries archive"
Expand All @@ -520,15 +510,15 @@ jobs:
run: |
build_tools/github_actions/docker_run.sh \
--env "IREE_TF_BINARIES_DIR=${TF_BINARIES_DIR}" \
--env "IREE_HOST_BINARY_ROOT=${HOST_BINARY_ROOT}" \
--env "IREE_HOST_BINARY_ROOT=${BUILD_DIR}/install" \
gcr.io/iree-oss/base@sha256:5d43683c6b50aebe1fca6c85f2012f3b0fa153bf4dd268e8767b619b1891423a \
build_tools/cmake/build_benchmarks.sh

############################## Crosscompilation ##############################
# Jobs that cross-compile IREE for other platforms
##############################################################################
android_arm64:
needs: [should_run, host_tools_assertions]
needs: [should_run, build_all]
if: needs.should_run.outputs.should-run == 'true'
runs-on:
# Hacks, and order matters. See the comment at the top of the file.
Expand All @@ -537,32 +527,29 @@ jobs:
- cpu
- os-family=Linux
env:
BUILD_DIR: "build-android-arm64"
ANDROID_ABI: "arm64-v8a"
HOST_BINARY_ROOT: ${{ needs.host_tools_assertions.outputs.host-binary-root }}
HOST_BINARY_ARCHIVE: ${{ needs.host_tools_assertions.outputs.host-binary-root }}.tar
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
with:
submodules: true
- name: "Downloading host tools"
uses: actions/download-artifact@fb598a63ae348fa914e94cd0ff38f362e927b741 # v3.0.0
with:
name: "${{ env.HOST_BINARY_ARCHIVE }}"
- name: "Extracting host tools archive"
run: |
tar -xvf ${HOST_BINARY_ARCHIVE}
- name: "Downloading build dir archive"
run: gcloud alpha storage cp "${BUILD_DIR_GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting install from build dir archive"
run: tar -xf "${BUILD_DIR_ARCHIVE}" "${BUILD_DIR}/install"
- name: "Building for Android"
run: |
build_tools/github_actions/docker_run.sh \
--env "ANDROID_ABI=${ANDROID_ABI}" \
--env "IREE_HOST_BINARY_ROOT=${HOST_BINARY_ROOT}" \
--env "IREE_HOST_BINARY_ROOT=${BUILD_DIR}/install" \
gcr.io/iree-oss/android@sha256:9bc723fc707a18bd0c1be9c12e01ea5bb7c7d77f607427879e10ffcffd7d2bb5 \
build_tools/cmake/build_android.sh

riscv32:
needs: [should_run, host_tools_assertions]
needs: [should_run, build_all]
if: needs.should_run.outputs.should-run == 'true'
runs-on:
# Pseudo-ternary hack and order matters. See comment at top of file.
Expand All @@ -573,32 +560,30 @@ jobs:
env:
BUILD_RISCV_DIR: "build-riscv-rv32-baremetal"
RISCV_CONFIG: "rv32-baremetal"
HOST_BINARY_ROOT: ${{ needs.host_tools_assertions.outputs.host-binary-root }}
HOST_BINARY_ARCHIVE: ${{ needs.host_tools_assertions.outputs.host-binary-root }}.tar
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
with:
submodules: true
- name: "Downloading host tools"
uses: actions/download-artifact@fb598a63ae348fa914e94cd0ff38f362e927b741 # v3.0.0
with:
name: "${{ env.HOST_BINARY_ARCHIVE }}"
- name: "Extracting host tools archive"
run: |
tar -xvf ${HOST_BINARY_ARCHIVE}
- name: "Downloading build dir archive"
run: gcloud alpha storage cp "${BUILD_DIR_GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting install from build dir archive"
run: tar -xf "${BUILD_DIR_ARCHIVE}" "${BUILD_DIR}/install"
- name: "Cross-compiling and testing riscv32"
run: |
./build_tools/github_actions/docker_run.sh \
--env "RISCV_CONFIG=${RISCV_CONFIG}" \
--env "BUILD_RISCV_DIR=${BUILD_RISCV_DIR}" \
--env "IREE_HOST_BINARY_ROOT=${HOST_BINARY_ROOT}" \
--env "IREE_HOST_BINARY_ROOT=${BUILD_DIR}/install" \
gcr.io/iree-oss/riscv@sha256:720bc0215d8462ea14352edc22710a6ce4c0c1daff581d179dd173885f1d8a35 \
bash -euo pipefail -c \
"./build_tools/cmake/build_riscv.sh && tests/riscv32/smoke.sh"

riscv64:
needs: [should_run, build_all, host_tools_assertions, build_tf_integrations]
needs: [should_run, build_all, build_tf_integrations]
if: needs.should_run.outputs.should-run == 'true'
runs-on:
# Pseudo-ternary hack and order matters. See comment at top of file.
Expand All @@ -609,43 +594,31 @@ jobs:
env:
BUILD_RISCV_DIR: "build-riscv-r64"
RISCV_CONFIG: "rv64"
HOST_BINARY_ROOT: ${{ needs.host_tools_assertions.outputs.host-binary-root }}
HOST_BINARY_ARCHIVE: ${{ needs.host_tools_assertions.outputs.host-binary-root }}.tar
BUILD_DIR: ${{ needs.build_all.outputs.build-dir }}
BUILD_DIR_ARCHIVE: ${{ needs.build_all.outputs.build-dir-archive }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.gcs-artifact }}
BUILD_DIR_GCS_ARTIFACT: ${{ needs.build_all.outputs.build-dir-gcs-artifact }}
TF_BINARIES_DIR: ${{ needs.build_tf_integrations.outputs.binaries-dir }}
TF_BINARIES_ARCHIVE: ${{ needs.build_tf_integrations.outputs.binaries-archive }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.gcs-artifact }}
TF_BINARIES_GCS_ARTIFACT: ${{ needs.build_tf_integrations.outputs.binaries-gcs-artifact }}
steps:
- name: "Checking out repository"
uses: actions/checkout@7884fcad6b5d53d10323aee724dc68d8b9096a2e # v2
with:
submodules: true
- name: "Downloading host tools"
uses: actions/download-artifact@fb598a63ae348fa914e94cd0ff38f362e927b741 # v3.0.0
with:
name: "${{ env.HOST_BINARY_ARCHIVE }}"
- name: "Extracting host tools archive"
run: |
tar -xvf ${HOST_BINARY_ARCHIVE}
- name: "Downloading build dir archive"
run: gcloud alpha storage cp "${BUILD_DIR_GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting build dir archive"
run: tar -xf "${BUILD_DIR_ARCHIVE}"
- name: "Downloading TF binaries archive"
run: gcloud alpha storage cp "${TF_BINARIES_GCS_ARTIFACT}" "${TF_BINARIES_ARCHIVE}"
- name: "Extracting TF binaries archive"
run: tar -xf "${TF_BINARIES_ARCHIVE}"
# FIXME: The riscv64 tests currently need the entire build dir archive
# just to get FileCheck from LLVM's bin directory. This is silly and we
# should make FileCheck available another way.
- name: "Downloading build dir archive"
run: gcloud alpha storage cp "${BUILD_DIR_GCS_ARTIFACT}" "${BUILD_DIR_ARCHIVE}"
- name: "Extracting archive"
run: tar -xf "${BUILD_DIR_ARCHIVE}"
- name: "Cross-compiling and testing riscv64"
run: |
./build_tools/github_actions/docker_run.sh \
--env "RISCV_CONFIG=${RISCV_CONFIG}" \
--env "BUILD_RISCV_DIR=${BUILD_RISCV_DIR}" \
--env "IREE_HOST_BINARY_ROOT=${HOST_BINARY_ROOT}" \
--env "IREE_HOST_BINARY_ROOT=${BUILD_DIR}/install" \
--env "IREE_IMPORT_TFLITE_BIN=${TF_BINARIES_DIR}/iree-import-tflite" \
--env "LLVM_BIN_DIR=${BUILD_DIR}/third_party/llvm-project/llvm/bin" \
gcr.io/iree-oss/riscv@sha256:720bc0215d8462ea14352edc22710a6ce4c0c1daff581d179dd173885f1d8a35 \
Expand All @@ -671,7 +644,6 @@ jobs:
# Subsets
- build_runtime
- test_runtime
- host_tools_assertions

# Tensorflow
- test_tf_integrations
Expand Down
Loading