Add Gradle's 'assemble' check to catch the distribution generation issues

[reta]

Description

Add Gradle's 'assemble' check to catch the distribution generation issues

Related Issues

Related to #8903

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21092/
• CommitID: 04122f1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21094/
• CommitID: bb841c2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21095/
• CommitID: 4eabd76
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21096/
• CommitID: 06b8622
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21097/
• CommitID: 55d725b
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21102/
• CommitID: 50a783e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21105/
• CommitID: 24ecb13
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21106/
• CommitID: 89a71a0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testDropPrimaryDuringReplication

• URL: https://build.ci.opensearch.org/job/gradle-check/21112/
• CommitID: bc94adb
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #8924 (a911bfa) into main (bb78930) will increase coverage by 0.01%.
Report is 3 commits behind head on main.
The diff coverage is 100.00%.

❗ Current head a911bfa differs from pull request most recent head 7dd8259. Consider uploading reports for the commit 7dd8259 to get more accurate results

@@             Coverage Diff              @@
##               main    #8924      +/-   ##
============================================
+ Coverage     71.06%   71.07%   +0.01%     
+ Complexity    57314    57280      -34     
============================================
  Files          4766     4765       -1     
  Lines        270453   270385      -68     
  Branches      39555    39546       -9     
============================================
- Hits         192197   192185      -12     
+ Misses        62045    61979      -66     
- Partials      16211    16221      +10     

Files Changed	Coverage Δ
...opensearch/gradle/docker/DockerSupportService.java	15.57% <100.00%> (+1.40%)	⬆️

... and 477 files with indirect coverage changes

[andrross]

@andrross @dblock @nknize would love to hear your thoughts on the matter. It turned out we don't really run any checks on distributions (and sadly, had a regression sneaked in #8903).

I would like to suggest to add one more assemble check to make sure the distributions are properly built across the set of available operating systems (Linux / MacOS / Windows). That would have helped us to catch the regression early on (the checks are failing exactly because of this problem).

Thank you folks.

@reta I like the intent here for sure! In the name of frugality, are we duplicating the compilation that is being done for these platforms in precommit or check? If so, is that worth worrying about? In theory it seems like we could piggyback assemble on the check action for ubuntu and the precommit action for macos and windows.

[reta]

@reta I like the intent here for sure! In the name of frugality, are we duplicating the compilation that is being done for these platforms in precommit or check?

Thanks a lot @andrross , yes, we do duplicate compilation and javadoc (at least)

In theory it seems like we could piggyback assemble on the check action for ubuntu and the precommit action for macos and windows.

The problem with check is that it runs off the Jenkins, whereas macos & windows precommits are Github actions, the later could be combined with assemble. What we could try is:

• bring back precommit for Linux
• but exclude it from Gradle check (Jenkins), still compilation would be repeated sadly

What do you think?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21282/
• CommitID: b6ab37c
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=search.aggregation/20_terms/string profiler via global ordinals}
      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testBasicTaskResourceTracking

• URL: https://build.ci.opensearch.org/job/gradle-check/21288/
• CommitID: c8ed39e
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testSearchAggregationWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.remotestore.RemoteStoreIT.testRemoteTranslogRestoreWithNoDataPostRefresh

• URL: https://build.ci.opensearch.org/job/gradle-check/21299/
• CommitID: f6643cf
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[andrross]

The problem with check is that it runs off the Jenkins, whereas macos & windows precommits are Github actions, the later could be combined with assemble. What we could try is:

bring back precommit for Linux
but exclude it from Gradle check (Jenkins), still compilation would be repeated sadly

@reta In some brief testing on my Ubuntu machine, doing precommit along with assemble doesn't really add much time versus just doing assemble. I think I'm inclined to do the following:

• bring back precommit for Ubuntu
• change precommit to run ./gradlew precommit assemble (and probably rename the action to "precommit and assemble" or something like that)
• keep the Ubuntu check as it is (probably not worth it to exclude precommit)

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21823/
• CommitID: 9d86c88
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/21834/
• CommitID: a911bfa

[opensearch-trigger-bot]

Compatibility status:

> Task :checkCompatibility
Checking compatibility for: https://github.com/opensearch-project/reporting.git with ref: main
Incompatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git]
Compatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 22m 35s

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/21997/
• CommitID: 7dd8259
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/22001/
• CommitID: 7dd8259
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/22018/
• CommitID: 7dd8259
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/22027/
• CommitID: 7dd8259

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-8924-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 63367a72a48ec7aa7a7c48a27d3adbeef17f6059
# Push it to GitHub
git push --set-upstream origin backport/backport-8924-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-8924-to-2.x.