[Decouple][CI] BWC use snapshots + improvements

[kavilla]

Is your feature request related to a problem? Please describe.

The BWC tests on 2.x are failing: https://github.com/opensearch-project/OpenSearch-Dashboards/actions/runs/5895740470/job/16189125311.

BWC test files: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/.github/workflows/build_and_test_workflow.yml#L349 utilize the release artifact which is built by a configuration sent externally. If there are issues in the build release for OpenSearch then we will be impacted, where snapshot is built more frequently.

Describe the solution you'd like

Immediate Actions:

• Update to use the snapshot URL similar to the cypress test workflow: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/.github/workflows/cypress_workflow.yml#L13

Follow-up Proposals:

• wait-for defaults to 60 seconds but gracefully error if OpenSearch can be hit but is failing.
• Default utilize the release artifact and fail safe to the snapshot if the release artifact for OpenSearch fails to run properly.
• fail-fast: false, it's important information to know which version is not compatible but currently it will cancel all the versions if one version fails
• Update release process to verify passing BWC test matrix
  • Did we skip failures and did we verify they are fine now

Questions:
Seems to be something broken in 2.10 release build of OpenSearch which will need monitoring if eventually it is not fix as it might mean breaking change in 2.x