Refactor environment variables used by test cases #114
Conversation
…efactor-test-workflow
| # copyright notice, and modified files need to carry a notice indicating | ||
| # that they have been altered from the originals. | ||
|
|
||
| name: CI |
There was a problem hiding this comment.
This workflow definitions replaces pr.yml and push.yml that were mostly the same and included duplicated job definitions.
Notable Changes
- no concurrent workflow runs per branch (previous workflows are cancelled automatically)
- previous behavior implemented via
ifconditions - rephrased
push tests->integration testsandpr tests->unit tests - added build dependencies to make sure resource expensive integration tests are only run when everything else passes
Let's discuss objections related to the updated CI approach in this thread.
| @@ -0,0 +1,55 @@ | |||
| # This code is part of Qiskit. | |||
There was a problem hiding this comment.
This workflow definitions replaces cron-prod.yml and cron-staging.yml that were mostly the same and included duplicated job definitions for different environments.
Notable changes
- avoid duplicated job definitions by using Github environments (see e.g.
legacy-production) - reduced the matrix options so that the integration tests are only run using the latest supported Python and ubuntu version
- version/os specific issues are expected to be uncovered by unit tests
- helps with stability and decreases test runtime because we're not "performance testing" the APIs with parallel test jobs
- left out the migration of
cron-slow.ymlandcron-slow-terra-main.ymlfor now- I believe the primary goal should be to stabilize the main path (integration tests) and then reenable slow tests and discuss if we need to run the same test with the latest versions of qiskit-terra (given we're no longer part of the meta package)
Let's discuss objections related to scheduled test cases in this thread.
| * QISKIT_IBM_STAGING_DEVICE: staging device to use. | ||
| * QISKIT_IBM_STAGING_PRIVATE_HGP: staging hub/group/project to use for private jobs. | ||
| """ | ||
| """Decorators used by unit tests.""" |
There was a problem hiding this comment.
In order to introduce support for running the integration tests against different APIs (legacy, Cloud) separately I consolidated the decorators we previously had.
Notable changes
- best described in code, see below
Let's discuss objections related to decorator changes in this thread.
|
I've set the environment secrets in the main repo. |
|
Thanks @rathishcholarajan for setting this up. I believe we still need to accept the risk to integrate it in order to see if the pieces work together. Let me know if anything is missing from my side. |
# Conflicts: # .github/workflows/main.yml # .github/workflows/pr.yml
# Conflicts: # test/utils/decorators.py
…efactor-test-workflow
There was a problem hiding this comment.
LGTM! This PR removes ability to run slow tests. We should handle this using this ticket #127
I see that a lot of tutorial tests are slow tests https://github.com/daka1510/qiskit-ibm-runtime/runs/4969466053?check_suite_focus=true#step:5:2256 and they are skipped. I am hesitant to merge this until we can figure out a way to run these slow tests in this PR or in a separate PR (and I want to merge both PRs together). (Initially I thought it was just one slow test, but now I see a lot after merging the tutorials PR)
|
@rathishcholarajan - good catch that the tutorial tests are also "slow tests". I just checked the logs of the most recent slow-test and found those tests are failing anyway with different errors that do not appear like "intermittent instabilities", e.g. I am happy to prioritize #127 right after this PR is integrated and added a second acceptance criteria that covers fixing those tutorial test cases as well. I don't see much value of reenabling slow tests in this PR as they will definitely continue to fail. Let me know if you feel strongly about it and I will add a workflow definition file that runs the failing slow tests as before (just separately against cloud/legacy). |
* fix counts in tests * fix qaoa test * update vqe * fix cloud programs
Summary
Primary goal
Enable integration test cases to run separately against IBM Quantum ("legacy") and IBM Cloud ("cloud") endpoints. Makes use of Github environment secrets.
Secondary goal
Reduce repetition across workflow definition files, reduce number of parallel integration test runs to help getting around test stability issues.
Details and comments
Fixes #30
Fixes Qiskit/qiskit-ibm-provider#204
Helpful for #109
Helpful for #93
Example Workflow
https://github.com/daka1510/qiskit-ibm-runtime/actions/runs/1728894685
Prerequisites