Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: 7.2.0 Analysis Snapshot Protocols #14569

Merged
merged 14 commits into from
Mar 7, 2024

Conversation

DerekMaggio
Copy link
Contributor

@DerekMaggio DerekMaggio commented Feb 29, 2024

Overview

Add protocols pulled from v7.2.0 pull requests for Analysis Snapshot testing. All work filed under RQA-2434

Test Plan

  • Add partial tip pickup protocols from Sanitti
  • Add ABR protocol from RABR-23
  • Run new protocols locally and verify results
  • Run in workflow dispatch action and verify success [link]

Changelog

Here are the protocols added and where I found them:

Review requests

There are a few pull requests that I had questions about. I will @ the people that worked on them to answer the questions

Risk assessment

None

@DerekMaggio DerekMaggio changed the title test: add test cases around dispense logic over different api versions test: 7.2.0 Analysis Snapshot Protocols Feb 29, 2024
Copy link

codecov bot commented Feb 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 67.50%. Comparing base (b533692) to head (5f0959b).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             edge   #14569   +/-   ##
=======================================
  Coverage   67.50%   67.50%           
=======================================
  Files        2514     2514           
  Lines       72375    72375           
  Branches     9317     9317           
=======================================
  Hits        48856    48856           
  Misses      21314    21314           
  Partials     2205     2205           
Flag Coverage Δ
app 63.99% <ø> (ø)
components 48.40% <ø> (ø)
labware-library 41.11% <ø> (ø)
protocol-designer 37.90% <ø> (ø)
step-generation 86.88% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

@DerekMaggio DerekMaggio force-pushed the 7.2.0-analysis-snapshot-protocols branch from cc89cc8 to 56c0390 Compare March 4, 2024 16:47
@DerekMaggio DerekMaggio requested a review from y3rsh March 4, 2024 16:47
@DerekMaggio DerekMaggio marked this pull request as ready for review March 4, 2024 17:21
@DerekMaggio DerekMaggio requested review from a team as code owners March 4, 2024 17:21
@sfoster1
Copy link
Member

sfoster1 commented Mar 4, 2024

I'd expect you can use RABR-23 to test #14437 yeah

@SyntaxColoring
Copy link
Contributor

Analysis can't capture it. It's only relevant when robot-server retrieves details about runs and analyses stored from prior boots.

Unfortunately no. The hole that that PR plugged is only for exceptions raised by certain Opentrons internal code.

Yep, I agree. That fix is for robot runtime behavior, which we can't cover with these analysis tests. Theoretically, we could do a similar snapshot thing for robot runtime behavior. I guess that would be g-code-testing.

Copy link
Contributor

@SyntaxColoring SyntaxColoring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some possible apiLevel mistakes, but other than that, this looks great to me. Thank you!

Copy link
Collaborator

@y3rsh y3rsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One step to add in the README otherwise great.

Comment on lines 98 to 101
# Not running this file because the error handling to catch that
# the flex is only supported on API versions 2.15 and above, does not exist
# Flex_None_None_TC_2_14_verifyThermocyclerLoadedSlots
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding back into the test with this task https://opentrons.atlassian.net/browse/RDEVOPS-71

@@ -0,0 +1,14 @@
# Pulled from: https://github.com/Opentrons/opentrons/pull/14491
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ these comments.


#### Running Analysis Snapshot Tests Locally

To run analysis snapshot tests locally, you run
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add the step to build the docker container since it must be built before running the test

TARGET="<target-branch-or-tag>" make build-opentrons-analysis

@DerekMaggio DerekMaggio added the DO NOT MERGE Indicates a PR should not be merged, even if there's a shiny green merge button available label Mar 5, 2024
@y3rsh y3rsh changed the base branch from chore_release-7.2.0 to edge March 6, 2024 19:13
@y3rsh y3rsh removed the DO NOT MERGE Indicates a PR should not be merged, even if there's a shiny green merge button available label Mar 7, 2024
@DerekMaggio DerekMaggio force-pushed the 7.2.0-analysis-snapshot-protocols branch from 257a8f7 to 5f0959b Compare March 7, 2024 19:59
@DerekMaggio DerekMaggio merged commit 0b40719 into edge Mar 7, 2024
32 checks passed
Carlos-fernandez pushed a commit that referenced this pull request May 20, 2024
Add protocols pulled from v7.2.0 pull requests for Analysis Snapshot
testing. All work filed under
[RQA-2434](https://opentrons.atlassian.net/browse/RQA-2434)

- [x] Add partial tip pickup protocols from Sanitti
- [x] Add ABR protocol from
[RABR-23](https://opentrons.atlassian.net/browse/RABR-23)
- [x] Run new protocols locally and verify results
- [x] Run in workflow dispatch action and verify success
[[link]](https://github.com/Opentrons/opentrons/actions/runs/8160792870/job/22308167177)

Here are the protocols added and where I found them:

- [RQA-2098](https://opentrons.atlassian.net/browse/RQA-2098)
-
Flex_P1000_96_Gripper_TC_TM_HS_AnalysisError_GripperCollisionWithTips.json
- #14491
  - Flex_None_None_TC_2_14_verifyThermocyclerLoadedSlots.py
  - Flex_None_None_TC_2_15_verifyThermocyclerLoadedSlots.py
  - Flex_None_None_TC_2_16_verifyThermocyclerLoadedSlots.py
  - Flex_None_None_TC_2_17_verifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_14_VerifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_15_VerifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_16_VerifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_17_VerifyThermocyclerLoadedSlots.py
- #14475
-
Flex_None_None_TC_2_16_AnalysisError_TrashBinAndThermocyclerConflict.py
  - OT2_None_None_2_16_verifyDoesNotDeadlock.py
-
OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin1.py
-
OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin2.py
- #14547
-
Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLid.py
- #14522
-
Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLidClips.py
- Dispense functionality validation
  - OT2_P300M_P20S_TC_HS_TM_2_15_dispense_changes.py
  - OT2_P300M_P20S_TC_HS_TM_2_16_dispense_changes.py
  - OT2_P300M_P20S_TC_HS_TM_2_17_dispense_changes.py
- #14253
  - OT2_P300S_None_2_16_verifyNoFloatingPointErrorInPipetting.py

There are a few pull requests that I had questions about. I will @ the
people that worked on them to answer the questions

- #14437
- @sfoster1, Can I use the protocol found in
[RABR-23](https://opentrons.atlassian.net/browse/RABR-23) to get this to
happen? Is there another way?
- #14509
- @SyntaxColoring, can analysis capture this change or is this only
relavant during actual protocol runtime?
- #14510
- @TamarZanzouri or @SyntaxColoring, can I raise a generic python
exception inside the protocol to trigger this functionality?
- #14512
- @Laura-Danielle or @SyntaxColoring, Iant to validate my logic. This
not a good canidate for snapshot testing because you have to run LPC and
Calibration which is not taken into account during analysis. Can you
confirm?

None

[RQA-2434]:
https://opentrons.atlassian.net/browse/RQA-2434?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[RQA-2098]:
https://opentrons.atlassian.net/browse/RQA-2098?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[RABR-23]:
https://opentrons.atlassian.net/browse/RABR-23?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
Carlos-fernandez pushed a commit that referenced this pull request Jun 3, 2024
Add protocols pulled from v7.2.0 pull requests for Analysis Snapshot
testing. All work filed under
[RQA-2434](https://opentrons.atlassian.net/browse/RQA-2434)

- [x] Add partial tip pickup protocols from Sanitti
- [x] Add ABR protocol from
[RABR-23](https://opentrons.atlassian.net/browse/RABR-23)
- [x] Run new protocols locally and verify results
- [x] Run in workflow dispatch action and verify success
[[link]](https://github.com/Opentrons/opentrons/actions/runs/8160792870/job/22308167177)

Here are the protocols added and where I found them:

- [RQA-2098](https://opentrons.atlassian.net/browse/RQA-2098)
-
Flex_P1000_96_Gripper_TC_TM_HS_AnalysisError_GripperCollisionWithTips.json
- #14491
  - Flex_None_None_TC_2_14_verifyThermocyclerLoadedSlots.py
  - Flex_None_None_TC_2_15_verifyThermocyclerLoadedSlots.py
  - Flex_None_None_TC_2_16_verifyThermocyclerLoadedSlots.py
  - Flex_None_None_TC_2_17_verifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_14_VerifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_15_VerifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_16_VerifyThermocyclerLoadedSlots.py
  - OT2_None_None_TC_2_17_VerifyThermocyclerLoadedSlots.py
- #14475
-
Flex_None_None_TC_2_16_AnalysisError_TrashBinAndThermocyclerConflict.py
  - OT2_None_None_2_16_verifyDoesNotDeadlock.py
-
OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin1.py
-
OT2_None_None_HS_2_16_AnalysisError_HeaterShakerConflictWithTrashBin2.py
- #14547
-
Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLid.py
- #14522
-
Flex_P1000_96_None_TC_2_16_AnalysisError_pipetteCollisionWithThermocyclerLidClips.py
- Dispense functionality validation
  - OT2_P300M_P20S_TC_HS_TM_2_15_dispense_changes.py
  - OT2_P300M_P20S_TC_HS_TM_2_16_dispense_changes.py
  - OT2_P300M_P20S_TC_HS_TM_2_17_dispense_changes.py
- #14253
  - OT2_P300S_None_2_16_verifyNoFloatingPointErrorInPipetting.py

There are a few pull requests that I had questions about. I will @ the
people that worked on them to answer the questions

- #14437
- @sfoster1, Can I use the protocol found in
[RABR-23](https://opentrons.atlassian.net/browse/RABR-23) to get this to
happen? Is there another way?
- #14509
- @SyntaxColoring, can analysis capture this change or is this only
relavant during actual protocol runtime?
- #14510
- @TamarZanzouri or @SyntaxColoring, can I raise a generic python
exception inside the protocol to trigger this functionality?
- #14512
- @Laura-Danielle or @SyntaxColoring, Iant to validate my logic. This
not a good canidate for snapshot testing because you have to run LPC and
Calibration which is not taken into account during analysis. Can you
confirm?

None

[RQA-2434]:
https://opentrons.atlassian.net/browse/RQA-2434?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[RQA-2098]:
https://opentrons.atlassian.net/browse/RQA-2098?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

[RABR-23]:
https://opentrons.atlassian.net/browse/RABR-23?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants