Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes to catch changes to versioned rules as well as rules going NOT-APPLICABLE #29

Merged
merged 4 commits into from
Jun 29, 2022
Merged

Fixes to catch changes to versioned rules as well as rules going NOT-APPLICABLE #29

merged 4 commits into from
Jun 29, 2022

Conversation

jhrozek
Copy link
Collaborator

@jhrozek jhrozek commented Jun 28, 2022

Please see the commit messages for more details.

  • deps: Update CO so that it supports showNotApplicable: just bumps the CO version. Unfortunately, I was unable to bump to the latest version, because then we get conflicts with the cluster-authentication-operator deps. But this version bump at least unblocks us.
  • helpers: Enable showNotApplicable checks: Adjusts the scanSetting so that we also at least consider check results that are N/A
  • helpers: Fail if there is no test result YAML file for current version, but exists for other versions: Helps catch results where we used to have per-version manifests, but not for the current version.
  • helpers: Fix looking for versioned rules - what it says on the box

@jhrozek
deps: Update CO so that it supports showNotApplicable
d749588 
We used to vendor an ancient version of CO where the APIs didn't even
support setting the showNotApplicable option.

Jira: CMP-1417
@jhrozek
helpers: Enable showNotApplicable checks
4898531 
Several versions ago, CO started defaulting to not creating check
results for rules that are not applicable. While that's good for the end
user, doing so in the e2e test environment means that we completely miss
checks that have become not applicable for a platform for one reason or
another. We should always evaluate all test results against the expected
results.

Jira: CMP-1417
@jhrozek
helpers: Fail if there is no test result YAML file for current versio…
2666d09 
…n, but exists for other versions

Right now, the CaC e2e tests do support versioned test results, but if a
test result does not exist for a specific version, the tests simply fall
back to the version-less test result and if that doesn't exist either,
the result is silently ignored.

This means that once we switch to a new version of OCP but forgot to add
a versioned test result for that version, the check is silently ignored.

We need to have a mechanism that would alert us when we switch to a new
version but don't add a versioned result for that version.

Jira: CMP-1415
@jhrozek
helpers: Fix looking for versioned rules
053e80a 
We used to look for versioned rules under ruleTestDir/manifest, so
usually ocp4/e2e/4.10.yml or such. But that was incorrect, the versioned
manifests live under the rule dir, not the directory where tests are
executed from.

Jira: CMP-1415
@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 28, 2022

btw it is expected that the tests will fail at the moment until we merge ComplianceAsCode/content#8996

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 28, 2022

btw it is expected that the tests will fail at the moment until we merge ComplianceAsCode/content#8996

but I did verify that the tests pass using a content that includes that fix. At least we'll be able to verify that fix nicely going from red to green.

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 29, 2022

Oops, I was wrong, there is a bug:

 # github.com/ComplianceAsCode/ocp4e2e [github.com/ComplianceAsCode/ocp4e2e.test]
./helpers.go:913:17: undefined: os.ReadDir
note: module requires Go 1.16

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 29, 2022

btw the known issue note looks good!

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 29, 2022

ah, that's because:

�[36mINFO�[0m[2022-06-28T13:15:08Z] Tagging openshift/release:golang-1.15 into pipeline:root.

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 29, 2022

ah, that's because:

�[36mINFO�[0m[2022-06-28T13:15:08Z] Tagging openshift/release:golang-1.15 into pipeline:root.

Sorry, this was meant for a different PR.

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 29, 2022

No, sorry, I need more coffee and/or more sleep. The known issue comment was meant for a different PR. The build issue should be fixed with openshift/release#29943

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 29, 2022

openshift/release#29943 was merged, let's try again if we get the "expected" failures.

@jhrozek
Copy link
Collaborator Author

jhrozek commented Jun 29, 2022

/retest

@rhmdnd
Copy link
Collaborator

rhmdnd commented Jun 29, 2022

/lgtm

@JAORMX JAORMX merged commit d9593c5 into ComplianceAsCode:main Jun 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jhrozek @rhmdnd @JAORMX