Update FIPS140-3 exclusions

[JinhangZhang]

Exclude these tests to make it easier for the team to triage test cases and problems that occur due to changes going forward.

Ignore all tests affected by FIPS 140-3 testing. A quick review of all the failing tests was done by various people on the development team to get a quick sense to the nature of the failure. At this time we aren't documenting and opening issues for each individual failure as this will take a long time and is currently holding up regression testing of various releases.

Future work is planned to address necessary modifications to various tests on a package by package basis.

[JinhangZhang]

@llxia @jasonkatonica As discussed, re-enable update where test ignores were removed in #18 & #19.

[JinhangZhang]

@pshipton @JasonFengJ9 please help to review

[keithc-ca]

This pull request seems misnamed. This appears to be the only test that is (re-enabled); every other change excludes extra tests or platforms.

[JinhangZhang]

Updated the name

[keithc-ca]

The commit message could still be better. This doesn't just "add" exclusions it also re-enables ConfigFileTest. Perhaps it should say something like "Update FIPS140-3 exclusions" (at a minimum). I would like to see more information, like references to issues that relate to the problems that will need to be addressed (the blank issues (#20, ibmruntimes/openj9-openjdk-jdk22#68) are not sufficient).

Another issue (that I've pointed out previously) is that things should be organized (e.g. sorted) to help people find what they're looking for. To that end, tests should be grouped and sorted by name within each group; platform names should be sorted by name.

[JinhangZhang]

• Update the name.
• It takes time to investigate and category those failures, but we do have a plan to check and address the failures one-by-one. Like TLS failures under javax/net/ssl folder. @jasonkatonica FYI.
• Sorted tests and platform names by name.

[keithc-ca]

The point of making reference to an issue in the problem list is to provide some background as to why the test was excluded. Please provide at least some of that background.

[jasonkatonica]

At this point in time we are only attempting to ignore all tests effected by FIPS 140-3 testing. A quick review of all the failing tests was done by various people on the development team to get a quick sense to the nature of the failure. At this time we don't plan on documenting and opening issues for each individual failure as this will take a long time and is currently holding up regression testing of various releases. This has been a similar strategy on other releases and platforms for some time until we are able to address necessary modifications to various tests on a package by package basis.

[keithc-ca]

I wasn't asking for separate issues for each failure, but neither of the issues referenced above have any details at all. Surely we can do better than that. I'd be fine with this if those issues described even broad categories of reasons for excluding tests (e.g. algorithm not allowed in FIPS-140-3). I don't know what the exclusion reasons are and nobody else could figure it out based on the links provided.

[jasonkatonica]

Hi Keith we don't plan on scheduling this work at this time. Based upon previous experiences categorizing FIPS 140-2 tests a few years ago took this team a few months to do a good job with. At this point in time we only plan on ignoring these failures without a deep dive into root cause, categorizations, and proposing long term fixes to either the tests or the FIPS support itself. The main goal at this point is to :

1. Make it easier for the team to triage test cases and problems that occur due to changes going forward. We can start with this while additional work is being done to fix tests and find root causes of each test failure.
2. We need to make additional updates on top of these updates for windows such that we can run the FIPS tests for 140-3 on that platform which is currently not being done.

I agree that we certainly can do better, and we plan to do so iteratively over time with various components. Work is underway for such a deep dive already on the TLS tests.

[pshipton]

I added a summary to the PR description.

[pshipton]

Added the same to #20

[JasonFengJ9]

@jasonkatonica please review.

[jasonkatonica]

This looks good to me. Ignoring further tests on Java 23 is expected.

[pshipton]

@keithc-ca are you good with this now?

[keithc-ca]

The commit message could still be better. This doesn't just "add" exclusions it also re-enables ConfigFileTest. Perhaps it should say something like "Update FIPS140-3 exclusions" (at a minimum). I would like to see more information, like references to issues that relate to the problems that will need to be addressed (the blank issues (#20, ibmruntimes/openj9-openjdk-jdk22#68) are not sufficient).

Another issue (that I've pointed out previously) is that things should be organized (e.g. sorted) to help people find what they're looking for. To that end, tests should be grouped and sorted by name within each group; platform names should be sorted by name.

[JinhangZhang]

• Update the name.
• It takes time to investigate and category those failures, but we do have a plan to check and address the failures one-by-one. Like TLS failures under javax/net/ssl folder. @jasonkatonica FYI.
• Sorted tests and platform names by name.

[llxia]

We're pending the JDK23 M1 FIPS testing on this. Could we create an issue to track the enhancements and address them in separate PRs later?

[keithc-ca]

We already have (at least two) issues. The problem is that they don't provide any information about why these tests are excluded. Creating another issue to see that the existing issues are filled in doesn't seem helpful.

In the interest of moving things along, I'm prepared to merge this (after the requested changes are made) with the understanding that future pull requests will have documented motivations. Please signal your agreement: @JinhangZhang and @jasonkatonica.

[pshipton]

I added a summary of the motivation to this PR description and #20

[keithc-ca]

I added a summary of the motivation to this PR description and #20

Thank you, but that is something that should be expected of the contributor.