Test Center - Unstable Test Results In Different Test Runs

[sunp1515]

Description

We are observing unstable test result for the test suite where we are observing that sometimes the test case pass and sometimes it doesn't without making any changes to the environment it running against. This issue prevents us from using Test Center Pipeline before and after any delivery configuration activation as a Test Stage to make sure test cases pass and everything is working as expected. We are using test-center CLI in Azure DevOps.

Version

test-center version 0.1.1

Command Used:

akamai -e $(pwd)/.edgerc test-center test --test-suite-name '$(testSuiteName)' --env=${TARGETENVIRONMENT} --force-color

Issue Details

Example 1:

• In this case, the User-Agent header is present in the headers but the case failed. The same test case passes in a different run. Please find the below screenshot for the result comparison:

Example 2:

• In this case, the Host header is present in the headers but the case failed. The same test case passes in a different run. Please find the below screenshot for the result comparison:

Example 3:

• This is an interesting case, the test case request is completely different than what we have defined in the test request. It is requesting some Google sites instead. The same test case passes in a different run. Please find the below screenshot for the result comparison:

These are random occurrences where any test cases fail and pass in subsequent runs so it is very unstable and does not give us confidence in using this as a Test Stage before and after property configuration deployments.

Thank you.

[rakayada]

Hi @sunp1515
 
We have analyzed the issue reported it looks like it is working as expected as per the logs and condition evaluation. Condition evaluation is done on response headers and not on request headers, Certain headers might be missing in response headers due to rules defined in property manager or security rules. We suggest rerunning the tests and letting us know if you are still facing issues. Please attach the required steps/URLs in detail to replicate the issue from our end.

We have released the GA version for cli-test-center 1.0.0 and the current release has multiple new features/bug fixes/library updates/code improvements.

Users need to upgrade to the latest test-center version 1.0.0 for cli-test-center commands to work properly. Please refer README.md file for more information on breaking changes.

Update to the latest cli-test-center version using this command - akamai update test-center

This new release might have fixed this issue as there are library updates and code improvements. Please check the same after upgrading the cli-test-center version.

Thanks!