Ideas for new tests and test process

[kosstennbl]

Some ideas for new tests, they are quite raw and are in need of further explanations and checks.

Tests:

Security:

• Good encryption used in secrets
  • For security reasons, Kubernetes secrets should be encrypted using common cryptographic algorithms.
  • Testing would most likely be limited to configuration checks.
  • Applicable for all CNFs that have any secrets defined.
• Security scans on images (identify vulnerabilities)
  • For security reasons, public container images should be scanned for security vulnerabilities, and if some known issues are found – updated to the latest versions.
  • The test would likely require some external tool for image scanning (i.e. “Clair”?)
  • Seems applicable for all CNFs, unless there are reasons to ignore CVEs.
• Intrusion tests (on open ports etc.)
  • CNF could have some unnecessarily exposed ports, which could be used by attackers for getting access to the application or DOS attacks.
  • Unclear how to automate tests like this. Maybe port scanning would be one way for preventing vulnerabilities like these.
  • Seems applicable for all CNFs, but possible implementation and test subject is very unclear.
• Service account permissions
  • As a common security measure, Service accounts should follow the principle of the least privilege.
  • Could be tested via configuration checks.
  • Applicable for all CNFs, that use Service accounts.
• Secure registry used in CNFs
  • For security reasons, images should be downloaded from the secure registries.
  • The test could check which registries are used for images and then check if they are secure.
  • Applicable for all CNFs.

Other test categories:

• PodAntiAffinityPreset -> hard, (or pods on different workers):
  • Setting rules for specific pod labels to run on different nodes is useful for high availability and failure tolerance.
  • Could be checked in configuration or dynamically by cluster manipulations.
  • Specific for some CNFs, not applicable for all of them.
• Failure threshold settings (Pod disruption budgets)
  • Deployments and Stateful sets should have configured thresholds for number of pods down at the same time. This helps with keeping availability of service during the upgrades / node maintenance, other actions that require pod shutdowns.
  • Could be tested by checking existence of PodDisruptionBudget policies in the chart, dynamically – by trying to perform operation on deployment and following live pod count.
  • Applicable everywhere, where High Availability matters
• Caching used (redis, memcache etc)
  • Where applicable, caching can significantly speed up applications and should be used.
  • Unclear how to test, maybe existence of pods with common caching images.
  • Quite specific, surely not applicable to every CNF.
• Image layers reusing/sharing, reasonable layer size/numbers
  • Sharing image layers could save significant amounts of storage space if some of them are the same in different images. Also, check for reasonable number of layers and their size could be performed in addition to existing reasonable image size check.
  • External tools would be needed (i.e “dive”).
  • Applicable for all CNFs.
• Check that deprecated settings are used
  • Some Kubernetes resources and parameters are considered deprecated and shouldn’t be used. I.e. - persistent volume reclaim policy ‘recycle’.
  • Check chart against the list of deprecated resources and parameters.
  • Applicable for all CNFs.

Other ideas:

• Analyze tools that are used and check whether they provide checks that cnf-testsuite does not use
  • Further test ideas could be taken from already used tools such as Kyverno, Litmus.
  • Adopting some new scenario from an already adapted tool should be easy.
• Health-checks before real tests are executed (platform, apps)
  • Not a test but rather an improvement idea for the testing process.
  • Before running Testsuite tests, it would be nice to check if platform and CNF are healthy: Configs are set up correctly and loaded, cluster is accessible, CNF pods are functional, etc.
  • Could be done together in one task or separated to multiple, being used for different operations.

[lixuna]

Thank you, @kosstennbl

This discussion thread can be used to discuss and prioritize new tests.

I'd suggest adding a new GitHub issue for each prioritized test to implement. Using one issue per test will help with visibility and tracking.