Update datasource metadata queries

[shreyabiradar07]

Description

This PR has the following changes:

• Updates NAMESPACE_QUERY, WORKLOAD_INFO_QUERY, CONTAINER_INFO_QUERY dsmetadata queries to provide the average namespace, workload and container status over the past 15 days.

• Also includes namespace label in CONTAINER_INFO query to perform the join operation on (pod, namespace) with the pod to handle the case where same workloads are present in different namespaces.

• Handles an edge case - wherein for workload type job with no containers was earlier resulting in NullPointerException, which is now fixed.

Fixes # (1316)

Type of change

• Bug fix
• Enhancement
• New feature
• Docs update
• Breaking change (What changes might users need to make in their application due to this PR?)
• Requires DB changes

How has this been tested?

Please describe the tests that were run to verify your changes and steps to reproduce. Please specify any test configuration required.

• New Test X
• Functional testsuite

Tested by running local monitoring demos on ResourceHub cluster - ./local_monitoring_demo.sh -c openshift -i quay.io/shbirada/update_dsmetadata_queries:v1

Fixed the NullPointerException seen before:

Error while converting DataSourceMetadata Object to KruizeDSMetadataEntry table due to Cannot invoke "java.util.HashMap.values()" because the return value of "com.autotune.common.data.dataSourceMetadata.DataSourceWorkload.getDataSourceContainerHashMap()" is null

java.lang.NullPointerException: Cannot invoke "java.util.HashMap.values()" because the return value of "com.autotune.common.data.dataSourceMetadata.DataSourceWorkload.getDataSourceContainerHashMap()" is null

pod logs after handling the edge case: https://privatebin.corp.redhat.com/?87922ed8fdb9df55#D6FBWxRYbKHTYEsTEcts1D3aNGtmb3jgEe3eXhf5FU2F

pod logs with Exception - https://privatebin.corp.redhat.com/?527498b097f364ac#F93EzwiZG121QPcc73FKk8NU9bCTqzLMgSsbPsHuoFVS

Test Configuration

• Kubernetes clusters tested on: ResourceHub OpenShift cluster

Checklist 🎯

• Followed coding guidelines
• Comments added
• Dependent changes merged
• Documentation updated
• Tests added or updated

Additional information

docker image - quay.io/shbirada/update_dsmetadata_queries:v1

[kusumachalasani]

The query changes looks good.

As discussed, please update the conditions on when the containers can be null for a workload.

[kusumachalasani]

As we have CONTAINER_INFO_QUERY which has both container and workload details I feel it is redundant to run WORKLOAD_INFO_QUERY and also we can avoid running an extra for loop for workload -

autotune/src/main/java/com/autotune/database/helper/DBHelpers.java

Line 1019 in 4c3195d

for (DataSourceWorkload dataSourceWorkload : dataSourceNamespace.getDataSourceWorkloadHashMap().values()) {

This can help in reducing the time taken to get the datasource details and also avoids redundancy.

[kusumachalasani]

The query changes LGTM!

[dinogun]

@shreyabiradar07 Please resolve the conflicts

[dinogun]

@shreyabiradar07 Can you please rebase on top of mvp_demo and test

[shreyabiradar07]

Local monitoring demo response : https://privatebin.corp.redhat.com/?6fb5876b4ae8e63f#AoXBYDV9BTfX15vYDexamQYrnbHpWiMBkuq7BAZzaAzX

Encountering the following assertion error for rest_apis/test_list_metadata.py local monitoring tests:

assert [<ValidationError: "'containers' is a required property">, <ValidationError: "'containers' is a required property">, <ValidationError: ''>] == ''

Failed test logs : https://privatebin.corp.redhat.com/?cd5f4646e2d1f0e3#41XkaVvktuBJXmQDm4F11WTvfPXq4Rh7DRN164LyGUhQ

[chandrams]

Why is containers removed?

[shreyabiradar07]

Below assertion error is encountered in edge cases where there are no containers for a job workload type as "containers" is a required field:

assert [<ValidationError: "'containers' is a required property">, <ValidationError: "'containers' is a required property">, <ValidationError: ''>] == ''

Following are the few examples for edge cases observed after updating the datasource metadata queries to gather metric data for last 15 days:

"collect-profiles-28807155": {
                  "workload_name": "collect-profiles-28807155",
                  "workload_type": "job"
                },
"image-pruner-28802880": {
                  "workload_name": "image-pruner-28802880",
                  "workload_type": "job"
                }

Failed test logs : https://privatebin.corp.redhat.com/?cd5f4646e2d1f0e3#41XkaVvktuBJXmQDm4F11WTvfPXq4Rh7DRN164LyGUhQ

[shreyabiradar07]

As we have CONTAINER_INFO_QUERY which has both container and workload details I feel it is redundant to run WORKLOAD_INFO_QUERY and also we can avoid running an extra for loop for workload -

autotune/src/main/java/com/autotune/database/helper/DBHelpers.java

Line 1019 in 4c3195d

for (DataSourceWorkload dataSourceWorkload : dataSourceNamespace.getDataSourceWorkloadHashMap().values()) {

This can help in reducing the time taken to get the datasource details and also avoids redundancy.

As suggested created issue #1345 to work on the enhancement of performance and queries, will work on the same in the next release

[shreyabiradar07]

The query changes looks good.

As discussed, please update the conditions on when the containers can be null for a workload.

Summarizing the steps followed to capture conditions when the containers can be null for a workload.

Tried importing metadata for 3 scenarios:

• before running a job
• when the job is in running state
• after completion of job

but couldn't capture the case - where containers are null

Next steps:

• Delete the existing job.
• After 15 mins, run the workload query and check if the workload related to job exists.
• Run the container query.

after 15m & 30m job data for both workload and container queries was present

container=null was observed specifically for namespace openshift-operator-lifecycle-manager , workload type job for example: 'collect-profiles-28795995

@kusumachalasani pointed out that collect-profiles jobs are very short lived (its ~ 4sec duration). So, the container metrics related to it doesn't have enough time to get it exported to prometheus because of which container details are null.

[chandrams]

LGTM