GPU MIG Right sizing recommendations by kruize

[bharathappali]

Describe the feature

Kruize reads CPU & memory usage data from the provided data source and comes up with the CPU and Memory right sizing recommendation. In a similar way it would be good to have the GPU MIG partition sizing recommendation for container which utilise GPU's

Examples or references

Most of the ML workloads need GPU power and advanced GPU's from NVIDIA support MIG (Multi instance GPU's) where a single Physical GPU can be partitioned into multi instances of virtual or logical GPU's which can be configured and shared across multiple containers. Ampere (from A30) and Hopper series GPU's provide this feature.

Suggest a solution

• Record the GPU related metrics
• Process the metrics along with CPU and Memory metrics
• Provide MIG partition recommendation

Additional Context

None

[bharathappali]

This new feature can be implemented in the following steps:

• Changes for Metrics profile yaml to add GPU related metrics - Adds GPU queries to performance profile #1313
• Add Interfaces needed for GPU recommendation - Add Accelerator specific interfaces and there respective implementations #1314
• Update the existing data structures (Container Data and Interval Results) to have GPU usage data - Add accelerator related fields in existing structures #1317
• Update RecommendationItem enum to accommodate MIG partitions - Adds changes to RecommendationItem enum to accomodate MIG strings #1318
• Dynamic GPU detection & Metrics collection for container which use GPU - Add changes to fetch Accelerator metrics at the time of recommendation generation #1319
• Recommendation Generation (Updates to Recommendation Engine & Models) - Adds Accelerator recommendation generation logic  #1320
• Add required adapters to solve the serialisation and deserialisation problems with new fields and structures - Add Adapter for RecommendationItem to serialize and deserialize the enum #1321
• Test Coverage - Add tests for validating recommendation json for accelerator values #1336

[dinogun]

@bharathappali Can this be closed now?

[bharathappali]

Yes @dinogun as all PR's are merged

[dinogun]

Please update the test PR details and close this

[dinogun]

@bharathappali I asked for the test PR details to be added in the description

[bharathappali]

Sorry for over look, Will be adding it now @dinogun

[bharathappali]

Reopened to updated the description with Test PR

[bharathappali]

Closing this issue as all the PR's are merged.