Add dynamic setup of gpu_limit in gke-job-template module

[mohitchaurasia91]

Triggered PR test for all blueprints with usage of gke-node-pool and gke-job-template module completed with OK status.

Verification

• Blueprint: gke-a3-megagpu
  • Test job logs from daily test build running on develop branch, nvidia-smi is showing stat from 1 gpu (default value set in gke-job-template module)
  • Test job logs from PR build running on feature branch, nvidia-smi is showing stat from 8 gpu (using gke-node-pool module output to set this up dynamically), user can still override it using var requested_gpu_per_pod

Submission Checklist

NOTE: Community submissions can take up to 2 weeks to be reviewed.

Please take the following actions before submitting this pull request.

• Fork your PR branch from the Toolkit "develop" branch (not main)
• Test all changes with pre-commit in a local branch #
• Confirm that "make tests" passes all tests
• Add or modify unit tests to cover code changes
• Ensure that unit test coverage remains above 80%
• Update all applicable documentation
• Follow Cluster Toolkit Contribution guidelines #

[annuay-google]

Please add a section/comment on verifying the change- is there a way to see how many GPUs nvidia-smi has requested for e.g.?

[mohitchaurasia91]

Please add a section/comment on verifying the change- is there a way to see how many GPUs nvidia-smi has requested for e.g.?

Updated PR description with logs from test jobs running on develop and feature branch.

[annuay-google]

LGTM