Add Tolerations and Affinity support

[vlameiras]

Hi there 👋

We were testing the project internally and noticed including Tolerations and Affinity support would be nice.

Hopefully this PR will do the trick.

Thanks for the great project!

[samos123]

We try to keep our default resource profile cloud agnostic. So I would prefer to remove the affinity rule

[nstogner]

Agreed - we can keep the tolerations though

[samos123]

Ah yeah true. Let's keep the toleration.

[nstogner]

I think we can accomplish a resource profile that works across most popular cluster types by leveraging how affinity nodeSelectorTerms are OR'd together instead of AND'd. Will open a new issue to track this. We should still remove the affinity & node selector in here for now.

[samos123]

Thanks for the PR. We do need this since adding taints for GPU nodes is common.

I did an initial review but let's wait for @nstogner to review too.

[samos123]

The quickstart e2e test CI is failing because kubeAI scaled down to 0:

[events -w] 0s          Normal   Killing                   pod/kubeai-56485ff664-mqpjg                Stopping container kubeai
[events -w] 0s          Normal   SuccessfulDelete          replicaset/kubeai-56485ff664               Deleted pod: kubeai-56485ff664-mqpjg
[events -w] 0s          Normal   ScalingReplicaSet         deployment/kubeai                          Scaled down replica set kubeai-56485ff664 to 0 from 1

[nstogner]

@vlameiras Thanks for the PR! We were missing unit tests for the modified function so I added a PR into your branch to add those in as well as update the integration tests: vlameiras#1

[vlameiras]

Thanks for the quick replies and for the detailed comments!

[samos123]

Please ignore my previous comment about e2e quickstart CI failing. It was flaky and I have fixed the flakiness here: #205

[nstogner]

I think we are good to merge once the changes to the default values.yaml file are reverted: #204 (comment)