Enable NAP in 'aaa' cluster

[mborsz]

Context:

Currently perf-dash has cpu request/limit set to 3 cores and it still insufficient:

Also is being continuously OOMKilled with memory limit set to 8GiB.

The node pool added in #659 for perfdash is using n1-standard-4 which provides 4 cores and 12 GiB of allocatable memory.

I would like to increase perf-dash's requests to at least 6 cpus and 16 GiB (to give some room for growth), but I'm not able to do that due to the used machine type.

Instead of adding 'pool-3' node pool with larger machine type, I would like to propose enabling Node auto-provisioning on this cluster that will be adding such node pools as needed, reducing maintenance burden.

[mborsz]

I would like to hear feedback on this change. I understand that there may be some reasons why we may want to avoid enabling this, but I would like to be sure that this is a conscious that we do not enable this here and continue manually adding larger and larger node pools.

[spiffxp]

/uncc @mikedanese @nikhita
/cc @spiffxp @BenTheElder
/assign @thockin @dims @ameukam
I'd like to hear your opinions. I have very little experience managing aaa specifically, or clusters using this feature.

[dims]

we forgot @cblecker :)

[ameukam]

@mborsz Did you consider the possibility we can change the instance type to n1-standard-8 or another instance type for the node-pool pool-2 instead of enable the auto-scaling at the cluster level ?

[thockin]

I'm skeptical of why perf-dash needs that much memory - the trajectory is problematic. That said, NAP seems like the rightest answer. This is sort of exactly what it is for. So I am +1

Isn't there an overall cluster limit on resources, too?

[mborsz]

I'm skeptical of why perf-dash needs that much memory - the trajectory is problematic. That said, NAP seems like the rightest answer. This is sort of exactly what it is for. So I am +1

perf-dash resource usage: perf-dash keeps all the data in memory and periodically refreshes state from GCS. We are continuously adding new tests or change existing ones (e.g. kubernetes/test-infra#20448), this will cause memory growth for sure.

Most of the cpu and memory is used when perf-dash either initializes or refreshes state, most likely for network buffers and json deserialization. For most of the time resource usage is around ~700MiB and cpu usage jumps between 0 and 3 cores (the current limit) when it refreshes state.

I'm think we can optimize perf-dash to reduce resource usage, but I'm not sure if this is a good area to invest time of our team there. For comparison: perf dash consumes currently 3 cores and 10Gi of memory, while e.g. our 5000 node test consumes ~5000 cores and ~20TiB of memory. I think that e.g. optimizing our 5000 node test has better ROI and we have some achievements there (e.g. kubernetes/perf-tests#561).

[mborsz]

Maciej Borsz Did you consider the possibility we can change the instance type to n1-standard-8 or another instance type for the node-pool pool-2 instead of enable the auto-scaling at the cluster level ?

Yes, it's not possible to change instance type, we need to create a new node pool with larger machine. Technically we can do that, but I'm afraid that some day we will need to add even larger node pool. This is why I suggest starting using NAP.

[BenTheElder]

I have no experience with NAP.

If we review the configuration for requested resources though then this seems fine. Instead of debating how big the node pool config should be set to we can debate how big the pod requests should be 🙃

[thockin]

+1 Ben This is the way. /lgtm /approve

…

On Fri, Feb 12, 2021 at 1:42 AM Benjamin Elder ***@***.***> wrote: I have no experience with NAP. If we review the configuration for requested resources though then this seems fine. Instead of debating how big the node pool config should be set to we can debate how big the pod requests should be 🙃 — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#1604 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKWAVHQP74Q3SQZ6LZ3F5DS6TZZTANCNFSM4XAPB62A> .

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mborsz, thockin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [thockin]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[thockin]

I didn't mean to unilaterally approve - looking for consensus :)

[ameukam]

I may be wrong, but I think resource_limits is a sub-block of cluster_autoscaling. You'll need to split the resource types.

[mborsz]

Fixed, thanks!

[ameukam]

/lgtm
/hold
If others want to comment.

[mm4tt]

Can we merge this? Or are we waiting for someone else approval?

[thockin]

I think we can enable this but someone needs to run terraform on it ASAP, so whomever has time should remove the hold and do it. Not me, at least not today.

[mborsz]

@thockin can you suggest anyone who will be able to run terraform on this in the next couple of days?

[mborsz]

@BenTheElder @ameukam @spiffxp can you run terraform on this?

[ameukam]

@mborsz Will run this Monday if nobody is faster than me.

@spiffxp NAP will create instances of the E2 family for any new workload.

[spiffxp]

I have no objections but can't commit to running this just now, if @ameukam wants to take it, go for it

[ameukam]

/hold cancel

[ameukam]

Running terraform apply -auto-approve -target google_container_cluster.cluster

[ameukam]

Change applied successfully :

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.