[GUI] [FEATURE] Support CPU Pinning

[harvesterhci-io-github-bot]

GUI Issue from #2305

• In the host details page -> Basics tab, a new attribute called CPU Manager will be displayed, whose value reflects the node label cpumanager.

• In Hosts page, we'll have an action called Enabled / Disabled CPU Manager to switch cpu-manager-policy to static or none for specified node.

  • disable action if harvesterhci.io/cpu-manager-update-status annotation either requested or running.
  • enable action once harvesterhci.io/cpu-manager-update-status becomes success or failed
  • See HEP API Change

• In Virtual Machines page, we'll have an checkbox to enable cpu-pinning with tooltip (Changing the CPU pinning setting requires a manual reboot of the virtual machine for the changes to take effect) beside it.
  • once checked, add spec.template.spec.domain.cpu. dedicatedCpuPlacement = true in VM CR YAML

• To migrate a CPU pinning enabled VM, filter out non-enabled CPU manager nodes in target node dropdown.

In Hosts page, we'll have dedicated cpu metrics to show how much dedicated cpus are allocated per node (No needed)

Backend PRs

• Add CPU Manager related configs harvester-installer#756
• Support CPU Pinning #6190

[harvesterhci-io-github-bot]

Pre Ready-For-Testing Checklist

• If labeled: require/HEP Has the Harvester Enhancement Proposal PR submitted?
  The HEP PR is at:

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Have the backend code been merged (harvester, harvester-installer, etc) (including backport-needed/*)?
  The PR is at:

  • Does the PR include the explanation for the fix or the feature?

  • Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
    The PR for the YAML change is at:
    The PR for the chart change is at:

• If labeled: area/ui Has the UI issue filed or ready to be merged?
  The UI issue/PR is at:

• If labeled: require/doc, require/knowledge-base Has the necessary document PR submitted or merged?
  The documentation/KB PR is at:

• If NOT labeled: not-require/test-plan Has the e2e test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue?

  • The automation skeleton PR is at:
  • The automation test case PR is at:

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at:

[harvesterhci-io-github-bot]

Automation e2e test issue: harvester/tests#1390

[torchiaf]

In Hosts page, we'll have dedicated cpu metrics to show how much dedicated cpus are allocated per node.

We already have the CPU column in the Hosts page for this, unless we want to show more specific info related to CPU pinning changes.

cc @bk201

[TachunLin]

Basic UI changes check for CPU Pinning

Verified fixed on v1.4-cd10d18-head (24/09/19).

Result

$\color{green}{\textsf{PASS}}$ Enable CPU manager feature $~~$

1. Can enable the cpu manager correctly and display status on the host page
   

2. Can display cpu manager enabled in the host details page
   

$\color{green}{\textsf{PASS}}$ Enable CPU pinning feature $~~$

1. Filter out non-enabled CPU manager nodes in target node of migration drop-down
   

$\color{green}{\textsf{PASS}}$ Add "dedicatedCpuPlacement" to the vm yaml $~~$

1. The spec.template.spec.domain.cpu.dedicatedCpuPlacement set to true in the VM CR yaml.
   

$\color{green}{\textsf{PASS}}$ Filter nodes when migrating cpu pinning vm $~~$

1. filter out non-enabled CPU manager nodes in target node of migration drop-down
   

Test Information

• Test Environment: Single nodes harvester on equinix bare machines
• Harvester version: v1.1-340486a3-head (23/11/20)

Verify Steps

CPU manager and pinning UI changes

1. Enable CPU manager on the host page
2. Enable CPU pinning in the vm config page
3. Check the yaml file of the cpu pinning enabled vm
4. Migrate cpu pinning vm to non cpu manager node
5. Check the drop-down message and target node list

Next Step

Test the cpu manger and pinning feature test plan in
#2305 (comment)

[TachunLin]

Verified fixed on v1.4-cd10d18-head (24/09/19). Close this issue

Result

$\color{green}{\textsf{PASS}}$ Test enable/disable cpu manager through API on Harvester $~~$

1. Can correctly enable CPU manager through API on Harvester node

2. Can correctly disable CPU manager through API on Harvester node

$\color{green}{\textsf{PASS}}$ Test enable/disable cpu pinning through UI on virtual machine $~~$

1. Can correctly enable CPU pinning through UI on virtual machine

2. CPU pinning vm have the correct vcpupin requests in the virsh xml content

harvester-node-2:~ # kubectl exec virt-launcher-test-tglz2 -- virsh dumpxml default_test | awk "/<cputune>/,/<\/cputune>/"
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>

3. User can't disable cpu manager when there is a cpu pinned vm on harvester-node-2

4. Stop the vm, then we can disable the cpu manager on harvester-node-2 (host the cpu pinning) vm

5. Can correctly disable CPU pining through UI on virtual machine

6. After restart the Non CPU pinning vm did not have vcpupin requests in the virsh xml content

harvester-node-2:~ # kubectl exec virt-launcher-test-qlpfl -- virsh dumpxml default_test | awk "/<cputune>/,/<\/cputune>/"
harvester-node-2:~ #

$\color{green}{\textsf{PASS}}$ Test the cpu pinning vm can request specific core cpu on host to it's vgpu $~~$

1. Can correctly enable CPU pinning through UI on virtual machine

2. CPU pinning vm have the correct vcpupin requests in the virsh xml content

harvester-node-2:~ # kubectl exec virt-launcher-test-tglz2 -- virsh dumpxml default_test | awk "/<cputune>/,/<\/cputune>/"
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>

$\color{green}{\textsf{PASS}}$ Verify only one master node could enable/disable cpu manager in the same time $~~$

1. Can't enable cpu manager on another node when there is a node still in updating.

2. Can't disable cpu manager on another node when there is a node still in updating.

$\color{green}{\textsf{PASS}}$ Test reboot node with cpu manager enabled $~~$

1. After reboot node, the node should back to Active

2. The Harvester cluster is working

harvester-node-0:~ # kubectl get nodes
NAME               STATUS   ROLES                       AGE   VERSION
harvester-node-0   Ready    control-plane,etcd,master   11d   v1.28.12+rke2r1
harvester-node-1   Ready    control-plane,etcd,master   11d   v1.28.12+rke2r1
harvester-node-2   Ready    control-plane,etcd,master   11d   v1.28.12+rke2r1

3. CPU manager is still enabled on that node

4. VM enabled cpu pinning should be running well

5. VM request dedicate cpu core on node

harvester-node-1:~ # kubectl exec virt-launcher-test-kr6xq -- virsh dumpxml default_test | awk "/<cputune>/,/<\/cputune>/"
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>

$\color{green}{\textsf{PASS}}$ Test reboot cpu pinned vm $~~$

1. After reboot vm, the CPU pinning configuration exists

harvester-node-0:~ # kubectl get virtualmachines test0 -n default -o yaml | grep dedicate
          dedicatedCpuPlacement: true

2. VM can restarted well into running

3. vm request dedicate cpu core on node

harvester-node-0:~ # kubectl exec virt-launcher-test0-qcpm8 -- virsh dumpxml default_test0 | awk "/<cputune>/,/<\/cputune>/"
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>

4. After reboot VM will be located on any node with cpu manager enabled
   Restarted vm 10 times, most of the time, vm will started on harvester-node-1

$\color{green}{\textsf{PASS}}$ Test migrate vm with cpu pinning $~~$

1. While migrating cpu pinning vm, can display all nodes have cpu manager enabled
   
   

2. Can migrate vm to the node with cpu manager enabled

3. VM can restarted well into running

4. Filter out the node with no cpu manager enabled (UI)

5. Non cpu-pinning vm can migrate to cpu-manager enabled node with the correct configuration

$\color{green}{\textsf{PASS}}$ Disable the node have cpu manager when with cpu pinning VM on it $~~$

1. Disable the node have cpu manager when with cpu pinning vm on it

2. After stop the running vm, we can disable the cpu manager on that node

3. Then we can start the cpu pinning vm well

4. And the cpu pinning vm will be started on the node have cpu manager enabled

$\color{green}{\textsf{PASS}}$ (Single node) Test upgrade path (v1.4.0-rc1 to v1.4.0-rc1) $~~$

1. Can correctly upgrade to v1.4.0-rc1 with cpu manager enabled and have cpu pinning vm
   

2. After upgrade, the cpu manager option is still enabled on the node
   

3. We can start the cpu pinning vm, it still enabled the cpu pinning option
   

4. vm still request dedicate cpu core on node

harv140rc1:~ # kubectl exec virt-launcher-test-xphlk -- virsh dumpxml default_test | awk "/<cputune>/,/<\/cputune>/"
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>

5. Can disable the cpu manager on node

$\color{green}{\textsf{PASS}}$ (Multiple nodes) Test upgrade path (v1.4.0-rc1 to v1.4.0-rc1) $~~$

1. Can correctly upgrade to v1.4.0-rc1 with cpu manager enabled and have cpu pinning vm

2. After upgrade, the cpu manager option is still enabled on the node

3. The cpu pinning vm, it running and enabled the cpu pinning option

4. vm still request dedicate cpu core on node

harvester-node-2:~ # kubectl exec virt-launcher-test1-q6c4f -- virsh dumpxml default_test1 | awk "/<cputune>/,/<\/cputune>/"
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>
harvester-node-2:~ # k9s
harvester-node-2:~ # kubectl exec virt-launcher-test-lb45m -- virsh dumpxml default_test | awk "/<cputune>/,/<\/cputune>/"
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>

5. Can disable the cpu manager on node

Test Information

• Test Environment: Three nodes kvm machines
• Harvester version: v1.4-cd10d18-head (24/09/19)

Verify Steps

Test enable/disable cpu manager through API on Harvester

1. Create api key: click dashboard top right profile picture > Account & API Keys > Create API Key
2. Use curl cmd to enable cpu manager on harvester-node-0

    curl 'https://192.168.100.131/v1/harvester/nodes/{node_name}?action=enableCPUManager' \
    -X 'POST' \
    -H 'Authorization: Bearer <API_KEY>' \
    --insecure

4. Check the cpu manager have enabled on the selected node on UI
5. Check the cpu manager have enabled on the selected node from node content

  kubectl get nodes harvester-node-0 -o yaml | grep cpumanager

7. Use curl command to disable CPU manger on harvester-node-0

    curl 'https://192.168.100.131/v1/harvester/nodes/{node_name}?action=enableCPUManager' \
    -X 'POST' \
    -H 'Authorization: Bearer <API_KEY>' \
    --insecure

8. Check the cpu manager have disabled on the selected node on UI
9. Check the cpu manager have disabled on the selected node from node content

Test enable/disable cpu pinning through API on virtual machine

1. Already enable cpu manager on the node ready to host the vm

2. Create a vm with 2 core cpu and 2Gi memory

3. Edit the yaml

4. Add dedicatedCpuPlacement: true to .spec.template.spec.domain.cpu
   

5. Create the vm

6. Access the Harvester node

7. Check the virt-launcher pod of the vm

8. Access the vm-launcher pod to get the virsh xml content of the vm

9. Check the vm have requested the 2 dedicated core CPU from node
   kubectl exec virt-launcher-test-t5m8g -- virsh dumpxml default_test | awk "/<cputune>/,/<\/cputune>/"

10. Disable the cpu manager on the node have cpu pinning vm

11. Check should be blocked with message

12. Stop the vm

13. Disable the cpu manager on the node again

14. Check can disable the cpu manager correctly

15. Start the vm

16. Remove the dedicatedCpuPlacement: true to .spec.template.spec.domain.cpu

17. Restart the vm

18. Check the vm did not request any core CPU from node

Verify only one master node could enable/disable cpu manager in the same time

1. Ensure non of the node have enable the cpu manger
2. Use curl command to enable cpu manager on the first node harvester-node-0
3. Don't wait for the update complete
4. Directly enable cpu manager on the second node harvester-node-1
5. Check should be blocked with error message
6. Wait for the update complete
7. Enable cup manager on the second node
8. After node-0 and node-1 already enabled cpu manager
9. Use curl command to disable cpu manager on the first node harvester-node-1
10. Don't wait for the update complete
11. Directly disable cpu manager on the second node harvester-node-1
12. Check should be blocked with error message

Verify there should be only one job enable/disable cpu manager per node (API only)

1. Ensure non of the node have enable the cpu manger
2. Use curl command to enable cpu manager on the first node harvester-node-0
3. Wait for the update complete
4. Enable cpu manager on the first node harvester-node-0
5. Check should be blocked with error message
6. Disable cpu manager on node-0
7. Wait for the update complete
8. Use curl command to disable cpu manager on the first node harvester-node-0
9. Check should be blocked with error message

Verify we can't enable/disable cpu manager when it is enabled/disabled and (API only)

1. Ensure non of the node have enable the cpu manger
2. Use curl command to enable cpu manager on the first node harvester-node-0
3. Don't wait for the update complete
4. Directly enable cpu manager on the second node harvester-node-0
5. Check should be blocked with error message
6. Enable cup manager on node-1
7. After node-1 already enabled cpu manag
8. Use curl command to disable cpu manager on the first node harvester-node-1
9. Don't wait for the update complete
10. Directly disable cpu manager on the second node harvester-node-1
11. Check should be blocked with error message

Test reboot node with cpu manager enabled

1. Harvester-node-0 have cpu manager enabled and have cpu pinning vm on it
2. Reboot node0
3. Check after reboot node, the node should back to Active
4. Check the Harvester cluster is working
5. Check the cpu manager is still enabled on that node
6. Check VM enabled cpu pinning should be running well
7. Check VM can request dedicate cpu core on node

Test reboot cpu pinned vm

1. Harvester-node-0 have cpu manager enabled and have cpu pinning vm on it
2. Reboot the vm
3. Check after reboot vm, the CPU pinning configuration exists
4. Check VM can restarted well into running
5. Check vm can request dedicate cpu core on node
6. Check after reboot VM will be located on any node with cpu manager enabled

Test migrate vm with cpu pinning

1. Harvester-node-0 have cpu manager enabled and have cpu pinning vm on it
2. Harvester-node-1 and node-2 also enable cpu manager
3. Migrate the vm
4. Check all nodes with cpu manager enabled can display on the migration target list
5. Check can migrate vm to the node with cpu manager enabled
6. Check the VM can restarted well into running
7. Disable the cpu manager on node-1 node
8. Migrate the vm
9. Check the target list do not display the node-1
10. Create another vm with no cpu pinning named no-pinning
11. Migrate the no-pinning vm
12. Check can display all the nodes on the target list
13. Check the migrate the vm and running well
14. Check the migrated vm did not have cpu pinning enabled.

Disable the node have cpu manager when with cpu pinning VM on it

1. Harvester-node-0 have cpu manager enabled and have cpu pinning vm on it
2. Disable the cpu manager when vm is running
3. Check should display the block message
4. Stop the vm
5. Disable the cpu manager again
6. Check can disable the cpu manager on node-0
7. Start the vm
8. Check the vm will be started on another node with cpu manager enabled