llama-cli CPU Control | Pin to Physical Cores

[Allan-Luu]

Hello, I'm trying to run llama-cli and pin the load onto the physical cores of my CPUs. On my processors, I have 128 physical cores and I want to run some tests on maybe the first 0-8, then 0-16, then 0-24, and so on adding 8 each time all the way until I get to using all 128 physical cores.

I'm trying to run this line:

./llama-cli -m /data/models/llama-2-7b.Q4_K_M.gguf -p "Tell me a joke" -n 8 --numa distribute -Crb 0-7 -Cb 0xFF --cpu-strict-batch 1

But when running htop I see most of the cores are lighting up, which is not my intention. I only want to see the processes run on the first 8 cores in this case.

Screenshot of htop CPU load when running the line of code above.

Would someone be able to help me figure out how to pin the load to the exact cores of the CPU?

[ggerganov]

Pinging @bmtwl for any insights

[bmtwl]

You're going to want to use numactl to give you fine-grained control over where the threads execute:
numactl -N0-7 -m0-7 /path/to/llama-cli -m /path/to/model.gguf -t 8 --numa numactl
You can check the layout of your system with numactl -H so that you're getting the behaviour you want when assigning numa nodes

[Allan-Luu]

Hey @bmtwl , do you know whether we have control of the split mode across CPUs? Similar to the --split-mode for GPUs?

[max-krasnyansky]

When we added the threadpool and the new --cpu-mask/range/strinct options we tried to avoid messing with the numa distribute logic.
So currently those two options (ie using both --numa distribute and --cpu-mask / --cpu-strict) are not compatible.
I'm going to follow up on this in the next round of threading updates (been meaning to work on that but keep getting distracted with other stuff).

@Allan-Luu based on your description above you don't even need to specify the --numa option.
Just use the --cpu-mask\range\strict options.

./llama-cli -m /data/models/llama-2-7b.Q4_K_M.gguf -p "Tell me a joke" -n 8 --cpu-range 0-7 --cpu-strict 1

This will strictly bind the threads to CPU cores 0-7, one thread per core.

[Allan-Luu]

thank you all for your help!

hmm, @max-krasnyansky when running the command in your response, it uses a thread count of 128 instead of 8 and uses all cores on the CPUs

./llama-cli -m /data/models/llama-2-7b.Q4_K_M.gguf -p "Tell me a joke" -n 8 --cpu-range 0-7 --cpu-strict 1

but when running with the suggestion from @bmtwl it works! I had to use -N0-1 to specify the 2 processors on my machine (2 chips), -C0,1,2,3,4,5,6,7 is specifying each CPU I intend to use, and -t 8 determines the number of threads I want for this run.

numactl -N0-1 -C0,1,2,3,4,5,6,7 ./llama-cli -m /data/models/llama2/llama-2-13b.Q2_K.gguf -t 8 --numa numactl

Is there any reason not to use numactl?

[max-krasnyansky]

Oh. Sorry. My bad. A little typo -n instead of -t.

./llama-cli -m /data/models/llama-2-7b.Q4_K_M.gguf -p "Tell me a joke" -t 8 --cpu-range 0-7 --cpu-strict 1

It's fine to use numactl in general but --cpu-mask\range\strict provide more precise control
(strict one thread per core, different masks for batch/non-batch, etc).

[MSZHabibie]

May I know why you included --numa numactl at the end? I tried running it without that, and it worked fine. BTW thanks for this post i can finally limit llama.cpp load model cpu core usage

[Allan-Luu]

I was using --numa numactl at the end based on bmtwl's answer above, didn't really consider whether to exclude it or not, but it's great to know that it works without it! And I'm glad my question was helpful to you.

And thanks for the explanation @max-krasnyansky I'll do some more testing when I get a chance to compare the two to see which one is a better fit for me.