Thread Settings When Using ONNX Runtime Generate() API

[mitsunami]

When running an LLM using the ONNX Runtime Generate() API, what is the default setting for threads affinity? How can I check and configure the threads settings?

Currently, I am running the phi3.cpp example from this repository and the Example from the onnxruntime-inference-examples repository on Android mobiles, but I am noticing a performance difference and would like to understand the reason. I suspect it might be due to different thread settings, but it would be great if you could let me know if there are other factors.

[RyanUnderhill]

The current defaults are the same as onnxruntime, we just use the defaults when creating the onnxruntime session. I think it uses half the available cores maximum, but this is all outside of our library.

[mitsunami]

Understood, I'll have a look at the onnxruntime settings.
Thanks for your reply, RyanUnderhill!

[InigoNovoaINGUBU]

Is there any source that explains how to set full core usage for onnxruntime-genai?
This is because I have under used vCPUs on containers (Azure Container Instances) and want to use all available cores and get a full or near 100% CPU use.

[RyanUnderhill]

@InigoNovoaINGUBU You can look at the Thread Management section here: https://onnxruntime.ai/docs/performance/tune-performance/

Having 100% CPU usage isn't always better performance since more CPU time can be spent spinning vs doing useful work. The best you can do is try different options to see which gives the best performance (which won't necessarily correlate with CPU %)

[InigoNovoaINGUBU]

@RyanUnderhill I understand that this performance tuning is done by changing SessionOptions parameters, but currently (and I may be wrong), onnxruntime-genai does not expose any way to change these SessionOptions, or it is not documented.

• Python Load Model
• C# Create Model
• C Load Model

How can I change the default SessionOptions or pass custom SessionOptions?

[kunal-vaishnavi]

You can find more information about that here.