OpenVINO

OpenVINO

OpenVINO is an open-source toolkit for optimizing and deploying deep learning models.

• Compiles models for your hardware.
• Supports Linux and Windows
• Supports CPU / iGPU / GPU / GNA / NPU
• Supports AMD GPUs on Windows with FP16 support. (Double performance over DirectML.)
• Supports INTEL GPUs.
• Supports NVIDIA GPUs.
• Supports CPUs with FP16, INT8 and INT4 support. (Double performance over PyTorch.)
• Supports multiple devices at the same time using Hetero Device.

It is basically a TensorRT / Olive competitor that works with any hardware.

INT8 / INT4

Currently, INT8 and INT4 with OpenVINO is CPU only. Enable Compress Model weights with NNCF option in Compute Settings to use it.
Select a 4 bit mode from OpenVINO compress mode for NNCF to use INT4.

Custom Devices

Use the OpenVINO devices to use option in Compute Settings if you want to specify a device.
Selecting multiple devices will use multiple devices as a HETERO device.

Using --device-id cli argument with the WebUI will use a GPU with the specified Device ID.
Using --use-cpu openvino cli argument with the WebUI will use the CPU.

Model Caching

OpenVINO will save compiled models to cache folder so you won't have to compile them again.
OpenVINO disable model caching option in Compute Settings will disable caching.
Directory for OpenVINO cache option in System Paths will set a new location for saving OpenVINO caches.

Limitations

Same limitations with TensorRT / Olive applies here too.
Compilation takes a few minutes and any change to Resolution / Batch Size / LoRa will trigger recompilation.
Attention Slicing and HyperTile will not work.
OpenVINO will lock you in the Diffusers backend.

Installation

Preparations

• Install the drivers for your device.
• Install git and python.
• Open CMD in a folder you want to install SD.Next.

Using SD.Next with OpenVINO

Install SD.Next from Github:

git clone https://github.com/vladmandic/automatic

Then enter into the automatic folder:

cd automatic

Then start WebUI with this command:

Windows:

.\webui.bat --use-openvino

Linux:

./webui.sh --use-openvino