Add preliminary support of OpenVINO as Keras 3 backend

[rkazants]

Details: Support OpenVINO as Keras 3 backend. This is inference-only backend. In order to switch on this, define environment variable as follows: os.environ["KERAS_BACKEND"] = "openvino" or use set_backend.

Here is an example how it works:
Install OpenVINO pip install openvino -U

import os
os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras

# build a simple network
x1 = keras.Input(shape=(2,), dtype='float32')
x2 = keras.Input(shape=(2,), dtype='float32')
add = keras.layers.Add()([x1, x2])
sigmoid = keras.activations.sigmoid(add)
model = keras.Model(inputs={'x1': x1, 'x2': x2}, outputs={'sigmoid': sigmoid})
model.summary()

# infer using OpenVINO
out = model.predict({'x1': np.array([[1.0, 2.0]], dtype=np.float32),
                     'x2': np.array([[3.0, 4.0]], dtype=np.float32)})
print('openvino out = ', out)

Added support for operations that allow to infer mobilenet_v3_small and bert_base_en_uncased from Keras Hub.

Example 1:

# from here https://keras.io/api/keras_nlp/models/bert/bert_text_classifier/#from_preset-method
import os

os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras
#import keras_hub

features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "segment_ids": np.array([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0]] * 2),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}

# Pretrained classifier without preprocessing.
#classifier = keras_hub.models.BertTextClassifier.from_preset(
#    "bert_base_en_uncased",
#    num_classes=4,
#    preprocessor=None,
#)
#keras.saving.save_model(classifier, "bert_base_en_uncased.keras")
classifier = keras.saving.load_model("bert_base_en_uncased.keras")
#output = classifier.predict(x=features, batch_size=2)
output = classifier.predict(features)

Example 2:

# from here https://keras.io/api/keras_cv/models/backbones/mobilenet_v3/
import os

os.environ["KERAS_BACKEND"] = "openvino"
import numpy as np
import keras
import keras_cv

rng = np.random.default_rng(23345243)

#input_data = np.random.rand(8, 224, 224, 3)
input_data = rng.uniform(0.0, 1.0, [8, 224, 224, 3]).astype(np.float32)
model = keras.saving.load_model("mobilenet_v3_small.keras")
#model = keras_cv.models.MobileNetV3Backbone.from_preset("mobilenet_v3_small",)

output = model.predict(input_data)

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[fchollet]

Thanks for the PR!

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 60.14735% with 595 lines in your changes missing coverage. Please review.

Project coverage is 81.87%. Comparing base (4c05e0c) to head (74778b1).

Files with missing lines	Patch %	Lines
keras/src/backend/openvino/numpy.py	53.71%	254 Missing and 20 partials ⚠️
keras/src/backend/openvino/core.py	59.40%	126 Missing and 23 partials ⚠️
keras/src/backend/openvino/trainer.py	27.00%	100 Missing ⚠️
keras/src/backend/openvino/nn.py	77.25%	49 Missing and 4 partials ⚠️
keras/src/backend/openvino/math.py	82.97%	6 Missing and 2 partials ⚠️
keras/src/backend/openvino/random.py	94.11%	1 Missing and 2 partials ⚠️
keras/src/utils/backend_utils.py	25.00%	3 Missing ⚠️
keras/src/backend/__init__.py	75.00%	0 Missing and 1 partial ⚠️
keras/src/backend/openvino/rnn.py	85.71%	1 Missing ⚠️
keras/src/export/export_lib.py	50.00%	0 Missing and 1 partial ⚠️
... and 2 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #19727      +/-   ##
==========================================
- Coverage   82.53%   81.87%   -0.66%     
==========================================
  Files         530      542      +12     
  Lines       49058    50550    +1492     
  Branches     7627     7809     +182     
==========================================
+ Hits        40489    41390     +901     
- Misses       6731     7271     +540     
- Partials     1838     1889      +51     

Flag	Coverage Δ
keras	81.71% <60.14%> (-0.66%)	⬇️
keras-jax	63.78% <0.06%> (-1.95%)	⬇️
keras-numpy	58.82% <0.13%> (-1.79%)	⬇️
keras-openvino	29.98% <60.01%> (?)
keras-tensorflow	64.54% <0.06%> (-1.97%)	⬇️
keras-torch	63.61% <0.06%> (-1.94%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[gbaned]

Hi @rkazants Can you please resolve the conflicts? Thank you!

[rkazants]

Hi @rkazants Can you please resolve the conflicts? Thank you!

I am working on this PR. I am trying to resolve comments.

Best regards,
Roman

[gbaned]

Hi @rkazants Any update on this PR? Please. Thank you!

[github-actions]

This PR is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[rkazants]

I am just back from vacation. I plan to continue from the next week.

[github-actions]

This PR is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.

[gbaned]

Hi @rkazants Any update on this PR? Please. Thank you!

[rkazants]

Hi @rkazants Any update on this PR? Please. Thank you!

Hi @gbaned, please anticipate the update early this week. Sorry for the delay.

Thanks,
Roman

[rkazants]

Thanks Roman! Is there any way to try this without the hardware?

Jeff,
sure, you can try this out on your laptop with x86, ARM CPU (Win, Linux, MacOS) or Intel GPU, NPU hardware? Just install OpenVINO by pip install -U openvino. See some examples in PR description.

[rkazants]

Hi @fchollet, @jeffcarp, all comments were resolved so far. Please let me know what else is needed for merging this PR.

Thanks a lot,
Roman

[jeffcarp]

Thanks Roman! LGTM from my side. Please also get an LGTM from Francois before proceeding.

[jeffcarp]

sure, you can try this out on your laptop with x86

Thanks! I was able to confirm running this branch locally.

[rkazants]

@fchollet, all is done. Please review it again.

Thank you very much,
Roman

[fchollet]

LGTM -- thank you for the contribution! 👍

Should we expect an announcement from Intel here? How should we communicate this to the developer community?

[rkazants]

LGTM -- thank you for the contribution! 👍

Should we expect an announcement from Intel here? How should we communicate this to the developer community?

Hi François (fchollet),

Really thanks for your big help in reviewing this PR.
From OpenVINO side, so plan to announce about this feature in the upcoming OpenVINO 2025.0 release in January. It will be declared as experimental feature. We will add this into OpenVINO release notes and add into documentation "how to deploy Keras 3 models using OpenVINO" (https://docs.openvino.ai/2024/openvino-workflow.html), add OpenVINO notebook to showcase Keras 3 + OpenVINO. Once we reach close parity with NumPy backend in terms of operation coverage and reach quite the same test passrate, we will remove "experimental". I will also make a post in LinkedIn and our Discord to invite open-source community to contribute to this very soon.

From Keras side, I will update Keras documentation as well and declare it as experimental feature. Please let me know how to proceed with Keras documentation. Also, can you add this into Keras release notes of the upcoming Keras release from master? Please let me know what will be Keras release number from master that will include this changes? And when release is planned?

Best regards,
Roman

[fchollet]

We will include the new backend in the release notes for the next release. Please feel free to propose edits to the docs website via a PR to keras-team/keras-io!