How to use iOS Metal Delegate (GpuDelegate)?

[mspnr]

It is mentioned in the Readme, that this function is available:

iOS Metal Delegate (GpuDelegate)

If I use the following parameter my app is starting and working OK:

 var interpreterOptions = InterpreterOptions()..threads = 4;

If I use the following parameter the app is starting, but is not working:

final gpuDelegate = GpuDelegate(
      options: GpuDelegateOptions(true, TFLGpuDelegateWaitType.active),
    );
var interpreterOptions = InterpreterOptions()..addDelegate(gpuDelegate);

In XCode log I see the following error:

Runner[1383:458287] flutter: Error while creating interpreter: Invalid argument(s): Failed to lookup symbol (dlsym(RTLD_DEFAULT, TFLGpuDelegateCreate): symbol not found)

From that I assumed, that an appropriate TensorFlowLiteC.framework should be used:

1. I tried the framework from release v0.5.0 and it is not working.
2. I even tried to follow the steps behind this link, but ended up with this issue: Int incompatibility error on building TensorFlowLiteC framework tensorflow/tensorflow#48464

So my question is: how to let iOS Metal Delegate work?
Do you have an appropriate and compiled TensorFlowLiteC.framework? Can you share it?

[mspnr]

Here is what I came up with.
I tried to document everything, that was not so obvious for me and maybe for others. Let me know if something is missing.

Building TF Lite

Once again here is the start point.

The recommended option is ending up with issue tensorflow/tensorflow#48464, so no outcome here:
bazel build --config=ios_fat -c opt //tensorflow/lite/ios:TensorFlowLiteC_framework

To build version 2.4.1 and earlier I used (by adding experimental/):
bazel build --config=ios_fat -c opt //tensorflow/lite/experimental/ios:TensorFlowLiteC_framework

To build the latest version for arm64 only (by replacing ios_fat via ios_arm64):
bazel build --config=ios_arm64 -c opt //tensorflow/lite/ios:TensorFlowLiteC_framework

The second and the third options produced TensorFlowLiteC_framework.zip as described here, but running the app ended up with error regarding missing TFLGpuDelegateCreate symbol.

So I assumed, that GPU delegate metal function is not enabled in the framework. The idea was to include metal_delegate.h into the compiled framework.

Building GPU delegate metal

Unfortunately I didn't find a way to include the GPU delegate metal function to TensorFlowLiteC_framework. At the end I stumbled on this line in tensorflow, which gave an idea, that the function can be built separately.

So to build GPU delegate metal function I used:
bazel build --config=ios_arm64 -c opt //tensorflow/lite/ios:TensorFlowLiteCMetal_framework
And got a separate TensorFlowLiteCMetal_framework.zip.

Enabling GPU delegate metal function

On the next step I placed TensorFlowLiteCMetal.framework into plugin ios folder and added it into tflite_flutter.podspec:

s.ios.vendored_frameworks = 'TensorFlowLiteC.framework', 'TensorFlowLiteCMetal.framework'
s.xcconfig = { 'OTHER_LDFLAGS' => '-framework TensorFlowLiteC -all_load -framework TensorFlowLiteCMetal -all_load' }

To apply the changes I cleaned and rebuilt the flutter project:
flutter clean
flutter build ios

After that the error (magically) disappeared. The app is working all right and more than twice faster, than without GPU acceleration.

Observations

The option suggested in Readme is working all right:

final gpuDelegate = GpuDelegate(
      options: GpuDelegateOptions(true, TFLGpuDelegateWaitType.active),
    );
var interpreterOptions = InterpreterOptions()..addDelegate(gpuDelegate);

The other option though mentioned here #60 (comment) and also present in the documentation is not working all right.

Conclusion

• I hope this kind of tutorial worth to be added to Readme or Wiki
• I archived the result described here via trial and error, so I am not sure if everything described here is correct and would appreciate any feedback from the community

[am15h]

@applikationsprogramvara Thank you so much for this great tutorial. I am including this in readme as well.

[danghoaidanh]

Is there any example to prove that iOS is recognizing things? I used below function as you said but no faces is recognized at all. The app starts but no faces is recognized when you capture faces. In summary, if you know any examples to prove that iOS is working to realize faces then please let me know. Android is working but iOS.

Future loadModeliOS() async {
try {
final gpuDelegate = GpuDelegate(
options: GpuDelegateOptions(true, TFLGpuDelegateWaitType.active),
);
var interpreterOptions = InterpreterOptions()..addDelegate(gpuDelegate);
interpreter = await Interpreter.fromAsset('mobilefacenet.tflite', options: interpreterOptions);
} on Exception {
//print('Failed to load model ios.');
}
}

[am15h]

Please checkout @danghoaidanh https://github.com/am15h/object_detection_flutter, it works on iOS and Android both.

[am15h]

Hi @applikationsprogramvara, I have added this tutorial in wiki, https://github.com/am15h/tflite_flutter_plugin/wiki/Build-iOS-binaries-with-GPU-delegate. Please review it and make edits if required. Thanks for your contribution.

[am15h]

@applikationsprogramvara Can you share the TensorFlowLiteCMetal_framework.zip, for uploading in release assets of this project.

[CaptainDario]

@am15h, @applikationsprogramvara is the TensorFlowLiteCMetal_framework.zip now available somwhere?
I am having the same issue as described here (#152 ). But for me it is not possible to build the framework from source as i have a very low spec Mac.

[BytesZero]

I refer to this pr : #200

[jaheyy]

In case anyone has troubles with building tensorflow locally, this package contains TensorFlowLiteC.framework and TensorFlowLiteCMetal.framework that worked for me.