GPU support for fastT5

[rupeshpoojary97]

You have used onnxruntime which is CPU compatible but are we looking forward to have onnxruntime-gpu?

[Ki6an]

I'm currently working on GPU support, there are a few issues, one of which is adding support for both onnxruntime (CPU) and onnxruntime-GPU in a single library and GPU also does not support quantization. hopefully, onnxruntime adds support for quantization for GPU soon.

[GenVr]

@Ki6an Meanwhile, is there any way to improve the inference times of a T5 on GPU?

[Ki6an]

@gennarovaccaro follow this thread #34 it'll give you a basic idea