[XPU] XPU accelerator support for Intel GPU device

[delock]

This PR includes XPU support for Intel GPU. With this PR, DeepSpeed can support XPU devices without install Intel Extension for DeepSpeed.

[delock]

SYCLAutoOPBuilder is integrated to convert CUDA kernels into SYCL kernels. Currently transformer inference kernels will be converted automatically during installation time. We are investigating whether we can expand this builder to other kernels so we can reduce SYCL kernel files.

@baodii who is working on SYCLAutoOPBuilder.

[delock]

@tjruwase With @baodii 's contribution we have SYCLAutoBuilder which converts CUDA kernel into SYCL kernel used by Intel GPU. Now we can remove most manually written SYCL kernel in this PR (only one left and fix is on the way).

[CaoZhongZ]

oneapi-src/SYCLomatic#1398 Issue for last residual SYCL porting, when it's done we'll fully migrate all kernels. @delock @baodii

[delock]

Hi @tjruwase @jeffra, after internal discussion, we removed SyclAutoOpBuilder due to consideration that altough we have validated automatically converted SYCL kernel at this time, if there is new CUDA code change in existing CUDA kernel and these new changes had been automatically converted with SyclAutoOPBuilder, we might have broken functionality and performance in converted SYCL code, if this process is fully automated.

Instead, we prefer to 1) convert CUDA kernels with SyclAutoOpBuilder offline, 2) validate these converted SYCL code 3) upstream the valdiated SYCL kernel. Current SYCL kernels in this PR is the result of this process.

Let us know your thoughts and comments and we can discussion how to go forward. Thanks!

[tjruwase]

@delock, thanks for the update. If I understand correctly, you plan to upstream new SYCL kernels via PRs?

[delock]

@delock, thanks for the update. If I understand correctly, you plan to upstream new SYCL kernels via PRs?

Hi @tjruwase , it depends. For supporting DeepSpeed OpBuilder we have two methods:

1. If the functionality in OpBuilder is relative specific to DeepSpeed, we plan to upstream SYCL kernels for these OpBuilder via PRs. Optimizer for DeepSpeed is the set of SYCL kernels we are upstreaming.
2. If the functionality in OpBuilder is relative generic, we plan to build the kernel inside Intel Extension for PyTorch, and reuse the functionality inside OpBuilder implementation. This is something similiar to NPU's implementation https://github.com/microsoft/DeepSpeed/blob/master/op_builder/npu/fused_adam.py , in this case no SYCL kernel will be upstreamed.

On a big picture, we expect most DeepSpeed feature needs OpBuilder supported through method 2 for XPU, with the intention that the functionality could also be reused elsewhere. We may see method 1 be used in two situations:

1. If we see the kernel function is DeepSpeed specific.
2. If there is contribution from other party, being able to implement through SYCLOpBuilder is a more direct way to contribute.

[tjruwase]

@delock, apologies for the delay in reviewing PR. We will prioritize this now.

[mrwyattii]

Thanks @delock this all looks great! Do you have tests that you are running internally to verify this code?

[delock]

Thanks @delock this all looks great! Do you have tests that you are running internally to verify this code?

Hi @mrwyattii yes we validate this code on XPU device regulary for inference and training workloads.