graph-v0.9

This is the Beta Update 3 release of oneDNN Graph API based on oneDNN v3.0.1.

Performance Optimizations

• Improved multi-level perceptron (MLP) and residual block subgraphs performance with oneDNN Graph Compiler backend on 4th generation Intel Xeon Scalable processors (formerly Sapphire Rapids).
• Improved dynamic shape performance for MLP and multi-head attention (MHA) patterns with oneDNN Graph Compiler backend.
• Improved performance of oneDNN Graph Compiler built-in code generator.

Functionality

• Extended the set of multi-head attention (MHA) variants supported by oneDNN Graph Compiler.

Known Issues and Limitations

• The weight’s opaque layout can be queried only from a compiled partition.

Thanks to the Contributors

This release contains contributions from the project core teams as well as Jiong Gong, Chunyuan Wu, Sanchit Jain, Yiqiang Li, Yunfei Mao, Kiefer Kuah and others.

v3.1-rc

This is a release candidate for oneDNN v3.1. Please provide feedback and submit defect reports via Github issues.

Performance Optimizations

• Intel Architecture Processors:

  • Improved performance for 4th generation Intel Xeon Scalable processor (formerly Sapphire Rapids).
  • Introduced initial optimizations for future Intel Xeon Scalable processor (code name Sierra Forest). The functionality is disabled by default and should be enabled via CPU dispatcher control.

• Intel Graphics Products:

  • Improved performance for Intel Data Center GPU Max Series (formerly Ponte Vecchio).
  • Improved performance for Intel Arc graphics (formerly Alchemist and DG2) and Intel Data Center GPU Flex Series (formerly Arctic Sound-M).
  • Improved concat primitive performance with per-argument scales on Intel GPUs.

• AArch64-based Processors:

  • Improved layer normalization primitive performance with Compute Library for the Arm Architecture (ACL).

• AMD GPUs:

  • Introduced optimized matmul implementation.

• RISC-V-based Processors:

  • Improved pooling primitive performance for processors with RISC-V vector extension (RVV) support.

Functionality

• Enabled Graph API as a production feature. Graph API is intended to simplify oneDNN integration into frameworks.
• Added an option to zero-out weight gradient in RNN primitive. See details in corresponding RFC.
• [experimental] Added support for sparse memory and dense by sparse matrix-matrix multiplication support in the matmul primitive. The functionality is supported on processors with Intel AVX2 and Intel AVX-512 instruction support.
• Introduced out-of-order queues support for OpenCL runtime. See the OpenCL Interoperability section in the Developer Guide for more details.
• Added support for the non-zero alpha parameter in the batch normalization ReLU post-op on Intel GPUs.
• Enabled the layer normalization primitive with f64 datatype support on Intel GPUs.
• Added support of per-argument scales in matmul, convolution, inner product, and reorder primitives on NVIDIA GPUs.

Validation

• Extended benchdnn with functional and performance validation for Graph API.

Breaking Changes

• Builds with OpenCL runtime will fail unless Graph API is disabled with ONEDNN_BUILD_GRAPH=OFF.

Known Issues and Limitations

• Graph API constant cache feature is disabled with SYCL CPU runtime due to an issue with the oneAPI DPC++ Compiler runtime. This will result in lower performance for some scenarios.

Thanks to the Contributors

This release contains contributions from the project core team as well as Amy Wignall @AmyWignall-arm, Annop Wongwathanarat @annop-w, @arlesniak, @bdmoore1, Crefeda Rodrigues @cfRod, David Svantesson @davsva01, Fadi Arafeh @fadara01, Jonathan Deakin @jondea, Kentaro Kawakami @kawakami-k, Pavel Zamelin @pazamelin, Pawel Piotrowicz @pawelpiotrowicz, Peter Caday @petercad, @ranzhejiang, and Sanchit Grover @sanchit-grover-intel. We would also like to thank everyone who asked questions and reported issues.

v3.0.1

This is a patch release containing the following changes to v3.0:

• Fixed potential correctness issue in convolution weight gradient with 1x1 filter and strides (e589966)
• Improved convolution, deconvolution, inner product, and matmul primitives performance with scales on Intel CPUs (38319f1, 18de927, b6170d1, 85171b0)
• Reverted MEMFD allocator in Xbyak to avoid fails in high load scenarios (eaaa41b)
• Fixed array out of bounds issue in bfloat16 convolution weight gradient on Intel CPUs (a17a64c)
• Improved compatibility with future versions of Intel GPU driver (eb7a0a0)
• Fixed segfault in fp16 and bfloat16 convolution backward propagation on systems with Intel AMX support (293561b)
• Fixed build issue with GCC 13 (1d7971c)
• Fixed correctness issue in int8 RNN primitive Vanilla GRU flavor on Intel CPUs (f4a149c, fbf8dca)
• Added check for unsupported arguments in binary primitive implementation for AArch64-based processors (5bb9070)
• Fixed correctness issue in int8 convolution with zero-points on Intel Data Center GPU Max Series (96e868c)
• Fixed runtime error in convolution primitive with small number of channels on Xe-based graphics (068893e)
• Removed use of OpenCL C variable length arrays in reduction primitive implementation for Intel GPUs (41e8612)
• Fixed correctness issue in matmul and inner product primitives on Intel Data Center GPU Max Series (a1e6bc5, dbb7c28)
• Fixed segfault in fp16 and bfloat16 convolution backward propagation on future Intel Xeon processors (code name Sierra Forest) (399b7c5)
• Fixed runtime error in Graph API for partitions with quantized matmul and add operations (f881da5, 699ba75, b8d21a5, 9421fb2)
• Fixed convolution performance regression on Xe-based graphics (1869bf2)
• Improved convolution performance with OHWI and OIHW weight formats on Intel Data Center GPU Max Series (2d0b31e, 5bd5d52)
• Fixed include files handling in build system affecting CMake projects relying on oneDNN (c616453)
• Added tbb::finalize to tests and examples to address intermittent test crashes with TBB runtime (891a415, c79e543, 8312c3a, 1a32b95, bd0389d, f05013d, ab7938f, 31c9e7b, f3261e4, d58ac41, f8c67b9, 258849b, b20a8c7)
• Fixed segfault in fp16 convolution primitive on future Intel Xeon processors (code name Granite Rapids) (a574fff)
• Fixed correctness issue in fp16 convolution primitive on future Intel Xeon processors (code name Sierra Forest) (f165ed8)
• Fixed correctness issue in int8 convolution primitive on Intel CPUs (ca15922, 27845b8)
• Fixed correctness issue in int8 convolution primitive on Intel Data Center GPU Max Series (8bb651c)
• Fixed correctness issue in resampling primitive with post-ops on Intel CPUs (aa52a51)
• Addressed excessive memory consumption in 3D convolution on Intel CPUs (3d6412a, 097acb5, fd69663)
• Fixed segfault in convolution with sum and relu post-ops on Intel CPUs (63ad769, 1b13037, 0a8116b, 9972cb8)
• Addressed convolution performance regression with small number of channels on Intel GPUs (d3af877)
• Worked around MSVS 2019 bug resulting in build fails on Windows (4024775)
• Updated code base formatting to clang-format 11 (23576f9, 0b1bf84)

graph-v0.8.1

This is a patch release containing the following changes to graph-v0.8:

• Upgraded oneDNN dependency from v2.7.2 to v2.7.3 (93237aa, 260bdb5)
• Fixed a correctness issue of quantized Convolution + Add fusion (26a9a5b, beba352)
• Fixed query_dynamic_outputs() interface implementation in graph compiler backend (8dbca04)

v2.7.3

This is a patch release containing the following changes to v2.7.2:

• Fixed segfault in int8 convolution with binary post-ops on Intel CPUs (c8d40c0)
• Applied workaround for tanh post-op on some Xe architecture based GPUs (3eb3267)
• Disabled fp16 post-ops with Compute Library for Arm Architecture (ACL) (f7b7dc0)
• Fixed incorrect results for sequence of eltwise post-op with same algorithm but different parameters (02c2678, 1c36e27, 81ba0fe)
• Fixed issue in convolution with groups and plain activation layout on Intel GPUs (df6f2e3, d0c14c2)
• Fixed reorder failures on Xe HPC architecture based GPUs (c3cb1d5)
• Fixed thread safety issue in convolution primitive (2955c9d)
• Fixed scratchpad allocation issue in matmul (989acd3)
• Disabled concat batching with scales on Intel GPUs since implementation doesn't support it yet (8aab73f, 1eac450, 82838de)
• Fixed segfault and correctness issues in convolution primitive with sum and relu post-ops on Intel CPUs (fc335be, 0f4697a, 60f1727, d28f2c1, 4761ee9, f674fbf)

v3.0

Performance Optimizations

• Intel Architecture Processors:
  • Improved performance for 4th generation Intel Xeon Scalable processor (formerly Sapphire Rapids).
  • Introduced FP16 support and initial optimizations for future Intel Xeon Scalable processor (code name Granite Rapids). The functionality is disabled by default and should be enabled via CPU dispatcher control.
• Intel Graphics Products:
  • Improved performance for Intel Data Center GPU Max Series (formerly Ponte Vecchio).
  • Improved performance for Intel Arc graphics (formerly Alchemist and DG2) and Intel Data Center GPU Flex Series (formerly Arctic Sound-M).
• AArch64-based Processors:
  • Improved reorder performance for processors with Scalable Vector Extensions (SVE) support.
  • Improved pooling performance with post-ops for processors with SVE 512 support.
  • Improved batch normalization performance with non-default flags for processors with SVE 512 support.
  • Improved performance of FP16 functionality with Compute Library for Arm Architecture (ACL).
  • Improved deconvolution performance with ACL.
• PowerPC64-based Processors:
  • Improved int8 GEMM performance.

Functionality

• Introduced new quantization scheme. Major changes include support for per-argument runtime scales in all primitives and unquantized bias.
• [experimental] Introduced Graph API support that simplifies oneDNN integration into applications. The functionality is disabled by default and can be enabled at build time with ONEDNN_BUILD_GRAPH=ON flag.
• Introduced support for Intel DPC++/C++ Compiler 2023.0, including new features from the SYCL 2020 standard.
• Extended persistent cache to cover GPU engine object. This improvement allows applications to further reduce oneDNN initialization time.
• Extended threadpool API with a function to indicate maximum available concurrency.
• Extended binary primitive implementation on GPU with bfloat16 source and int8 destination support.
• Introduced pooling and reduction primitives support on AMD GPUs.
• Introduced reduction primitive support on NVIDIA GPUs.

Usability

• Extended the set of supported format tags to cover formats used in applications.

Validation

• Extended the GoogleTest (gtest) suite with support for Parametric Rectified Linear Unit (PReLU) primitive.

Breaking Changes

• Removed deprecated APIs.
• Removed operation descriptor object and made memory descriptor object opaque. See details in operation and memory descriptors RFC.
• Removed creation time primitive scales support and primitive output scales support. See details in quantization scaling RFC.
• Removed support for Intel DPC++/C++ Compiler 2022 and SYCL 1.2.1 (aka SYCL 2017) standard support. Use Intel DPC++/C++ Compiler and SYCL 2020 standard instead.
• Removed Winograd convolution implementation for int8 data type.
• Updated minimal supported ACL version to 22.08 (was 22.05).

Thanks to the Contributors

This release contains contributions from the project core team as well as @akshatasangelkar, Aryan Karumuri @AryanKarumuri, Crefeda Rodrigues @cfRod, Divakar Mariyanna @bmdivakar, Gordon Fossum @austinpagan, Jonathan Deakin @jondea, Kentaro Kawakami @kawakami-k, lilianhuang @lilh9598, Milos Puzovic @milpuz01, Mona Minakshi @monaminakshi, Nathan John Sircombe @nSircombe, Peter Caday @petercad, and Sreekanth Yalachigere @sreekanth-yalachigere. We would also like to thank everyone who asked questions and reported issues.

graph-v0.8

This is the Beta Update 2 release of oneDNN Graph API based on oneDNN v2.7.2.

Functionality

• Added HardSigmoid operation.
• Added block tensor layout support to improve performance on Xe architecture-based GPUs.
• Added support of IOX and XOI weight formats for ConvTranspose operation.
• Added query_dynamic_outputs API to support dynamic shapes in the graph. This functionality allows Graph API to infer output tensors shapes based on input tensors.
• Experimental: Introduced dynamic shapes support for MHA via oneDNN Graph Compiler.

Known Issues and Limitations

• The weight’s opaque layout can be queried only from a compiled partition, which requires that input tensor shapes must be known at compilation time.
• MHA and MLP fusion are not activated on machines without Intel AVX-512 support.

Thanks to the Contributors

This release contains contributions from the project core teams as well as Jiong Gong, Chunyuan Wu, Sanchit Jain, Yiqiang Li, Yunfei Mao, Kiefer Kuah and others.

v3.0-rc

This is a release candidate for oneDNN v3.0. Please provide feedback and submit defect reports via Github issues.

Performance Optimizations

• Intel Architecture Processors:
  • Improved performance for 4th generation Intel Xeon Scalable processor (formerly Sapphire Rapids).
  • Introduced FP16 support and initial optimizations for future Intel Xeon Scalable processor (code name Granite Rapids).
• Intel Graphics Products:
  • Improved performance for Intel Data Center GPU Max Series (formerly Ponte Vecchio).
  • Improved performance for Intel Arc graphics (formerly Alchemist and DG2) and Intel Data Center GPU Flex Series (formerly Arctic Sound-M).
• AArch64-based Processors:
  • Improved reorder performance for processors with Scalable Vector Extensions (SVE) support.
  • Improved pooling performance with post-ops for processors with SVE 512 support.
  • Improved batch normalization performance with non-default flags for processors with SVE 512 support.
  • Improved performance of FP16 functionality with Compute Library for Arm Architecture (ACL).
  • Improved deconvolution performance with ACL.
• PowerPC64-based Processors:
  • Improved int8 GEMM performance.

Functionality

• Introduced new quantization scheme. Major changes include support for per-argument runtime scales in all primitives and unquantized bias.
• [experimental] Introduced Graph API support that simplifies oneDNN integration into applications. The functionality is disabled by default and can be enabled at build time with ONEDNN_BUILD_GRAPH=ON flag.
• Introduced support for Intel DPC++/C++ Compiler 2023.0, including new features from the SYCL 2020 standard.
• Extended persistent cache to cover GPU engine object. This improvement allows applications to further reduce oneDNN initialization time.
• Extended threadpool API with a function to indicate maximum available concurrency.
• Extended binary primitive implementation on GPU with bfloat16 source and int8 destination support.
• Introduced pooling and reduction primitives support on AMD GPUs.
• Introduced reduction primitive support on NVIDIA GPUs.

Usability

• Extended the set of supported format tags to cover formats used in applications.

Validation

• Extended the GoogleTest (gtest) suite with support for Parametric Rectified Linear Unit (PReLU) primitive.

Breaking Changes

• Removed deprecated APIs.
• Removed operation descriptor object and made memory descriptor object opaque. See details in operation and memory descriptors RFC.
• Removed creation time primitive scales support and primitive output scales support. See details in quantization scaling RFC.
• Removed support for Intel DPC++/C++ Compiler with SYCL 1.2.1 (aka SYCL 2017) standard.
• Removed Winograd convolution implementation for int8 data type.
• Updated minimal supported ACL version to 22.08 (was 22.05).

Thanks to the Contributors

This release contains contributions from the project core team as well as @akshatasangelkar, Aryan Karumuri @AryanKarumuri, Crefeda Rodrigues @cfRod, Divakar Mariyanna @bmdivakar, Gordon Fossum @austinpagan, Jonathan Deakin @jondea, Kentaro Kawakami @kawakami-k, lilianhuang @lilh9598, Milos Puzovic @milpuz01, Mona Minakshi @monaminakshi, Nathan John Sircombe @nSircombe, Peter Caday @petercad, and Sreekanth Yalachigere @sreekanth-yalachigere. We would also like to thank everyone who asked questions and reported issues.

graph-v0.7.2

This is a patch release containing the following changes to graph-v0.7.1:

• Upgraded oneDNN dependency to v2.7.2 (dec9f8c)

v2.7.2

This is a patch release containing the following changes to v2.7.1:

• Fixed segfaults in deconvolution backpropagation with ACL on AArch64-based processors (f02e6f3)
• Fixed code generation issues in Intel AVX2 convolution implementation (2ba2523, b60633f, 844326b, 2009164)
• Fixed correcteness issues and runtime errors in deconvolution with binary post-ops on Intel GPUs (dd54d39)
• Improved performance of convolutions with small number of channels and large spatial sizes on systems with Intel AMX (26f97dc, 4cb648d)
• Fixed runtime error in int8 convolutions with groups on Xe architecture based GPUs (e5a70f4)
• Improved inner product weight gradient performance on Xe architecture based GPUs (9e9b859, 12ec4e3)
• Improved batch normalization performance with threadpool threading (4fd5ab2)
• Improved inner product performance with binary post-ops in broadcast mode on Intel CPUs (d43c70d, 49ca4e1)
• Fixed segfaults and correctness issues in sum primitive with threadpool threading (ee7a321)
• Extended persistent cache API to cover engine objects (58481d6, 5f69dad, 16c0a95, 068071b)
• Added support for newer versions of Intel GPU drivers (7144393)
• Updated ITT API version to 3.23.0 (d23cc95)
• Fixed convolution correctness issue on Intel Data Center GPU Flex Series (365ac20)
• Fixed fp64 convolution correctness issue on Intel Data Center GPU MAX Series (9d4bf94, 6705403)
• Fixed correctness issues in reduction primitive with binary post-op on Intel GPUs (ae9d075, e3b80c5)
• Improved convolution performance on on Intel Data Center GPU MAX Series (90be8d5, caf4863)
• Fixed build errors with ONEDNN_ENABLE_PRIMITIVE_GPU_ISA build option (de2db04)
• Fixed correctness issues in convolution with per-tensor binary post-ops on Intel CPUs (9cf9c18)
• Improved convolution performance on Intel Data Center GPU Flex Series (8b08a07)