[Clang][SYCL] Introduce clang-sycl-linker to link SYCL offloading dev…

…ice code (Part 1 of many) (llvm#112245)

This PR is one of the many PRs in the SYCL upstreaming effort focusing
on device code linking during the SYCL offload compilation process. RFC:
https://discourse.llvm.org/t/rfc-offloading-design-for-sycl-offload-kind-and-spir-targets/74088

In this PR, we introduce a new tool that will be used to perform device
code linking for SYCL offload kind. It accepts SYCL device objects in
LLVM IR bitcode format and will generate a fully linked device object
that can then be wrapped and linked into the host object.

A primary use case for this tool is to perform device code linking for
objects with SYCL offload kind inside the clang-linker-wrapper. It can
also be invoked via clang driver as follows:

`clang --target=spirv64 --sycl-link input.bc`

Device code linking for SYCL offloading kind has a number of known
quirks that makes it difficult to use in a unified offloading setting.
Two of the primary issues are:
1. Several finalization steps are required to be run on the fully-linked
LLVM IR bitcode to gaurantee conformance to SYCL standards. This step is
unique to SYCL offloading compilation flow.
2. SPIR-V LLVM Translator tool is an extenal tool and hence SPIR-V IR
code generation cannot be done as part of LTO. This limitation will be
lifted once SPIR-V backend is available as a viable LLVM backend.

Hence, we introduce this new tool to provide a clean wrapper to perform
SYCL device linking.

Co-Author: Michael Toguchi
Thanks

---------

Signed-off-by: Arvind Sudarsanam <arvind.sudarsanam@intel.com>

clang/docs/ClangSYCLLinker.rst

@@ -0,0 +1,82 @@
+=======================
+Clang SYCL Linker
+=======================
+
+.. contents::
+   :local:
+
+.. _clang-sycl-linker:
+
+Introduction
+============
+
+This tool works as a wrapper around the SYCL device code linking process.
+The purpose of this tool is to provide an interface to link SYCL device bitcode
+in LLVM IR format, SYCL device bitcode in SPIR-V IR format, and native binary
+objects, and then use the SPIR-V LLVM Translator tool on fully linked device
+objects to produce the final output.
+After the linking stage, the fully linked device code in LLVM IR format may
+undergo several SYCL-specific finalization steps before the SPIR-V code
+generation step.
+The tool will also support the Ahead-Of-Time (AOT) compilation flow. AOT
+compilation is the process of invoking the back-end at compile time to produce
+the final binary, as opposed to just-in-time (JIT) compilation when final code
+generation is deferred until application runtime.
+
+Device code linking for SYCL offloading has several known quirks that
+make it difficult to use in a unified offloading setting. Two of the primary
+issues are:
+1. Several finalization steps are required to be run on the fully linked LLVM
+IR bitcode to guarantee conformance to SYCL standards. This step is unique to
+the SYCL offloading compilation flow.
+2. The SPIR-V LLVM Translator tool is an external tool and hence SPIR-V IR code
+generation cannot be done as part of LTO. This limitation can be lifted once
+the SPIR-V backend is available as a viable LLVM backend.
+
+This tool has been proposed to work around these issues.
+
+Usage
+=====
+
+This tool can be used with the following options. Several of these options will
+be passed down to downstream tools like 'llvm-link', 'llvm-spirv', etc.
+
+.. code-block:: console
+
+  OVERVIEW: A utility that wraps around the SYCL device code linking process.
+  This enables linking and code generation for SPIR-V JIT targets and AOT
+  targets.
+
+  USAGE: clang-sycl-linker [options]
+
+  OPTIONS:
+    --arch <value>                Specify the name of the target architecture.
+    --dry-run                     Print generated commands without running.
+    -g                            Specify that this was a debug compile.
+    -help-hidden                  Display all available options
+    -help                         Display available options (--help-hidden for more)
+    --library-path=<dir>          Set the library path for SYCL device libraries
+    --device-libs=<value>         A comma separated list of device libraries that are linked during the device link
+    -o <path>                     Path to file to write output
+    --save-temps                  Save intermediate results
+    --triple <value>              Specify the target triple.
+    --version                     Display the version number and exit
+    -v                            Print verbose information
+    -spirv-dump-device-code=<dir> Directory to dump SPIR-V IR code into
+    -is-windows-msvc-env          Specify if we are compiling under windows environment
+    -llvm-spirv-options=<value>   Pass options to llvm-spirv tool
+    --llvm-spirv-path=<dir>       Set the system llvm-spirv path
+
+Example
+=======
+
+This tool is intended to be invoked when targeting any of the target offloading
+toolchains. When the --sycl-link option is passed to the clang driver, the
+driver will invoke the linking job of the target offloading toolchain, which in
+turn will invoke this tool. This tool can be used to create one or more fully
+linked device images that are ready to be wrapped and linked with host code to
+generate the final executable.
+
+.. code-block:: console
+
+  clang-sycl-linker --triple spirv64 --arch native input.bc

clang/docs/index.rst

@@ -98,6 +98,7 @@ Using Clang Tools
    ClangOffloadBundler
    ClangOffloadPackager
    ClangRepl
+   ClangSYCLLinker
 
 Design Documents
 ================

clang/include/clang/Driver/Options.td

@@ -6775,7 +6775,10 @@ def fsycl : Flag<["-"], "fsycl">,
 def fno_sycl : Flag<["-"], "fno-sycl">,
   Visibility<[ClangOption, CLOption]>,
   Group<sycl_Group>, HelpText<"Disables SYCL kernels compilation for device">;
-
+def sycl_link : Flag<["--"], "sycl-link">, Flags<[HelpHidden]>,
+  Visibility<[ClangOption, CLOption]>,
+  Group<sycl_Group>, HelpText<"Perform link through clang-sycl-linker via the target "
+  "offloading toolchain.">;
 // OS-specific options
 let Flags = [TargetSpecific] in {
 defm android_pad_segment : BooleanFFlag<"android-pad-segment">, Group<f_Group>;

clang/lib/Driver/Driver.cpp

@@ -4791,6 +4791,11 @@ Action *Driver::ConstructPhaseAction(
   if (Phase == phases::Assemble && Input->getType() != types::TY_PP_Asm)
     return Input;
 
+  // Use of --sycl-link will only allow for the link phase to occur. This is
+  // for all input files.
+  if (Args.hasArg(options::OPT_sycl_link) && Phase != phases::Link)
+    return Input;
+
   // Build the appropriate action.
   switch (Phase) {
   case phases::Link:

clang/lib/Driver/ToolChains/SPIRV.cpp

@@ -95,7 +95,21 @@ void SPIRV::Linker::ConstructJob(Compilation &C, const JobAction &JA,
   CmdArgs.push_back("-o");
   CmdArgs.push_back(Output.getFilename());
 
+  // Use of --sycl-link will call the clang-sycl-linker instead of
+  // the default linker (spirv-link).
+  if (Args.hasArg(options::OPT_sycl_link))
+    Linker = ToolChain.GetProgramPath("clang-sycl-linker");
   C.addCommand(std::make_unique<Command>(JA, *this, ResponseFileSupport::None(),
                                          Args.MakeArgString(Linker), CmdArgs,
                                          Inputs, Output));
 }
+
+SPIRVToolChain::SPIRVToolChain(const Driver &D, const llvm::Triple &Triple,
+                               const ArgList &Args)
+    : ToolChain(D, Triple, Args) {
+  // TODO: Revisit need/use of --sycl-link option once SYCL toolchain is
+  // available and SYCL linking support is moved there.
+  NativeLLVMSupport = Args.hasArg(options::OPT_sycl_link);
+}
+
+bool SPIRVToolChain::HasNativeLLVMSupport() const { return NativeLLVMSupport; }

clang/lib/Driver/ToolChains/SPIRV.h

@@ -57,8 +57,7 @@ class LLVM_LIBRARY_VISIBILITY SPIRVToolChain final : public ToolChain {
 
 public:
   SPIRVToolChain(const Driver &D, const llvm::Triple &Triple,
-                 const llvm::opt::ArgList &Args)
-      : ToolChain(D, Triple, Args) {}
+                 const llvm::opt::ArgList &Args);
 
   bool useIntegratedAs() const override { return true; }
 
@@ -72,6 +71,7 @@ class LLVM_LIBRARY_VISIBILITY SPIRVToolChain final : public ToolChain {
   }
   bool isPICDefaultForced() const override { return false; }
   bool SupportsProfiling() const override { return false; }
+  bool HasNativeLLVMSupport() const override;
 
   clang::driver::Tool *SelectTool(const JobAction &JA) const override;
 
@@ -81,6 +81,7 @@ class LLVM_LIBRARY_VISIBILITY SPIRVToolChain final : public ToolChain {
 
 private:
   clang::driver::Tool *getTranslator() const;
+  bool NativeLLVMSupport;
 };
 
 } // namespace toolchains

clang/test/CMakeLists.txt

@@ -80,6 +80,7 @@ list(APPEND CLANG_TEST_DEPS
   clang-nvlink-wrapper
   clang-offload-bundler
   clang-offload-packager
+  clang-sycl-linker
   diagtool
   hmaptool
   )

clang/test/Driver/clang-sycl-linker-test.cpp

@@ -0,0 +1,48 @@
+// Tests the clang-sycl-linker tool.
+//
+// Test a simple case without arguments.
+// RUN: %clangxx -emit-llvm -c %s -o %t_1.bc
+// RUN: %clangxx -emit-llvm -c %s -o %t_2.bc
+// RUN: clang-sycl-linker --dry-run -triple spirv64 %t_1.bc %t_2.bc -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=SIMPLE
+// SIMPLE: "{{.*}}llvm-link{{.*}}" {{.*}}.bc {{.*}}.bc -o [[FIRSTLLVMLINKOUT:.*]].bc --suppress-warnings
+// SIMPLE-NEXT: "{{.*}}llvm-spirv{{.*}}" {{.*}}-o a.spv [[FIRSTLLVMLINKOUT]].bc
+//
+// Test that llvm-link is not called when only one input is present.
+// RUN: clang-sycl-linker --dry-run -triple spirv64 %t_1.bc -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=SIMPLE-NO-LINK
+// SIMPLE-NO-LINK: "{{.*}}llvm-spirv{{.*}}" {{.*}}-o a.spv {{.*}}.bc
+//
+// Test a simple case with device library files specified.
+// RUN: touch %T/lib1.bc
+// RUN: touch %T/lib2.bc
+// RUN: clang-sycl-linker --dry-run -triple spirv64 %t_1.bc %t_2.bc --library-path=%T --device-libs=lib1.bc,lib2.bc -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=DEVLIBS
+// DEVLIBS: "{{.*}}llvm-link{{.*}}" {{.*}}.bc {{.*}}.bc -o [[FIRSTLLVMLINKOUT:.*]].bc --suppress-warnings
+// DEVLIBS-NEXT: "{{.*}}llvm-link{{.*}}" -only-needed [[FIRSTLLVMLINKOUT]].bc {{.*}}lib1.bc {{.*}}lib2.bc -o [[SECONDLLVMLINKOUT:.*]].bc --suppress-warnings
+// DEVLIBS-NEXT: "{{.*}}llvm-spirv{{.*}}" {{.*}}-o a.spv [[SECONDLLVMLINKOUT]].bc
+//
+// Test a simple case with .o (fat object) as input.
+// TODO: Remove this test once fat object support is added.
+// RUN: %clangxx -c %s -o %t.o
+// RUN: not clang-sycl-linker --dry-run -triple spirv64 %t.o -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=FILETYPEERROR
+// FILETYPEERROR: Unsupported file type
+//
+// Test to see if device library related errors are emitted.
+// RUN: not clang-sycl-linker --dry-run -triple spirv64 %t_1.bc %t_2.bc --library-path=%T --device-libs= -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=DEVLIBSERR1
+// DEVLIBSERR1: Number of device library files cannot be zero
+// RUN: not clang-sycl-linker --dry-run -triple spirv64 %t_1.bc %t_2.bc --library-path=%T --device-libs=lib1.bc,lib2.bc,lib3.bc -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=DEVLIBSERR2
+// DEVLIBSERR2: '{{.*}}lib3.bc' SYCL device library file is not found
+//
+// Test if correct set of llvm-spirv options are emitted for windows environment.
+// RUN: clang-sycl-linker --dry-run -triple spirv64 --is-windows-msvc-env %t_1.bc %t_2.bc -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=LLVMOPTSWIN
+// LLVMOPTSWIN: -spirv-debug-info-version=ocl-100 -spirv-allow-extra-diexpressions -spirv-allow-unknown-intrinsics=llvm.genx. -spirv-ext=
+//
+// Test if correct set of llvm-spirv options are emitted for linux environment.
+// RUN: clang-sycl-linker --dry-run -triple spirv64  %t_1.bc %t_2.bc -o a.spv 2>&1 \
+// RUN:   | FileCheck %s --check-prefix=LLVMOPTSLIN
+// LLVMOPTSLIN: -spirv-debug-info-version=nonsemantic-shader-200 -spirv-allow-unknown-intrinsics=llvm.genx. -spirv-ext=

clang/test/Driver/sycl-link-spirv-target.cpp

@@ -0,0 +1,9 @@
+// Tests the driver when linking LLVM IR bitcode files and targeting SPIR-V
+// architecture.
+//
+// Test that -Xlinker options are being passed to clang-sycl-linker.
+// RUN: touch %t.bc
+// RUN: %clangxx -### --target=spirv64 --sycl-link -Xlinker --llvm-spirv-path=/tmp \
+// RUN:   -Xlinker --library-path=/tmp -Xlinker --device-libs=lib1.bc,lib2.bc %t.bc 2>&1 \
+// RUN:   | FileCheck %s -check-prefix=XLINKEROPTS
+// XLINKEROPTS: "{{.*}}clang-sycl-linker{{.*}}" "--llvm-spirv-path=/tmp" "--library-path=/tmp" "--device-libs=lib1.bc,lib2.bc" "{{.*}}.bc" "-o" "a.out"

clang/test/lit.cfg.py

@@ -96,6 +96,7 @@
     "yaml2obj",
     "clang-linker-wrapper",
     "clang-nvlink-wrapper",
+    "clang-sycl-linker",
     "llvm-lto",
     "llvm-lto2",
     "llvm-profdata",

clang/tools/CMakeLists.txt

@@ -12,6 +12,7 @@ add_clang_subdirectory(clang-nvlink-wrapper)
 add_clang_subdirectory(clang-offload-packager)
 add_clang_subdirectory(clang-offload-bundler)
 add_clang_subdirectory(clang-scan-deps)
+add_clang_subdirectory(clang-sycl-linker)
 add_clang_subdirectory(clang-installapi)
 if(HAVE_CLANG_REPL_SUPPORT)
   add_clang_subdirectory(clang-repl)

clang/tools/clang-sycl-linker/CMakeLists.txt

@@ -0,0 +1,28 @@
+set(LLVM_LINK_COMPONENTS
+  ${LLVM_TARGETS_TO_BUILD}
+  Option
+  )
+
+set(LLVM_TARGET_DEFINITIONS SYCLLinkOpts.td)
+tablegen(LLVM SYCLLinkOpts.inc -gen-opt-parser-defs)
+add_public_tablegen_target(SYCLLinkerOpts)
+
+if(NOT CLANG_BUILT_STANDALONE)
+  set(tablegen_deps intrinsics_gen SYCLLinkerOpts)
+endif()
+
+add_clang_tool(clang-sycl-linker
+  ClangSYCLLinker.cpp
+
+  DEPENDS
+  ${tablegen_deps}
+  )
+
+set(CLANG_SYCL_LINKER_LIB_DEPS
+  clangBasic
+  )
+
+target_link_libraries(clang-sycl-linker
+  PRIVATE
+  ${CLANG_SYCL_LINKER_LIB_DEPS}
+  )