Generated kokkos-kernels file names are too long and are crashing cloning Trilinos on Windows

[bartlettroscoe]

As reported in:

• Long file names in KokkosKernels have broken git clone of Trilinos on Windows trilinos/Trilinos#4571

the generated files names for explicit instantiations are too long. The clone of Trilinos on Windoes gives:

G:\raq\scale_dev\dev>git clone https://github.com/trilinos/Trilinos.git Trilinos
Cloning into 'Trilinos'...
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 957120 (delta 17), reused 14 (delta 7), pack-reused 957073
Receiving objects: 100% (957120/957120), 562.05 MiB | 26.56 MiB/s, done.
Resolving deltas: 100% (771153/771153), done.
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_numeric/KokkosSparse_gauss_seidel_numeric_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp: Filename too long
Checking out files: 100% (56578/56578), done.
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

Can KokkosKernels instead consider compressing the names of these files? I can provide some suggestions on how to do that if that would be helpful.

[srajama1]

Oh come on ..... Does Kokkos work in Windows first ? We really don't have cycles to support windows or access to even a test machine where we can test it.

[nmhamster]

Aren't we required to support Trillinos on Windows for a customer use case?

[srajama1]

I think they use legacy Trilinos stack ...

@bartlettroscoe : My "Oh come on" was directed at the silliness of the error, not at the request to support windows.

[bartlettroscoe]

@srajama1, because we use CMake and C++11, it really is not that hard to support Windows. It is just silly stuff like this that causes a problem.

[jhux2]

@csiefer2

[jwillenbring]

@srajama1

Oh come on ..... Does Kokkos work in Windows first ? We really don't have cycles to support windows or access to even a test machine where we can test it.

I ran into this issue today too. We have at least one major customer who uses Windows. Right now they just use the legacy stack, that is true, but the really ugly part of this error is that your git clone fails, so it isn't just that one cannot use Kokkos Kernels.

[mhoemmen]

It's probably not hard to modify the script for generating filenames, so that it abbreviates types. You already need some kind of mangling for that anyway, to generate legit filenames without weird characters in them.

[srajama1]

Alright. we will find a workaround.

[akhilpotla]

Same issue on my Ubuntu 16.04 machine.
Running git checkout -f HEAD does not resolve the issue.

Scanning dependencies of target Trilinos
[ 11%] Creating directories for 'Trilinos'
[ 22%] Performing download step (git clone) for 'Trilinos'
Cloning into 'trilinos-src'...
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_apply/KokkosSparse_gauss_seidel_apply_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_numeric/KokkosSparse_gauss_seidel_numeric_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_numeric/KokkosSparse_gauss_seidel_numeric_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_numeric/KokkosSparse_gauss_seidel_numeric_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_numeric/KokkosSparse_gauss_seidel_numeric_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__int_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutLeft_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
error: unable to create file packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_float__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp (File name too long)
Checking out files: 100% (55829/55829), done.
fatal: unable to checkout working tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

[sethrj]

Any update on this? Our group would really like to be able to update to a newer Trilinos (for starters, one that can even build with the current version of tribits...) and since we need to be able to clone and build on Windows machines, this bug is blocking.

[sethrj]

Also is there something preventing these files from being generated at configure time? It seems to me like it might be better to have a CMake script do this based on the enabled configurations rather than pre-generating everything in a bash script and having CMake "glob" the needed files.

[srajama1]

@sethrj : There is a PR waiting to go in. @ndellingwood Is there any thing blocking it from merging it ?

[sethrj]

Awesome!

[bartlettroscoe]

FYI: Look like #424 might just have this fixed (just merged to separate KokkosKernels repo yesterday).

But we can't take advantage of this until Kokkos and KokkosKernels get updated (shapshotted) in Trilinos. Any idea when that might occur?

@sethrj, can someone try cloning the kokkos-kernels GitHub repo and see if this fixes the Windows clone problem?

[sethrj]

@bartlettroscoe It works, but confusingly, at least on the machine and in the directory I chose, checking out the fix's parent commit worked as well... as did the Trilinos master. For at least a0198b2beadaa821ebf856e1b15ccc2558b1b824, the largest file in the repository has length 233:

packages/kokkos-kernels/src/impl/generated_specializations_cpp/gauss_seidel_symbolic/KokkosSparse_gauss_seidel_symbolic_eti_spec_inst_Kokkos_complex_double__size_t_int64_t_LayoutRight_Cuda_CudaHostPinnedSpace_CudaHostPinnedSpace.cpp

but ironically I wasn't able to check out the latest TriBITS, which has a 250-character relative path:

test/ci_support/cdash_analyze_and_report/raw_cdash_data_twoif_12_twif_9/test_history/2018-10-28-mutrino-Trilinos-atdm-mutrino-intel-opt-openmp-KNL-Intrepid2_unit-test_Discretization_Basis_HCURL_TRI_In_FEM_Serial_Test_01_SLFadDouble_MPI_1-HIST-30.json

so perhaps an issue needs to be opened for that ;)

[bartlettroscoe]

so perhaps an issue needs to be opened for that ;)

I added TriBITSPub/TriBITS#297. That can be fixed. (Ironically, I had to add a feature to compress those file names using the generation of a hash in order to address some test names in one of our ATDM codes that is three times that long :-). Therefore, for the purpose of this test, I think I can just have to code shorten the file names in testing mode.) Also ironic is that snapshotted TriBITS into Trilinos does not contain this test directory.

@sethrj, how do you reproduce these clone failures on Windows? I always using git under cygwin so I never see problems like this.

[bartlettroscoe]

Has anyone verified that updated Kokkos Kernels in Trilinos will clone in Windows?

[ndellingwood]

@bartlettroscoe the changes haven't made it into Trilinos yet, I closed issues marked "InDevelop" to make the changelog for the release candidate that we will merge into Trilinos when we complete the promotion process after the holidays. Reopening this issue until confirmation is made post-promotion.

[sethrj]

Just to finalize this: I am now able to clone the latest Trilinos master on Windows. Yay!