Build failing on tip : "./build.sh --subsetCategory CoreCLR -c Release -arch x64"

[sdmaclea]

  error: Could not read profile /home/stmaclea/.nuget/packages/optimization.linux-x64.pgo.coreclr/99.99.99-master-20200228.3/data/clrjit.profdata: Unsupported instrumentation profile format version
  [ 75%] Building CXX object src/utilcode/CMakeFiles/utilcode_dac.dir/configuration.cpp.o
  [ 76%] Building CXX object src/utilcode/CMakeFiles/utilcode_dac.dir/collections.cpp.o
  1 error generated.
  make[2]: *** [src/jit/standalone/CMakeFiles/clrjit.dir/__/alloc.cpp.o] Error 1
  make[1]: *** [src/jit/standalone/CMakeFiles/clrjit.dir/all] Error 2
  src/jit/standalone/CMakeFiles/clrjit.dir/build.make:79: recipe for target 'src/jit/standalone/CMakeFiles/clrjit.dir/__/alloc.cpp.o' failed
  CMakeFiles/Makefile2:2423: recipe for target 'src/jit/standalone/CMakeFiles/clrjit.dir/all' failed
  make[1]: *** Waiting for unfinished jobs....

[janvorli]

I have seen this problem in the past and in my case, doing a clean build (git clean -xdf executed in the root of the repo and then rebuild of everything) fixed the problem. Maybe I had to delete the .nuget folder too.

[sdmaclea]

I did try ./build.sh --clean which didn't work resolve the issue.

Perhaps --clean also needs to be fixed.

[hoyosjs]

Clean hasn't worked in a while or been supported I believe, at least in coreclr. I am not sure why it wasn't removed.

[ViktorHofer]

FWIW clean is working fine but it just cleans the artifacts folder.

[ViktorHofer]

I'm talking about .\build.cmd/sh -clean: https://github.com/dotnet/runtime/blob/master/eng/common/build.ps1#L123-L129

[hoyosjs]

The old coreclr clean used to do a bit more, like cleaning the package cache as it used to be directory based. we decided to defer the logic to nuget/dotnet nuget

[sdmaclea]

It seems strange that we are using caches outside of the build directory i.e.~/.nuget. It seems to make the build less robust. Perhaps CI doesn't do this?

[ViktorHofer]

The old coreclr clean used to do a bit more, like cleaning the package cache as it used to be directory based.

Clearing the nuget cache should be done via either dotnet nuget or nuget. There's no need to replicate that logic in dotnet/runtime: dotnet nuget locals -c global-packages. Do you have specific suggestion how to improve the current -clean target?

Perhaps CI doesn't do this?

Right, CI doesn't use the user wide cache but instead puts all packages under runtime/.packages.

[hoyosjs]

Indeed, ci generates a local .packages folder. Ah, Viktor got to it first :)

Do you have specific suggestion how to improve the current -clean target?

I honestly believe that what we are doing is the right thing. In this case it even feels like deleting the cached version of the optimization package is enough.

[sdmaclea]

I honestly believe that what we are doing is the right thing. In this case it even feels like deleting the cached version of the optimization package is enough.

Why?

I see several things we are doing wrong:

• The build is not reproducible.
• The build depending on global machine state.
• The dependency is undocumented.
• The --clean operation doesn't clean everything and there is no --deep-clean option
• The dev build is different from the CI build, The dev experience is effectively untested by CI.

[sdmaclea]

I retried the build like this and it still failed.

git clean -xdf
rm -rf ~/.nuget
./build.sh --subsetCategory CoreCLR -c Release -arch x64

With the same error.

[jashook]

Did we ever get to the root cause?

[sdmaclea]

No I never figured out root cause. I just reverted to an older previously working commit and forgot about it.

When I later updated to tip, the issue was resolved.

I don't think I could repro this anymore... Tempted to close because it cannot be reproed.

[ViktorHofer]

Ok let's close this then. Feel free to reopen.

[trylek]

I'm hitting this to this day when trying to build CoreCLR on Ubuntu in release mode; last time I hit this literally 2 minutes back. I'll be happy to assist any follow-up investigation by collecting artifacts, logs or anything else. @davidwrighton / @janvorli, have you got any suggestions as to how to continue the investigation? Is there an infra problem like pulling down a too old pgo data package or is it a compiler problem, what component consumes the optimization data on Linux, do we need to install a newer clang version, cmake or something else? Does there exist any tooling that would let me dump the pgo data and share its properties?

Thanks a lot

Tomas

[trylek]

  [ 42%] Building CXX object jit/CMakeFiles/clrjit.dir/alloc.cpp.o
  error: Could not read profile /home/trylek/.nuget/packages/optimization.linux-x64.pgo.coreclr/1.0.0-prerelease.21451.4/data/coreclr.profdata: Unsupported instrumentation profile format version

[akoeplinger]

I'm hitting this in #61001 as well, had to workaround by passing /p:NativeOptimizationDataSupported=false to build.sh

[davidwrighton]

I'll take a look at fixing the code flow here

[davidwrighton]

But my guess is that you have a version of clang that isn't able to consume the profile data we produce. Are you building with the same version of clang that is used in the offical docker build images?

[akoeplinger]

In #61001 I was updateing from Ubuntu 16.04 to 18.04 so it's possible that there is a different clang version being used.

[lambdageek]

Hitting this on Debian trixie/sid while trying to build release/9.0-rc2 with ./build.sh clr+libs -c Release :

...
  error: Error in reading profile /home/aleksey/.nuget/packages/optimization.linux-x64.pgo.coreclr/1.0.0-prerelease.24409.2/data/coreclr.profdata: unsupported instrumentation profile format version
...
$ clang --version
Debian clang version 16.0.6 (27+b1)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

[am11]

$ clang --version

the relevant version shows up in cmake config logs. Do you see 16 in those logs too?