Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running --minimal smoke-tests in Ubuntu 22.04 #2824

Closed
mirespace opened this issue Apr 7, 2022 · 3 comments
Closed

Running --minimal smoke-tests in Ubuntu 22.04 #2824

mirespace opened this issue Apr 7, 2022 · 3 comments
Labels
area-testing Improvements in CI and testing

Comments

@mirespace
Copy link

mirespace commented Apr 7, 2022

Hi, I'm trying to use the smoke tests in https://github.com/dotnet/installer/tree/release/6.0.1xx/src/SourceBuild/tarball/content/test/Microsoft.DotNet.SourceBuild.SmokeTests as building tests for dotnet packaging in Ubuntu 22.04 using tag 6.0.103-source.build. I know this is kind of legacy as per #1654, but by now is what I'm executing.

I got some problems executing it from the building. First was the smoke-test.sh file, I had to correct the artifacts folder:

--- a/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh
+++ b/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh
@@ -7,7 +7,7 @@ VERSION_PREFIX=6.0
 # See https://github.com/dotnet/source-build/issues/579, this version
 # needs to be compatible with the runtime produced from source-build
 DEV_CERTS_VERSION_DEFAULT=6.0.0-preview.6.21355.2
-ARTIFACTS_DIR="$SCRIPT_ROOT/../../../../../../../artifacts/"
+ARTIFACTS_DIR="$SCRIPT_ROOT/../../../../artifacts/"
 executingUserHome=${HOME:-}
 
 export DOTNET_CLI_TELEMETRY_OPTOUT=1

After that, If I execute manually it (./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64) is all OK -log attached all-tests.log - (except for me because of the part where it's downloading stuff to be able to run the tests).

So, let's go to the problems:

  1. Trying the options minimal, excluding omnisharptest and onlinetest into smoke-test.sh to see if it can only test local stuff... It doesn't. Executing ./assets/smoke-tests/smoke-test.sh --dotnetDir {Directory.GetParent(DotNetHelper.DotNetPath)} --projectOutput --archiveRestoredPackages --targetRid {Config.TargetRid} --minimal --excludeOnlineTests --excludeOmniSharpTests all resolved is ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests :
[...]
build/dotnet6-KxJTNR/dotnet6-6.0.103/t

est/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:PublishTrimmed=true -property:PublishReadyToRun=true -property:SelfContained=True -property:
_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=linux-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/logs/CSharp_classlib_local_nohttps_publish-self-contained-R2R-portable.binlog ./CSharp_classlib.csproj
Determining projects to restore...
Restored /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/CSharp_classlib.csproj (in 120 ms).
CSharp_classlib -> /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/bin/Debug/net6.0/linux-x64/CSharp_classlib.dll
Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
CSharp_classlib -> /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/bin/Debug/net6.0/linux-x64/publish/
finished language C#, type classlib
starting language C#, type xunit
running new
The template "xUnit Test Project" was created successfully.
running restore
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -nologo -maxcpucount -target:Restore -verbosity:m /bl:/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/logs/CSharp_xunit_local_nohttps_restore.binlog ./CSharp_xunit.csproj
Determining projects to restore...
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package Microsoft.NET.Test.Sdk. No packages exist with this id in source(s): source-built-packages
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package xunit. No packages exist with this id in source(s): source-built-packages
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package xunit.runner.visualstudio. No packages exist with this id in source(s): source-built-packages
/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package coverlet.collector. No packages exist with this id in source(s): source-built-packages
Failed to restore /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj (in 169 ms).
make[1]: *** [debian/rules:132: override_dh_install] Error 1
make[1]: Leaving directory '/build/dotnet6-KxJTNR/dotnet6-6.0.103'
make: *** [debian/rules:30: binary] Error 2

Therefore I can't use this as a workaround for the next point @MichaelSimons :( (although I'm working on how to download this stuff while creating the tarball... no good progress on this yet).

  1. Executing the VERBOSE=1 ./build.sh --run-smoke-test allowing to download requisites (as we see it worked manually previously we suppose it has to work... it doesn't):
VERBOSE=1 ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.102, bootstrap Arcade 6.0.0-beta.21609.4, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-aZycfo/dotnet6-6.0.103/.dotnet/sdk/6.0.102/MSBuild.dll /bl:/build/dotnet6-aZycfo/dotnet6-6.0.103/artifacts/log/Debug/BuildTests_0330135715.binlog /clp:v=m /fileLoggerParameters:LogFile=/build/dotnet6-aZycfo/dotnet6-6.0.103/artifacts/logs/BuildTests_0330135715.log /flp:v=detailed /t:RunSmokeTest /build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  
  Welcome to .NET 6.0!
  ---------------------
  SDK Version: 6.0.102
  
  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  Learn about HTTPS: https://aka.ms/dotnet-https
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Determining projects to restore...
    Restored /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj (in 5 sec).
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Microsoft.DotNet.SourceBuild.SmokeTests -> /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll
  Test run for /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (.NETCoreApp,Version=v6.0)
  Microsoft (R) Test Execution Command Line Tool Version 17.0.0
  Copyright (c) Microsoft Corporation.  All rights reserved.
  
  Starting test execution, please wait...
  A total of 1 test files matched the specified pattern.
  [xUnit.net 00:39:36.88]     Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [FAIL]
    Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [39 m 36 s]
EXEC : error Message:  [/build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj]
     Assert.Equal() Failure
  Expected: 0
  Actual:   127
    Stack Trace:
       at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 43
    Standard Output Messages:
   Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64
  
  
  Results File: /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-03-30_13_57_28.trx
  
  Failed!  - Failed:     1, Passed:     3, Skipped:     0, Total:     4, Duration: 39 m 33 s - /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)
/build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-aZycfo/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1.
make[1]: *** [debian/rules:129: override_dh_install] Error 1
make[1]: Leaving directory '/build/dotnet6-aZycfo/dotnet6-6.0.103'
make: *** [debian/rules:30: binary] Error 2

How can it be possible if the command stated here Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 worked manually before ? I put some prints out in the SmokeTest.Cs file for executeResult.StdOut, executeResult.StdErr and executeResult.ExitCode (to see where the 1 code comes from) and got the following:

testing-smoke-0406114624/CSharp_console/bin/Debug/net6.0/publish/
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=True -property:_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=ubuntu.22.04-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/testing-smoke-0406114624/logs/CSharp_console_local_nohttps_publish-self-contained-ubuntu.22.04-x64.binlog ./CSharp_console.csproj
Determining projects to restore...
StdOut: /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/testing-smoke-0406114624/CSharp_console/CSharp_console.csproj] [/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj]
StdErr:
ExitCode: 1
Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [6 s]
EXEC : error Message:  [/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj]
Assert.Equal() Failure
Expected: 0
Actual:   1
Stack Trace:
at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 47
Standard Output Messages:
Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

Results File: /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-04-06_11_46_24.trx

Failed!  - Failed:     1, Passed:     3, Skipped:     0, Total:     4, Duration: 6 s - /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)
/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-OIjAuR/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1.
make[1]: *** [debian/rules:131: override_dh_install] Error 1
make[1]: Leaving directory '/build/dotnet6-OIjAuR/dotnet6-6.0.103'
make: *** [debian/rules:30: binary] Error 2

It seems another wrong artifact folder set elsewhere : /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. ... could it be a possibility?

Thanks in advance for looking into this.

@MichaelSimons MichaelSimons transferred this issue from dotnet/installer Apr 11, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added area-testing Improvements in CI and testing untriaged labels Apr 11, 2022
@MichaelSimons
Copy link
Member

@mirespace, I have a couple suggestions for the two different attempts you tried.

  1. Can you provide clarity on what path you are executing ./assets/smoke-tests/smoke-test.sh from? It appears that you maybe invoking the script directly out of the source directory. This isn't support as you are discovering. Try invoking it out of the build output directory /test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/smoke-test.sh. When doing so, revert your change to the artifacts variable.

    That said, I don't suspect this is the cause of the failure. Can checked to see if the packages being loaded exist in /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/../../../../artifacts/obj/x64/Release/blob-feed/packages/? From the log, this is where it is looking for the Microsoft.NET.Test.Sdk, xunit, xunit.runner.visualstudio, and coverlet.collector packages

  2. In the /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release directory there should be a testing-smoke*/logs directory for each run. Can you share the logs for the a run the demonstrates the problem?

    When attempting to run the tests via the build.sh script, did you revert you changes to modify the artifacts variable.

@mirespace
Copy link
Author

mirespace commented Apr 13, 2022

Hi @MichaelSimons! I run all the process from the scratch again. To force the parameters in the .sh script, I made this patch:

diff --git a/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs b/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs
index f6a1d96b4..2c732a64b 100644
--- a/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs
+++ b/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs
@@ -24,7 +24,7 @@ public SmokeTests(ITestOutputHelper outputHelper)
     [Fact]
     public void SmokeTestsScript()
     {
-        string smokeTestArgs = $"--dotnetDir {Directory.GetParent(DotNetHelper.DotNetPath)} --projectOutput --archiveRestoredPackages --targetRid {Config.TargetRid}";
+        string smokeTestArgs = $"--dotnetDir {Directory.GetParent(DotNetHelper.DotNetPath)} --projectOutput --archiveRestoredPackages --targetRid {Config.TargetRid} --minimal --excludeOnlineTests --excludeOmniSharpTests";
         if (Config.TargetRid.Contains("osx"))
         {
             smokeTestArgs += " --excludeWebHttpsTests";

  1. The complete command is <<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh --dotnetDir /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests: is executing from the top of the source tree building source (dotnet6-random as temporal folder for building):
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

I didn't realize when open the issue that it fails because the bin folder is not present:

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh: line 485: /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/dotnet: No such file or directory

because it would be done by ./build.sh --run-smoke-test (I guess this is a step before invoking the smoke-test.sh script).

so, directly, it can't be called.

  1. I reverted the change to the artifacts variable this time, and I executed from the top of the source tree the build command:
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# VERBOSE=1 ./build.sh --run-smoke-test

It failed a first time for the xUnit.net test:

(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# VERBOSE=1 ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.102, bootstrap Arcade 6.0.0-beta.21609.4, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/.dotnet/sdk/6.0.102/MSBuild.dll /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/artifacts/log/Debug/BuildTests_0413105044.binlog /clp:v=m /fileLoggerParameters:LogFile=/build/dotnet6-4woFhD/dotnet6-6.0.103/artifacts/logs/BuildTests_0413105044.log /flp:v=detailed /t:RunSmokeTest /build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  
  Welcome to .NET 6.0!
  ---------------------
  SDK Version: 6.0.102
  
  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  Learn about HTTPS: https://aka.ms/dotnet-https
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Determining projects to restore...
    Restored /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj (in 4.2 sec).
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Microsoft.DotNet.SourceBuild.SmokeTests -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll
  Test run for /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (.NETCoreApp,Version=v6.0)
  Microsoft (R) Test Execution Command Line Tool Version 17.0.0
  Copyright (c) Microsoft Corporation.  All rights reserved.
  
  Starting test execution, please wait...
  A total of 1 test files matched the specified pattern.
  [xUnit.net 00:00:40.08]     Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [FAIL]
    Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [39 s]
EXEC : error Message:  [/build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj]
     Assert.Equal() Failure
  Expected: 0
  Actual:   1
    Stack Trace:
       at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 43
    Standard Output Messages:
   Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests
  
  
  Results File: /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-04-13_10_50_56.trx
  
  Failed!  - Failed:     1, Passed:     3, Skipped:     0, Total:     4, Duration: 36 s - /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)
/build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-4woFhD/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1.

And, no, the Microsoft.NET.Test.Sdk, xunit, xunit.runner.visualstudio, and coverlet.collector packages are not in /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/../../../../../../../artifacts/obj/x64/Release/blob-feed/packages/. So, yes, that's the reason of failing, but... if the parameters are --minimal and --excludeOnlineTests everything needed to run the test shouldn't already be there and not need to be downloaded?

  1. The only error I see in that files is related with the artifacts folder location:
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests# ack error testing-smoke-041*/logs/*.log
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj]
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests# 

complete file is (indeed, only this log has been generated):

(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests# cat testing-smoke-0413110747/logs/smoke-test.log
    running new

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.103

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
The template "Console App" was created successfully.
    new succeeded
    running restore
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -nologo -maxcpucount -target:Restore -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_restore.binlog ./CSharp_console.csproj
  Determining projects to restore...
  Restored /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj (in 71 ms).
    restore succeeded
    running build
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -consoleloggerparameters:Summary -maxcpucount -restore -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_build.binlog ./CSharp_console.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/CSharp_console.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.02
    build succeeded
    running run
Hello, World!
    run succeeded
    running multi-rid-publish
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=False -property:_CommandLineDefinedSelfContained=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_publish-fx-dep.binlog ./CSharp_console.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/CSharp_console.dll
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/publish/
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=True -property:_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=ubuntu.22.04-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_publish-self-contained-ubuntu.22.04-x64.binlog ./CSharp_console.csproj
  Determining projects to restore...
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj]

@MichaelSimons
Copy link
Member

[Triage] Closing as the recently changes to the smoke-tests removed the script. The ability to run tests in an offline mode is tracked with #2844.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area-testing Improvements in CI and testing
Projects
None yet
Development

No branches or pull requests

2 participants