Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: UseDotnetV2 - Fails silently to download #20508

Closed
6 of 7 tasks
JasonWhall opened this issue Oct 3, 2024 · 30 comments
Closed
6 of 7 tasks

[BUG]: UseDotnetV2 - Fails silently to download #20508

JasonWhall opened this issue Oct 3, 2024 · 30 comments
Assignees
Labels
Area: ABTT Akvelon Build Tasks Team area of work awaiting deployment Related changes are waiting for deployment to be completed bug Task: UseDotNet

Comments

@JasonWhall
Copy link
Contributor

JasonWhall commented Oct 3, 2024

New issue checklist

Task name

UseDotnet

Task version

2.246.1

Issue Description

It seems the task is silently stopping after attempting to download the zip containing a version of the .NET sdk and then continues onto the next step. I would expect this to either error and fail the task or to successfully log that it has downloaded the tool and extracted into the tool directory

The offending area of code that it seems to be exiting from is in the azure-pipelines-tool-lib repository noted here - https://github.com/microsoft/azure-pipelines-tool-lib/blob/41acd733ff90c8bbc02c42376f46800b794cdc99/tool.ts#L251

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server Latest/Ubuntu Latest

Relevant log output

Downloading: https://download.visualstudio.microsoft.com/download/pr/c1b355f8-c828-4d2d-a0f5-a0695834be68/2c7a7983c02bebffc071648658b33b73/dotnet-sdk-8.0.107-win-x64.zip
##[debug]destination C:\a\_temp\df4b4a33-ebde-4814-802e-b45b3bdd2674
##[debug]downloading
##[debug]Content-Length of downloaded file: 282187308
##[debug]creating stream
Finishing: Install .NET SDK


### Full task logs with system.debug enabled

<details>
  <pre> [REPLACE THIS WITH YOUR INFORMATION] </pre>
</details>


### Repro steps

_No response_
@BastienPerdriau
Copy link

BastienPerdriau commented Oct 3, 2024

We have the same issue here since this morning.

Was there a recent update to the task?
EDIT: I can see this update for this task on the release listed in the repo : #20446

@v-schhabra v-schhabra added Area: ABTT Akvelon Build Tasks Team area of work and removed Area: Release labels Oct 3, 2024
@AleksuKey
Copy link

We are having this issue since this morning too.

@double-sigma
Copy link

Same issue started today morning for our .NET builds on Self-Hosted agents.

Task         : Use .NET Core
Version      : 2.246.1

@DergachevE
Copy link
Contributor

Thanks for raising this issue to us.
The team will look into it and come with updates.

As per recent changes:

I can see this update for this task on the release listed in the repo : #20446
This PR introduces a new 2.246.2 task version which is not yet rolled out

@BastienPerdriau
Copy link

I rolled back to 2.243.1 which was working a few days / weeks ago and I have the same output of download not always waited without raising any error, so following tasks using dotnet are failing.
But it's a bit random, and when a task finally manage to download and extract the SDK, it can be reused on following jobs if there are on the same agent.

@Mjinx
Copy link

Mjinx commented Oct 3, 2024

We also been having the same issue since this morning,

Version : 2.246.1

Detected platform (Primary): linux-x64
Detected platform (Legacy): -x64
##[debug]Got download URL for platform with rid: linux-x64
##[debug]
##[debug]Agent.Version=3.245.0
##[debug]Agent.TempDirectory=/agent/_work/_temp
##[debug]testing directory '/agent/_work/_temp'
Downloading: https://download.visualstudio.microsoft.com/download/pr/1ebffeb0-f090-4001-9f13-69f112936a70/5dbc249b375cca13ec4d97d48ea93b28/dotnet-sdk-8.0.402-linux-x64.tar.gz
##[debug]destination /agent/_work/_temp/dbc4b586-698d-49d5-9658-8214325f95da
##[debug]downloading
##[debug]Content-Length of downloaded file: 211866121
##[debug]creating stream
##[debug]PERF: RetryHelper Method:System.Threading.Tasks.Task <RunAsyncInternal>b__9() : took 2045.1937 ms
##[debug]PERF WARNING: RetryHelper Method:System.Threading.Tasks.Task <RunAsyncInternal>b__9() : took 2045.1937 ms
Finishing: Install .NET SDK

@mbyt
Copy link

mbyt commented Oct 4, 2024

We have the same issue since yesterday:

Agent name: 'Hosted Agent'
Current agent version: '3.245.0'
Microsoft Windows Server 2022
Image: windows-2022
Version: 20240929.1.0
Downloading task: UseDotNet (2.246.1)

@JasonWhall it also happens / fails on MS Hosted, thus you could check also that box in the description:

  • Microsoft Hosted

@Carike
Copy link

Carike commented Oct 4, 2024

We have encountered the same issue on multiple pipelines and this is impacting our deliverables.

Could this please be escalated or priority elevated?

Thank you in advance

@AleksuKey
Copy link

Hi @DergachevE , could this issue be escalated? This is creating a huge impact in our dev teams unfortunately.

@tijlreynhout
Copy link

We seem to be having the same problem but it doesn't always happen for us. I have no idea why it does or doesn't work.

Agent name: 'Hosted Agent'
Current agent version: '3.245.0'
Operating System
Microsoft Windows Server 2022
10.0.20348
Datacenter
Runner Image
Image: windows-2022
Version: 20240929.1.0

The task downloads the zip file, then finishes without extracting/installing the sdk. The next step is restoring nuget packages but this fails because the sdk version is not found.

image

@Skypesilver
Copy link

It happens about 50% of the time for us. I see reverting to version 2.243.1 did not help a poster above. Perhaps reverting to an even earlier version would help?

@sicil1ano
Copy link

It happens about 50% of the time for us. I see reverting to version 2.243.1 did not help a poster above. Perhaps reverting to an even earlier version would help?

Unfortunately the issue happens even with previous versions. We rolled back to 2.238.1 and still same problem, build failing 50% of the time.
I guess we will just wait for a proper fix.

@double-sigma
Copy link

Today I saw this in the logs if this is of help:

Legacy:-x64
Detected platform (Primary): linux-x64
Detected platform (Legacy): -x64
Downloading: https://download.visualstudio.microsoft.com/download/pr/1ebffeb0-f090-4001-9f13-69f112936a70/5dbc249b375cca13ec4d97d48ea93b28/dotnet-sdk-8.0.402-linux-x64.tar.gz
##[warning]Could not download installation package from this URL: https://download.visualstudio.microsoft.com/download/pr/1ebffeb0-f090-4001-9f13-69f112936a70/5dbc249b375cca13ec4d97d48ea93b28/dotnet-sdk-8.0.402-linux-x64.tar.gz Error: Error: Aborted
    at IncomingMessage.<anonymous> (/workspace/_tasks/UseDotNet_b0ce7256-7898-45d3-9cb5-176b752bfea6/2.246.1/node_modules/azure-pipelines-tool-lib/tool.js:241:36)
    at IncomingMessage.emit (node:events:519:28)
    at IncomingMessage._destroy (node:_http_incoming:224:10)
    at _destroy (node:internal/streams/destroy:121:10)
    at IncomingMessage.destroy (node:internal/streams/destroy:83:5)
    at TLSSocket.socketCloseListener (node:_http_client:471:11)
    at TLSSocket.emit (node:events:531:35)
    at node:net:339:12
    at TCP.done (node:_tls_wrap:657:7)
Using fallback url for download: https://dotnetcli.azureedge.net/dotnet/Sdk/8.0.402/dotnet-sdk-8.0.402-linux-x64.tar.gz
Downloading: https://dotnetcli.azureedge.net/dotnet/Sdk/8.0.402/dotnet-sdk-8.0.402-linux-x64.tar.gz
##[warning]Content-Length (211866121 bytes) did not match downloaded file size (261578 bytes).
Extracting downloaded package /workspace/_temp/23a1b4eb-1e5b-44b5-b677-96f2d62dd70c.
Extracting archive
/usr/bin/tar xC /workspace/_temp/2c0b7ec4-1af0-4a21-a0be-a96d71c3f537 -f /workspace/_temp/23a1b4eb-1e5b-44b5-b677-96f2d62dd70c

Successfully installed .NET Core sdk version 8.0.402.

@double-sigma
Copy link

As a workaround if you use a custom image for you agent as we do (self-hosted agent pool on AKS), then you can just bake in the dotnet sdk:

ARG DOTNET_VERSION="8.0.402-1"
RUN wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb \
     -O packages-microsoft-prod.deb && dpkg -i packages-microsoft-prod.deb \
     && apt-get update && apt-get install -y --no-install-recommends dotnet-sdk-8.0=${DOTNET_VERSION}

I see that our builds where dotnet failed to download are now passing. Download failure rate is quite high, more than 50%.

@finkinfridom
Copy link

We started getting the above errors from 30 minutes ago. The issue now seems like happening 100% of the time.
Any suggestion to workaround this issue? maybe adding a custom script (similar to the one above) to be executed in the pipeline?

@KonstantinTyukalov
Copy link
Contributor

Hi folks, as a quick workaround we suggest using the following scripts before the UseDotnet step execution:

for windows:

- powershell: |
    # your target dotnet version
    $version = "8.0.107"

    # your target install path
    $sdkInstallationPath = Join-Path $env:AGENT_TOOLSDIRECTORY dotnet

    $scriptUrl = "https://dot.net/v1/dotnet-install.ps1"
    $scriptPath = Join-Path  $env:AGENT_TEMPDIRECTORY dotnet-install.ps1
    Invoke-WebRequest -Uri $scriptUrl -OutFile "$scriptPath"

    & $scriptPath -Version $version -InstallDir $sdkInstallationPath -Verbose

    "$version" > "${sdkInstallationPath}\sdk\${version}.complete"

  condition: eq(variables['Agent.OS'], 'Windows_NT')

for linux & macos:

- bash: |
      # your target dotnet version
      version="8.0.107"

      # your target install path
      installationPath="$(Agent.ToolsDirectory)/dotnet"

      script_path="$(Agent.TempDirectory)/dotnet-install.sh"
      curl -o $script_path "https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh"
      chmod +x $script_path

      $script_path --version $version --install-dir $installationPath --verbose

      echo "$version" > "${installationPath}/sdk/${version}.complete"

  condition: ne(variables['Agent.OS'], 'Windows_NT')

With these scripts, UseDotnet task will find downloaded dotnet version and skip archive fetching.

!!!! Please make sure version and installationPath variables in scripts are matching the UseDotnet task input values

@Vhab
Copy link

Vhab commented Oct 5, 2024

Thanks for raising this issue to us. The team will look into it and come with updates.

As per recent changes:

I can see this update for this task on the release listed in the repo : #20446
This PR introduces a new 2.246.2 task version which is not yet rolled out

Hi @DergachevE,

Thank you for recognizing this is an issue, but would it be possible to escalate the severity of this up a few notches?

We're currently heavily affected by this issue, and it's wrecking havoc across multiple pipelines.
And given the nature of the issue and the multitude of responses to this thread, it's hard to believe this isn't negatively affecting a large set of your customers at the moment.

@DerAlbertCom
Copy link

Hi @DergachevE

Yes, please give the rollout a priority, this is terrible for a couple of days now.

Most of our Builds are failing because of that, and we have to rerun it manually. And the Workarounds are not really suitable for us.

@anttix
Copy link

anttix commented Oct 7, 2024

Hi @DergachevE and @KonstantinTyukalov!

We see builds failing due to the same issue. While I appreciate the provided workarounds, is there a way to escalate this issue to speed up a resolution?

Thank you.

@shurick81
Copy link

shurick81 commented Oct 7, 2024

Here's a script for installing dotnet runtime, not full SDK:

- powershell: |
    # your target dotnet version
    $version = $env:version;

    # your target install path
    $sdkInstallationPath = Join-Path $env:AGENT_TOOLSDIRECTORY dotnet

    $scriptUrl = "https://dot.net/v1/dotnet-install.ps1"
    $scriptPath = Join-Path  $env:AGENT_TEMPDIRECTORY dotnet-install.ps1
    Invoke-WebRequest -Uri $scriptUrl -OutFile "$scriptPath"

    & $scriptPath -Runtime dotnet -Version $version -InstallDir $sdkInstallationPath -Verbose

    "$version" > "${sdkInstallationPath}\shared\Microsoft.NETCore.App\${version}.complete"
  displayName: Installing .NET
  env:
    version: 3.1.32

@shurick81
Copy link

shurick81 commented Oct 7, 2024

Also had to change to $sdkInstallationPath = "C:\Program Files\dotnet" so that apps finds the installed version in the default dotnet path. Otherwise I get this:

You must install or update .NET to run this application.

App: D:\a\1\s\someapp.exe
Architecture: x64
Framework: 'Microsoft.NETCore.App', version '3.1.0' (x64)
.NET location: C:\Program Files\dotnet\

The following frameworks were found:
  6.0.5 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  6.0.26 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  6.0.32 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  6.0.33 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  7.0.5 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  7.0.20 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  8.0.8 at [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Learn more:
https://aka.ms/dotnet/app-launch-failed

To install missing framework, download:
https://aka.ms/dotnet-core-applaunch?framework=Microsoft.NETCore.App&framework_version=3.1.0&arch=x64&rid=win-x64&os=win10

After all, this is the code that worked:

- powershell: |
    # your target dotnet version
    $version = $env:version;

    # your target install path
    $sdkInstallationPath = "C:\Program Files\dotnet"

    $scriptUrl = "https://dot.net/v1/dotnet-install.ps1"
    $scriptPath = Join-Path  $env:AGENT_TEMPDIRECTORY dotnet-install.ps1
    Invoke-WebRequest -Uri $scriptUrl -OutFile "$scriptPath"

    & $scriptPath -Runtime dotnet -Version $version -InstallDir $sdkInstallationPath -Verbose

    Get-ChildItem "$sdkInstallationPath\shared\Microsoft.NETCore.App"
    "$version" > "$sdkInstallationPath\shared\Microsoft.NETCore.App\${version}.complete"
  displayName: Installing .NET
  env:
    version: 3.1.32

@VtotheK
Copy link

VtotheK commented Oct 8, 2024

Same issue here, happens randomly (maybe 50% of runs fail). Starts downloading the SDK, returns early and subsequent nuget installation fails.

Detected platform (Primary): linux-x64
Detected platform (Legacy): -x64
Downloading: https://download.visualstudio.microsoft.com/download/pr/1ebffeb0-f090-4001-9f13-69f112936a70/5dbc249b375cca13ec4d97d48ea93b28/dotnet-sdk-8.0.402-linux-x64.tar.gz
Finishing: Install .NET Core SDK

@mortenbock
Copy link

I made a pipeline with the following steps, and ran it a few times:

      - task: UseDotNet@2
        displayName: Install .NET 6.0
        inputs:
          version: 6.0.x

      - task: UseDotNet@2
        displayName: Install .NET 8.0
        inputs:
          version: 8.0.x

      - script: dotnet --info
        displayName: dotnet info

The result was the following:

Agent Run v6 installed v8 installed
ubuntu-22.04 1 x x
ubuntu-22.04 2 x x
ubuntu-22.04 3 x
ubuntu-22.04 4 x x
ubuntu-24.04 1 x x
ubuntu-24.04 2 x
ubuntu-24.04 3 x
ubuntu-24.04 4 x x

Please prioritize a fix for this.

@KonstantinTyukalov KonstantinTyukalov added the awaiting deployment Related changes are waiting for deployment to be completed label Oct 8, 2024
@KonstantinTyukalov
Copy link
Contributor

KonstantinTyukalov commented Oct 8, 2024

Hi everyone, we are currently releasing fixed task versions - 2.247.0 and 2.247.1
If you see them working in your pipeline, please share feedback if the issue is resolved in your case or let us know if you are still experiencing issues.

Thanks

@DerAlbertCom
Copy link

2.247.1 first run succeeded (will watch it). 15 minutes ago with 246.1 was failing.

@anttix
Copy link

anttix commented Oct 8, 2024

First few builds using 2.247.1 were successful. Will report back once I have more data.

@sicil1ano
Copy link

sicil1ano commented Oct 8, 2024

Hi everyone, we are currently releasing fixed task versions - 2.247.0 and 2.247.1 If you see them working in your pipeline, please share feedback if the issue is resolved in your case or let us know if you are still experiencing issues.

Thanks

Hallelujah!

Sorry for my frustration, but sometimes I think we developers, users of these products, are not only paying users, but are also considered your QA.
I don't understand otherwise how it's possible to roll out code that would work only 50% of the time or less, causing problems worldwide.

@kimsey0
Copy link

kimsey0 commented Oct 8, 2024

Also seems to work nicely for us. Thanks!

@AleksuKey
Copy link

The fix works fine for us.

@DergachevE
Copy link
Contributor

Thanks everyone for your patience!
Looks like the issue was fixed so I am closing this issue, but feel free to re-open if needed.

@sicil1ano I understand your frustration.
The team was working hard on the RC-ing the issue, but considering the fact that previous versions of the task also started failing (but were working before) we had to investigate a combination of factors. This took us more time to deliver the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: ABTT Akvelon Build Tasks Team area of work awaiting deployment Related changes are waiting for deployment to be completed bug Task: UseDotNet
Projects
None yet
Development

No branches or pull requests