azd deploy fails part way through deployment in Azure DevOps pipeline, client assertion not in valid time range

[mip1983]

• Make sure you've installed the latest version using instructions in the wiki

Output from azd version
azd version 1.8.0 (commit 8246323)

Describe the bug

My CI/CD pipeline has been working, but recently this week has started failing ~80% of the time about 20m into the deployment.
My app is .NET Aspire and deploys two containers, an API and web, much like the starter template in visual studio 2022. The blazor web project takes a while to build, largely due to AOT I think.

The pipeline is setup much like the documentation specifies, with a build step to use az cli auth. It's failing during the Azure CLI deploy application step (that runs the inline script 'azd deploy --no-prompt'.

It deploys the API container fine, but now often fails on the longer build web project, this is what I see in the output on Azure DevOps:

Deploying services (azd deploy)

Deploying service ecodriver-analytics-api
Deploying service ecodriver-analytics-api (Logging in to registry)
Deploying service ecodriver-analytics-api (Pushing container image)
Deploying service ecodriver-analytics-api (Updating container app)
Deploying service ecodriver-analytics-api (Fetching endpoints for container app service)
  (✓) Done: Deploying service ecodriver-analytics-api
  - Endpoint: https://ecodriver-analytics-api.whitepond-f4e41896.uksouth.azurecontainerapps.io/

Deploying service ecodriver-analytics-web
Deploying service ecodriver-analytics-web (Logging in to registry)
Deploying service ecodriver-analytics-web (Pushing container image)
Deploying service ecodriver-analytics-web (Updating container app)
  (x) Failed: Deploying service ecodriver-analytics-web

ERROR: failed deploying service 'ecodriver-analytics-web': failing invoking action 'deploy', updating container app service: applying manifest: AzureCLICredential: ERROR: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-04-09T15:31:05.4899889Z, assertion valid from 2024-04-09T15:12:53.0000000Z, expiry time of assertion 2024-04-09T15:22:52.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 95c2035a-27fb-40bb-8e73-5574f4751800 Correlation ID: c87ea102-7f26-4c42-9e24-ab243f38ce8b Timestamp: 2024-04-09 15:31:05Z
Interactive authentication is needed. Please run:
az login

To Reproduce

Not sure exactly what you could try, you need an aspire application structured like the starter template, that takes a while to build, with AOT compilation enabled (I have WasmStripILAfterAOT as false as it seems to cause issues).

Then setup a CI/CD pipeline and see if you get similar results on multiple build/releases.

Expected behavior

CI/CD pipeline builds and deploys successfully.

Environment
Information on your environment:
* .NET 8 Aspire preview 4
* Visual Studio 2022 Preview, Azure DevOps

Additional context
Only started recently in the last week or so.

[weikanglim]

@mip1983 Do you happen to know if the service connection used here is using Federated Credentials? There's a current limitation where az provided tokens have a 10-minute lifetime, I suspect you're hitting the issue described in Azure/login#372 as a result.

To work around, based on this comment, you may be able to try adding another line to azd deploy --no-prompt, i.e.:

inlineScript: |
  az account get-access-token --scope https://management.azure.com//.default --output none
  azd deploy --no-prompt

and see if that helps.

[mip1983]

Yea, using Federated Credentials on the service connection. Though it was working ok until quite recently with similar build times, so a tad confusing what's changed.

I've set up a new service principle connection and that seems happy, but I might need the line for the token for the prod deployment, will give that a try.

[microsoft-github-policy-service]

Hi @mip1983. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

[microsoft-github-policy-service]

Hi @mip1983, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.