IssueCertificate fails for renewals due to missing next nonce needed to sign request payload

[GeorgDangl]

Yesterday morning, three scheduled renewals (all that ran) failed in a row.

The error message for the exception in Application Insights was:

Exception while executing function: IssueCertificate Orchestrator function 'IssueCertificate' failed: The activity function 'AnswerChallenges' failed: "missing next nonce needed to sign request payload". See the function execution logs for additional details. 

With this call stack:

Microsoft.Azure.WebJobs.Host.FunctionInvocationException:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:326)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__15.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:94)
Inner exception DurableTask.Core.Exceptions.OrchestrationFailureException handled at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw:
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableTaskExtension+<>c__DisplayClass68_0+<<OrchestrationMiddleware>b__1>d.MoveNext (Microsoft.Azure.WebJobs.Extensions.DurableTask, Version=2.0.0.0, Culture=neutral, PublicKeyToken=014045d636e89289: D:\a\r1\a\azure-functions-durable-extension\src\WebJobs.Extensions.DurableTask\DurableTaskExtension.cs:715)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1+<>c__DisplayClass7_0+<<TryExecuteAsync>b__0>d.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\TriggeredFunctionExecutor.cs:50)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeWithTimeoutAsync>d__27.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:555)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:501)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__20.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.29.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:279)

For this specific invocation, the entries from the logs look like this:

The error message from the Slack webhook looks like this:

Microsoft.Azure.WebJobs.Extensions.DurableTask.FunctionFailedException: The activity function 'AnswerChallenges' failed: "Failed to deserialize exception from TaskActivity: {"$type":"ACMESharp.Protocol.AcmeProtocolException, ACMESharp","ProblemType":0,"ProblemTypeRaw":null,"ProblemDetail":null,"ProblemStatus":-1,"StackTrace":"   at ACMESharp.Protocol.AcmeProtocolClient.SendAcmeAsync(Uri uri, HttpMethod method, Object message, HttpStatusCode[] expectedStatuses, Boolean skipNonce, Boolean skipSigning, Boolean includePublicKey, CancellationToken cancel, String opName)\r\n   at ACMESharp.Protocol.AcmeProtocolClient.GetNonceAsync(CancellationToken cancel)\r\n   at KeyVault.Acmebot.Internal.AcmeProtocolClientFactory.CreateClientAsync() in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Internal/AcmeProtocolClientFactory.cs:line 56\r\n   at KeyVault.Acmebot.Functions.SharedActivity.AnswerChallenges(IReadOnlyList`1 challengeResults) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedActivity.cs:line 281\r\n   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs:line 20\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs:line 52\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 555\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 501\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 279","Message":"Unexpected response status code [TooManyRequests] for [GetNonceAsync]","Data":{"$type":"System.Collections.ListDictionaryInternal, System.Private.CoreLib"},"InnerException":null,"HelpLink":null,"Source":"ACMESharp","HResult":-2146233088}". See the function execution logs for additional details.
---> DurableTask.Core.Exceptions.TaskFailedExceptionDeserializationException: Failed to deserialize exception from TaskActivity: {"$type":"ACMESharp.Protocol.AcmeProtocolException, ACMESharp","ProblemType":0,"ProblemTypeRaw":null,"ProblemDetail":null,"ProblemStatus":-1,"StackTrace":"   at ACMESharp.Protocol.AcmeProtocolClient.SendAcmeAsync(Uri uri, HttpMethod method, Object message, HttpStatusCode[] expectedStatuses, Boolean skipNonce, Boolean skipSigning, Boolean includePublicKey, CancellationToken cancel, String opName)\r\n   at ACMESharp.Protocol.AcmeProtocolClient.GetNonceAsync(CancellationToken cancel)\r\n   at KeyVault.Acmebot.Internal.AcmeProtocolClientFactory.CreateClientAsync() in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Internal/AcmeProtocolClientFactory.cs:line 56\r\n   at KeyVault.Acmebot.Functions.SharedActivity.AnswerChallenges(IReadOnlyList`1 challengeResults) in /home/runner/work/keyvault-acmebot/keyvault-acmebot/KeyVault.Acmebot/Functions/SharedActivity.cs:line 281\r\n   at Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs:line 20\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs:line 52\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 555\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 501\r\n   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 279","Message":"Unexpected response status code [TooManyRequests] for [GetNonceAsync]","Data":{"$type":"System.Collections.ListDictionaryInternal, System.Private.CoreLib"},"InnerException":null,"HelpLink":null,"Source":"ACMESharp","HResult":-2146233088}
---> Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type ACMESharp.Protocol.AcmeProtocolException. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'ProblemType', line 1, position 77.
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue)
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
  at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
  at DurableTask.Core.Serializing.JsonDataConverter.Deserialize(String data, Type objectType) in C:\source\durabletask\src\DurableTask.Core\Serializing\JsonDataConverter.cs:line 104
  at DurableTask.Core.Serializing.DataConverter.Deserialize[T](String data) in C:\source\durabletask\src\DurableTask.Core\Serializing\DataConverter.cs:line 54
  at DurableTask.Core.Common.Utils.RetrieveCause(String details, DataConverter converter) in C:\source\durabletask\src\DurableTask.Core\Common\Utils.cs:line 401
  --- End of inner exception stack trace ---
  --- End of inner exception stack trace ---
  at Microsoft.Azure.WebJobs.Extensions.DurableTask.DurableOrchestrationContext.CallDurableTaskFunctionAsync[TRe…

Digging a bit into that message reveals this bit:

"Message":"Unexpected response status code [TooManyRequests] for [GetNonceAsync]"

Might this be related to #300? I'm not sure where the GetNonceAsync call goes - does this call an API on Let's Encrypt, or is this on the DNS provider?

Actually, digging even further into that, I get this in Application Insights:

So, looks like https://acme-v02.api.letsencrypt.org/acme/new-nonce is return HTTP 429 "Too Many Requests" status code.

[shibayan]

@GeorgDangl Thanks for the bug report.
Added retries at the ACME request level in #337, but no response is returned for newNonce requests, so handling status code 429 seems to be necessary.

A retry for status code 429 will be built in soon.

[shibayan]

This problem has been fixed in v3.6.7. Thanks!