Customer issue: STS Assume Role signature mismatch

[skotambkar]

Customer reported signature mismatch error when assuming role via STS, need to investigate the same.

Customer issue: #883 (comment)

Reproducible code as provided by customer: #883 (comment)

[gmauleon]

Just for information, I got the same problem with v0.30.0 on eks calls.

[jasdel]

@gmauleon are you consistently running into this issue or is it periodically occurring? In addition, what AWS region is your application configured for when it encounters this error?

[gmauleon]

Consistent on my side. I shelved it and planned to use v1 for the time being but could certainly retry it this week if you need more informations.

[jasdel]

Thanks for the update @gmauleon. I'm trying to reproduce this issue, but not able to with latest release version v0.30.0, and HEAD of the v2 SDK repo.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials/stscreds"
	"github.com/aws/aws-sdk-go-v2/service/sts"
)

func main() {
	if err := do(); err != nil {
		log.Fatal(err)
	}
}

func do() error {
	cfg, err := config.LoadDefaultConfig(
		config.WithRegion("us-west-2"),
		config.WithClientLogMode(aws.LogSigning),
	)
	if err != nil {
		return err
	}

	stsclient := sts.NewFromConfig(cfg)

	role := fmt.Sprintf("arn:aws:iam::%s:role/%s", accountID, roleName)
	provider := stscreds.NewAssumeRoleProvider(stsclient, role)

	// SDK tip:
	cfg.Credentials = aws.NewCredentialsCache(provider)
	// SDK v0.30.0
	cfg.Credentials = &aws.CredentialsCache{Provider: provider}

	creds, err := cfg.Credentials.Retrieve(context.Background())
	if err != nil {
		return fmt.Errorf("failed to get credentials, %w", err)
	}

	log.Println("Credentials ", creds.AccessKeyID, creds.Source)

	return nil
}

I'm using the following sample code to reproduce this issue. If you have a sample that is able to reproduce this problem that would be very helpful to investigate this issue.

# Update to the tip of the smithy-go currently, `511e8c85` 
go get github.com/awslabs/smithy-go@ac098bf4

# Update to the tip of the SDK currently, `511e8c85`
go get github.com/aws/aws-sdk-go-v2@511e8c85
go get github.com/aws/aws-sdk-go-v2/service/sts@511e8c85
go get github.com/aws/aws-sdk-go-v2/credentials@511e8c85
go get github.com/aws/aws-sdk-go-v2/config@511e8c85

[jasdel]

The only way I've been able to reproduce this is when the original LoadDefaultConfig is unable to retrieve credentials to do the assume role with. Could you look if the cfg.Credentials value is a aws.AnonymousCredentials, or nil?

[gmauleon]

Thanks a lot for looking into this, I'll be out and will not be able to look at it until most likely January, sorry about that :( I'll see if I can look at my code next week but no promises. Have some happy holidays Jason!

…

On 2020-12-18 3:08 p.m., Jason Del Ponte wrote: The only way I've been able to reproduce this is when the original |LoadDefaultConfig| is unable to retrieve credentials to do the assume role with. Could you looking if the |cfg.Credentials| value is a aws.AnonymousCredentials <https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/#AnonymousCredentials>, or |nil|? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACMH332WJEZOTE57W452TKDSVOZFFANCNFSM4T2Z7BHA>.

[Chandrian]

Using v0.4, I'm getting the same issue (I think) failed to sign request: failed to retrieve credentials: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: ..., api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
if I use:

ctx := context.TODO()
	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRegion("us-east-1"),
		//config.WithClientLogMode(aws.LogSigning),
	)
	if err != nil {
		log.Fatal(err)
	}

	stsclient := sts.NewFromConfig(cfg)

	provider := stscreds.NewAssumeRoleProvider(stsclient, *assumeRole.masterARN)

	cfg.Credentials = aws.NewCredentialsCache(provider)

And then directly try to use it:

orgClient := organizations.NewFromConfig(cfg)
listAccountsForParentInput := &organizations.ListAccountsForParentInput{
		ParentId: aws.String(ou),
	}
	p := organizations.NewListAccountsForParentPaginator(orgClient, listAccountsForParentInput)
	var emails []string
	for p.HasMorePages() {
		listAccountsForParentOutput, err := p.NextPage(context.Background())
		if err != nil {
			log.Panicf("failed to get the page for accounts in assume OU: %v", err)
		}
	}

But if in between I add the following, everything works fine.... Any idea why?

creds, err := cfg.Credentials.Retrieve(context.Background())
	if err != nil {
		log.Fatal(err)
	}

[jasdel]

Thanks for the update @Chandrian I've been able to reproduce the issue with the following example using your provided input.

func main() {
	if err := countBuckets(context.TODO()); err != nil {
		log.Fatal(err)
	}
}

func countBuckets(ctx context.Context) error {
	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRegion("us-west-2"),
		config.WithClientLogMode(aws.LogSigning|aws.LogRequest|aws.LogResponseWithBody),
	)
	if err != nil {
		return fmt.Errorf("failed to load config, %w", err)
	}

	const role = "arn:aws:iam::<account>:role/S3ReadOnly"

	stsClient := sts.NewFromConfig(cfg)
	provider := stscreds.NewAssumeRoleProvider(stsClient, role)

	cfg.Credentials = aws.NewCredentialsCache(provider)

	s3Client := s3.NewFromConfig(cfg)
	result, err := s3Client.ListBuckets(ctx, nil)
	if err != nil {
		return fmt.Errorf("failed to list buckets, %w", err)
	}
	log.Println("buckets", len(result.Buckets))

	return nil
}

Log output of request

SDK 2021/01/05 14:30:52 DEBUG Request Signature:
---[ CANONICAL STRING  ]-----------------------------
POST
/

amz-sdk-invocation-id:7e5581df-451b-40bc-b597-87980191039a
amz-sdk-request:attempt=1; max=3
content-length:165
content-type:application/x-www-form-urlencoded
host:sts.us-west-2.amazonaws.com
x-amz-date:20210105T223052Z

amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20210105T223052Z
20210105/us-west-2/sts/aws4_request
01cb54b891c981d067ae340b7466903c08482ec7e1abb98dac5712fb2fb6b60d
-----------------------------------------------------
SDK 2021/01/05 14:30:52 DEBUG Request
POST / HTTP/1.1
Host: sts.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/0.31.0 GOOS/darwin GOARCH/amd64 GO/go1.15.6 sts
Content-Length: 165
Amz-Sdk-Invocation-Id: 7e5581df-451b-40bc-b597-87980191039a
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=<AccessKeyID>/20210105/us-west-2/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=bcf78b6df8908c8ac63073da076a5bc2848fd7e60aba39bfdc96704fe55e9fb9
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20210105T223052Z
Accept-Encoding: gzip

SDK 2021/01/05 14:30:52 DEBUG Response
HTTP/1.1 403 Forbidden
Content-Length: 431
Content-Type: text/xml
Date: Tue, 05 Jan 2021 22:30:51 GMT
X-Amzn-Requestid: 085924bc-3838-4f82-9d0a-e00431418df8

<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
  </Error>
  <RequestId>085924bc-3838-4f82-9d0a-e00431418df8</RequestId>
</ErrorResponse>
2021/01/05 14:30:52 failed to list buckets, operation error S3: ListBuckets, failed to sign request: failed to retrieve credentials: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 085924bc-3838-4f82-9d0a-e00431418df8, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
exit status 1

[Chandrian]

Yes, without the retrieve it does happen 100% of the time. with the retrieve, it never happens

[jasdel]

Looks like the issue causing the signature to be invalid is that something is preventing the SDK from computing the sha256 of the request body. Since its not computing the body, the sha256 of empty string is being used instead e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

[jasdel]

The issue is occurring because the SDK is effectively polluting of the Context value passed down from the top level operation -> signer -> credentials retrieve -> assume role. In my above code example this is the reason the content sha256 of an empty string value was being used with the STS request. S3's ListBuckets operation call has not body, i.e. empty string. This same issue would be similar for any nested API call invoked within the chain of another operation's middleware stack. Basically we'll need to "clear/blank" a set of context values that must not be passed down to nested operation calls within a middleware stack. content-sha256 being the example in this case.

We're working on a fix for this bug, and will update this issue when a fix is available.

---

With that said the following is a workaround that will ensure the content-sha256 value will be cleared for nested operations like the STS assume role.

Custom middleware that clears the content-shaw256 context value.

type clearContextContentSha256 struct{}

func (clearContextContentSha256) ID() string {
	return "clear content-sha256 workaround"
}

func (clearContextContentSha256) HandleInitialize(
	ctx context.Context, in middleware.InitializeInput, next middleware.InitializeHandler) (
	out middleware.InitializeOutput, metadata middleware.Metadata, err error,
) {
	ctx = v4.SetPayloadHash(ctx, "")
	return next.HandleInitialize(ctx, in)
}

This is then used when loading the config to ensure all operations don't pollute nested operation calls.

cfg, err := config.LoadDefaultConfig(ctx,
	config.WithRegion("us-west-2"),
	config.WithClientLogMode(aws.LogSigning|aws.LogRequestWithBody|aws.LogResponseWithBody),
	config.WithAPIOptions([]func(stack *middleware.Stack) error{
		func(stack *middleware.Stack) error {
			return stack.Initialize.Add(clearContextContentSha256{}, middleware.Before)
		},
	}),
)
if err != nil {
	return fmt.Errorf("failed to load config, %w", err)
}

[Chandrian]

I tried the workaround and it is working! Thank you for the quick turn around on this!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.