panic: runtime error: invalid memory address or nil pointer dereference

[cv711]

Describe the bug

I'm starting a SSM session with port forwarding to connect to our MySQL through our bastion ec2 instance. The session starts and awaits a connection at the defined port. When I tried to connect to the DB we get the panic defined below.

Expected Behavior

Should happily forward the connection without a panic.

Current Behavior

aws ssm start-session \
    --profile $PROFILE \
    --region $REGION \
    --target $instance \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"host":["mysql-host"],"portNumber":["9030"], "localPortNumber":["9030"]}'


Starting session with SessionId: <SESSIONID-0be6e845e23adb7ed>
Port 9030 opened for sessionId <SESSIONID-0be6e845e23adb7ed>.
Waiting for connections...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x102679e44]

goroutine 40 [running]:
github.com/aws/aws-sdk-go/aws/credentials/ssocreds.(*SSOTokenProvider).refreshToken(0x1400014ef60, {{{0x14000568000, 0xe6}, 0x14000134ee8, {0x140005680f0, 0xe6}, {0x140001426c0, 0x22}, {0x1400056a000, 0x778}}, ...})
	/local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/ssocreds/token_provider.go:115 +0x2d4
github.com/aws/aws-sdk-go/aws/credentials/ssocreds.(*SSOTokenProvider).RetrieveBearerToken(0x1400014ef60, {0x1022e0e84?, 0x1400016f7c0?})
	/local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/ssocreds/token_provider.go:86 +0x108
github.com/aws/aws-sdk-go/aws/credentials/ssocreds.(*Provider).RetrieveWithContext(0x14000181480, {0x102a698f0, 0x140003ed490})
	/local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/ssocreds/provider.go:107 +0x248
github.com/aws/aws-sdk-go/aws/credentials.(*Credentials).singleRetrieve(0x14000181500, {0x102a698f0, 0x140003ed490})
	/local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/credentials.go:277 +0x28c
github.com/aws/aws-sdk-go/aws/credentials.(*Credentials).GetWithContext.func1()
	/local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/aws/credentials/credentials.go:255 +0x84
github.com/aws/aws-sdk-go/internal/sync/singleflight.(*Group).doCall(0x14000181500, 0x1400011baa0, {0x0, 0x0}, 0x0?)
	/local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/internal/sync/singleflight/singleflight.go:97 +0x38
created by github.com/aws/aws-sdk-go/internal/sync/singleflight.(*Group).DoChan
	/local/p4clients/pkgbuild-pmMNN/workspace/src/SSMCLI/vendor/src/github.com/aws/aws-sdk-go/internal/sync/singleflight/singleflight.go:90 +0x3d0

Command '['session-manager-plugin',...

Reproduction Steps

aws ssm start-session
--profile $PROFILE
--region $REGION
--target $instance
--document-name AWS-StartPortForwardingSessionToRemoteHost
--parameters '{"host":["mysql-host"],"portNumber":["9030"], "localPortNumber":["9030"]}'

different terminal

mysql -h mysql-host -P 9030 -u user_ro -p

Possible Solution

No response

Additional Information/Context

No response

SDK version used

aws-cli/2.13.8 Python/3.11.4 Darwin/22.5.0 source/arm64 prompt/off

Environment details (Version of Go (go version)? OS name and version, etc.)

go version go1.20.6 darwin/arm64

[wty-Bryant]

Hi, thx for reporting issue. I'm trying to reproduce the error. Could you explain more detail of (1) content of session and profile (2) code to create the session using local profile if there's any? You can remove any sensitive info. Thx

[cv711]

so profile is set to nest-sandbox and the ~/.aws/config looks like this:

[profile nest-sandbox]
sso_session = staging
sso_account_id = <account_id>
sso_role_name = nest-sandbox
region = eu-west-1
output = json

and before starting a session with SSM I am authenticating with the following command:
$> aws sso login --profile=nest-sandbox

Let me know if you need more info.
Thanks!

[isredstar]

I also occurs the similar problem:

[wty-Bryant]

@isredstar Hello could you show more detail of your local profile, sso session and the code to reproduce the npe? It seems like your cred provider is nil, and different credential could be retrieved according to context of a profile.

[wty-Bryant]

@cv711 We discussed with service team for possible causes of the npe, when an access token is returned from ssooidc, an ExpiresIn will always be returned and the return err will be nil, which is confusing that this line panic without err returned. Could you capture the actual createResult so we can check which var is nil

Currently possible workaround is to dump your sso cache to redact the AT/RT/ClientSecret, and we will update token provider code here to print the nil error rather than panic, which is not expected in sdk

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.