Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unable to capture/except sts error since upgrading from 1.13.1 #1866

Closed
pcanham opened this issue Oct 29, 2019 · 6 comments
Closed

unable to capture/except sts error since upgrading from 1.13.1 #1866

pcanham opened this issue Oct 29, 2019 · 6 comments
Assignees
@swetashre
Labels
closed-for-staleness guidance Question that needs advice or information.

Comments

@pcanham
Copy link

pcanham commented Oct 29, 2019 •
edited
Loading

The following code snippet use to work for finding out whether STS was enabled within a specific AWS region, but since upgrading the botocore library this is failing, for some reason you are not able to capture the error, it just performs a stacktrace

def get_enabled_regions(boto3_session: boto3.Session, service: str):
    regions = boto3_session.get_available_regions(service)
    enabled_regions = set()
    for region in regions:
        try:
            sts_client = boto3_session.client('sts', region_name=region)
            sts_client.get_caller_identity()
            enabled_regions.add(region)
        except ClientError as e:
            if e.response['Error']['Code'] == "InvalidClientTokenId":
                # error code received when region is disabled
                log.error('{}: "{}" not allowed please check whether STS is enabled for this region'.format(
                    e.response['Error']['Code'], region))
                pass
            else:
                log.critical(e)
                #raise
        except Exception as e:
            log.critical(e)
    return enabled_regions

Error

[2019-10-29 22:14:20] [INFO] Credentials found in config file: ~/.aws/config
[2019-10-29 22:14:21] [WARNING] Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/Users/pcanham/virtenvs/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 502, in _protected_refresh
    metadata = self._refresh_using()
  File "/Users/pcanham/virtenvs/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 643, in fetch_credentials
    return self._get_cached_credentials()
  File "/Users/pcanham/virtenvs/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 653, in _get_cached_credentials
    response = self._get_credentials()
  File "/Users/pcanham/virtenvs/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 786, in _get_credentials
    return client.assume_role(**kwargs)
  File "/Users/pcanham/virtenvs/venv02/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/pcanham/virtenvs/venv02/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
[2019-10-29 22:14:21] [ERROR] InvalidClientTokenId: "ap-east-1" not allowed please check whether STS is enabled for this region

Additional information, the permissions i am getting is from an AssumeRole which works with the aws cli tool, using role_arn and source_profile within the credentials file
@swetashre swetashre self-assigned this Oct 30, 2019
@swetashre
Copy link
Contributor

@pcanham - Thank you for your post. I am not able to reproduce the issue when i ran your code with latest version of botocore. Can you please make sure that you are able to run sts assume_role call without error before running the script ?

If you are getting error with assume_role call then you may have to set your credentials again for the environment.

If you are not getting error with sts assume_role but still you are not able to run your script then please provide me full stack trace. I need full stack trace in order to know what exactly happening as i am not able to reproduce the issue with your script.

@swetashre swetashre added closing-soon response-requested Waiting on additional info and feedback. labels Oct 31, 2019
@no-response
Copy link

no-response bot commented Nov 7, 2019

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

@no-response no-response bot closed this as completed Nov 7, 2019
@pcanham
Copy link
Author

pcanham commented Nov 8, 2019 •
edited
Loading

Hi,

I have upgraded the library to 1.13.13 and get errors still here is the stacktrace

./region_list.py --profile press-hosting-strategy
[2019-11-08 12:13:11] [INFO] Credentials found in config file: ~/.aws/config
[2019-11-08 12:13:12] [WARNING] Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 502, in _protected_refresh
    metadata = self._refresh_using()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 643, in fetch_credentials
    return self._get_cached_credentials()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 653, in _get_cached_credentials
    response = self._get_credentials()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 786, in _get_credentials
    return client.assume_role(**kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
[2019-11-08 12:13:12] [CRITICAL] An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
Traceback (most recent call last):
  File "./region_list.py", line 87, in <module>
    for region in get_enabled_regions(initial_session, 'ec2'):
  File "./region_list.py", line 48, in get_enabled_regions
    sts_client.get_caller_identity()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/signers.py", line 149, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/signers.py", line 229, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 591, in get_frozen_credentials
    self._refresh()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 486, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 502, in _protected_refresh
    metadata = self._refresh_using()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 643, in fetch_credentials
    return self._get_cached_credentials()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 653, in _get_cached_credentials
    response = self._get_credentials()
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/credentials.py", line 786, in _get_credentials
    return client.assume_role(**kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/pcanham/venv02/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

I have tested the permissions via the aws cli and have no issues with the AssumeRole operation.

To ensure this was not a permissions issue i have also moved to a different role which would give me full admin access to the account (created via AWS Organisations) called "OrganizationAccountAccessRole"

format of the boto3 credentials file is

[profile primary]
aws_access_key_id = XXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXX
region = eu-west-1

[profile XXXXXXX]
role_arn = arn:aws:iam::XXXXXXXXXXXX:role/OrganizationAccountAccessRole
source_profile = primary
region = eu-west-1

primary is the master account and 'XXXXXXX' is the account which i am trying to perform the STS checks against.

I havent had a chance to check the code, but from looking at the logging outputs for 1.13.1 and 1.13.13 there is a warning which is extra which i am wondering is invalidating something

[WARNING] Refreshing temporary credentials failed during mandatory refresh period.

Additional Notes, if you go into IAM and then Account settings, i have on purpose disabled certain STS endpoints, not sure whether this is why i am getting the stack trace

Screenshot 2019-11-08 at 14 07 28

@no-response no-response bot removed the closing-soon label Nov 8, 2019
@no-response no-response bot reopened this Nov 8, 2019
@pcanham
Copy link
Author

pcanham commented Dec 17, 2019

Thought I would test with the latest version and it still fails.

Version : 1.13.40

botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.
[2019-12-17 10:20:13] [ERROR] [__main__(36):get_enabled_regions] InvalidClientTokenId: "eu-central-1" not allowed please check whether STS is enabled for this region
^CTraceback (most recent call last):
  File "./eip_audit.py", line 176, in <module>
    for region in get_enabled_regions(initial_session, 'ec2'):
  File "./eip_audit.py", line 30, in get_enabled_regions
    sts_client.get_caller_identity()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 272, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 563, in _make_api_call
    operation_model, request_dict, request_context)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 582, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/signers.py", line 152, in sign
    auth = self.get_auth_instance(**kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/signers.py", line 232, in get_auth_instance
    frozen_credentials = self._credentials.get_frozen_credentials()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 591, in get_frozen_credentials
    self._refresh()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 486, in _refresh
    self._protected_refresh(is_mandatory=is_mandatory_refresh)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 502, in _protected_refresh
    metadata = self._refresh_using()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 643, in fetch_credentials
    return self._get_cached_credentials()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 653, in _get_cached_credentials
    response = self._get_credentials()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 786, in _get_credentials
    return client.assume_role(**kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 272, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 563, in _make_api_call
    operation_model, request_dict, request_context)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 582, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 135, in _send_request
    request, operation_model, context)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 167, in _get_response
    request, operation_model)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/httpsession.py", line 263, in send
    chunked=self._chunked(request.headers),
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/urllib3/connection.py", line 394, in connect
    ssl_context=context,
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
KeyboardInterrupt

@pcanham
Copy link
Author

pcanham commented Dec 17, 2019

Bit more information, found my script was not displaying the following error

[2019-12-17 11:08:48] [ERROR] [__main__(36):get_enabled_regions] InvalidClientTokenId: "ap-east-1" not allowed please check whether STS is enabled for this region
[2019-12-17 11:08:49] [WARNING] [botocore.credentials(507):_protected_refresh] Refreshing temporary credentials failed during mandatory refresh period.
Traceback (most recent call last):
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 502, in _protected_refresh
    metadata = self._refresh_using()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 643, in fetch_credentials
    return self._get_cached_credentials()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 653, in _get_cached_credentials
    response = self._get_credentials()
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/credentials.py", line 786, in _get_credentials
    return client.assume_role(**kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 272, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/pcanham/.asdf/installs/python/3.7.5/lib/python3.7/site-packages/botocore/client.py", line 576, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidClientTokenId) when calling the AssumeRole operation: The security token included in the request is invalid.

@swetashre swetashre added guidance Question that needs advice or information. and removed response-requested Waiting on additional info and feedback. labels Mar 16, 2020
@swetashre
Copy link
Contributor

@pcanham - Sorry for the late reply. It looks like the credentials being used to the make the initial assume role call are expired. Can you make any other API calls with the original credentials?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@swetashre swetashre

Labels
closed-for-staleness guidance Question that needs advice or information.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pcanham @swetashre