Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paginate ebs_snapshot_count check #32

Merged
merged 4 commits into from
Jan 10, 2022
Merged

Paginate ebs_snapshot_count check #32

merged 4 commits into from
Jan 10, 2022

Conversation

kedoodle
Copy link
Contributor

@kedoodle kedoodle commented Jan 7, 2022

This PR fixes a memory usage issue whereby quota checks make a single boto3 call, even when supposedly paginating. Symptoms described in this comment on #31.

  • Uses the existing paginator to count the ebs_snapshot_count quota
  • Sets a default pagination page size (otherwise the paginator just gets all resources in my testing of describe_snapshots)
  • Fixes up an error log message

See memory usage spikes to ~800 MiB on 1.10.0 and is fairly stable at ~50 MiB on this branch.
image

Before:

  • Locally
    $ time aws-quota-checker check ebs_snapshot_count
    
    AWS profile: default | AWS region: ap-southeast-2 | Active checks: ebs_snapshot_count
    EBS Snapshots per Region [xxx/ap-southeast-2]: 34896/100000 ✓
    aws-quota-checker check ebs_snapshot_count  12.72s user 2.32s system 3% cpu 7:27.41 total
    
  • In cluster (~3 minutes):
    AWS profile: default | AWS region: ap-southeast-2 | Active checks: 
    cf_stack_count,ebs_snapshot_count,rds_instances,s3_bucket_count
    ...
    07-Jan-22 06:10:21 [INFO] aws_quota.prometheus - refreshing current values
    07-Jan-22 06:13:39 [INFO] aws_quota.prometheus - current values refreshed
    

After:

  • Locally:
    $ time aws-quota-checker check ebs_snapshot_count
    
    AWS profile: default | AWS region: ap-southeast-2 | Active checks: ebs_snapshot_count
    EBS Snapshots per Region [xxx/ap-southeast-2]: 34899/100000 ✓
    aws-quota-checker check ebs_snapshot_count  12.33s user 1.36s system 16% cpu 1:25.01 total
    
  • In cluster (~1 minute):
    AWS profile: default | AWS region: ap-southeast-2 | Active checks: 
    cf_stack_count,ebs_snapshot_count,rds_instances,s3_bucket_count
    ...
    07-Jan-22 08:47:46 [INFO] aws_quota.prometheus - refreshing current values
    07-Jan-22 08:48:55 [INFO] aws_quota.prometheus - current values refreshed
    

@kedoodle kedoodle mentioned this pull request Jan 7, 2022
@brennerm
Copy link
Owner

Thanks for the contribution, looking good! 👍

@brennerm brennerm merged commit 3d49e6e into brennerm:master Jan 10, 2022
@brennerm
Copy link
Owner

Change has been released in version 1.11.0.

@kedoodle kedoodle deleted the paginate-ebs-snapshot-count-check branch January 11, 2022 22:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants