A library and command line tool for concurrent bulk investigation and occasional manipulation of multiple clouds.
Do you
- have multiple AWS accounts or multiple regions and need to list resources across them?
- get tired of the aws cli baroque command syntax?
- get annoyed by how slow and awkward the aws cli is?
This tool might be for you (at least eventually)
Futures: many more resource types, DigitalOcean, GCP, and Kubernetes
Becoming Useful. Only handles AWS, only handles a few resource types, but handles many accounts/regions concurrently.
Pick one:
- MacOS:
brew install deweysasser/tap/Cumulus - Pre-built binaries from GitHub release page.
- Other platforms (deb, rpm) TBD as I get a curiosity bump or someone needs them.
- You can always
go install github.com/deweysasser/cumulus@latest(currently the only thing you'll miss out on is the Makefile embedding version information in the binary.)
There's nothing to set up. The command will pick up all your profiles from ~/.aws/credentials,
deduplicate them (i.e. you may have more than 1 pointing to an account -- it will take care of it),
and use them.
If you use AWS creds in your environment -- file a bug and I'll make that work. It's not really important to me using multiple clouds.
If you have ~/.aws.credentials that you can use with the AWS CLI, try out cumulus instance list
and go!
cumulus (account|instance|snapshot|machine-image|volume|dns zone|dns record) list -I <column to include> -X <column to exclude> -l filter_column=value
All include, exclude, filter columns and filter values are regular expressions.
For filters, using multiple filters on the command line (... -l one=two -l three=four) will select
the UNION of objects matching the two filters, where using commas in a single filter
expression (...-l one=two,three=four) will select the INTERSECTION of objects matching the
individual filters.
Use --verbose or -v to show all available column and run statistics. Use --quiet or -q to
suppress the output of the column header row.
$ time bin/cumulus instance list > /dev/null
real 0m5.585s
user 0m0.718s
sys 0m0.141s
8:19PM INF discovered instance count=259
8:19PM INF timer average_latency=195.194586 count=47 name="AWS API calls" rate=8.566558814468866
8:19PM INF timer totals TotalCalls=47 TotalDuration=9174.145546 name="AWS API calls"
8:19PM INF run time duration=5484.798748
$ time ./bin/cumulus instance list -I '^type$' -X . | sort | uniq -c | sort -rn
59 c5.2xlarge
4 c5.xlarge
3 c5.4xlarge
1 c5.18xlarge
...
real 0m5.935s
user 0m0.738s
sys 0m0.150s
(Breakdown: List the instances, include the field matching regexp ^type$ in output, exclude all
other fields. You can figure out the rest of the pipeline.)
# Show the "environment" tag in a column
$ cumulus instance list -I tag:environment
# show only instances where the "environment" tag is "production"
$ cumulus instance list -l tag:environment=production
# show all instances where *any* column contains the word "test"
$ cumulus instance list -l .=test
# show all available fields for snapshots. Warning: there are a *LOT* of them.
$ cumulus snapshot list -v
# show snapshots tagged with "Release" containing "v1.0" (it's a regexp) *AND* "Status" containing "alpha"
$ cumulus snapshot list -l tag:Release=v1.0,tag:Status=alpha
It has similar performance for listing instances, zones...everything else (except, oddly, the API for S3 is pretty slow)
$ bin/cumulus -h
Usage: cumulus <command>
Flags:
-h, --help Show context-sensitive help.
--version Show program version
Output
--debug Show debugging information
--output-format="auto" How to show program output (auto|terminal|jsonl)
-q, --quiet Be less verbose than usual
-v, --verbose Be more verbose than usual
Profile
--profile.cpu profile the CPU
--profile.memory profile the Memory usage
Commands:
account list
instance list
snapshot list
machine-image list
volume list
dns zone list
dns record list
Becoming Useful. Only handles AWS, only handles a few resource types, but handles many accounts/regions concurrently.
Allows column selection and filtering.
Very fast
CLI might be awkward for now -- I haven't fully exercises all the options and made sure that everything behaves senibly all the time (e.g. when running the "--version" flag, you still have to give a subcommand)
Currently handles:
- AWS
- listing
- accounts
- instances
- snapshots
- AMIs
- EBS volumes
- Route53 Zones
- Route53 Recordsets
- listing
- Codgen as much as possible
- Handle many more AWS types (all the ones I use regularly)
- Add kubernetes (discover clusters across accounts, then discover resources inside them)
- Add Digital Ocean
- Add GCP
- Add Azure? (I don't currently use Azure, but maybe by this time I'll find someone who does to add to it)
The AWS command line is slow. S....l....o....w.
It's also...awkwardly laid out, in the format of "service - endpoint". GCP and Digital Ocean CLIs are much more intuitively designed in the form of "subject - object - verb - modifiers".
And, last, the AWS CLI works with a single account, and a single region, at a time. Even wehn I'm exclusively using AWS, this is not the right abstraction.
Oh, and very last...I have hit some non-linearity in the AWS command line, where large result sets are incredibly delayed. I'm not that patient.
So the goal of this tool is to produce a cloud library that's much better structured, faster, concurrent, and not tied strictly to AWS.
That last part is the most experimental. I have tried better structures for library and found them good. In this tool, I'm attempting a "common denominator" API approach to multiple clouds and...well...we're going to see how well that works.
Also, I'm using that library to create a better CLI.
Please use Github Issues for all bugs and feature requests.