azure-cli package could lose some weight

[akx]

The azure-cli package (Ubuntu Xenial) could stand to lose some weight.

😲 and 😳 don't begin to describe my reaction at the size difference between the AWS CLI and the Azure CLI packages.

Package: awscli
Version: 1.11.13-1ubuntu1~16.04.0
Depends: python3, python3-botocore (>= 1.4.70), python3-colorama, python3-docutils, python3-rsa, python3-s3transfer, python3:any (>= 3.3.2-2~)
Installed-Size: 2.9 MB

Package: azure-cli
Version: 2.0.45-1~xenial
Installed-Size: 347 MB
Depends: libc6 (>= 2.17), libssl1.0.0 (>= 1.0.2~beta3)

• Why must the package bundle its own version of Python 3.6? Couldn't it just depend on a system python3 (on platforms where it is known there is a recent enough Python 3)?
  • Why does that bundled version also include the Python 3.6 test suite (55.8 MB)?
  • Why are all .py files precompiled to .pyc (twice in the case of the stdlib)?
• Why are the py2 versions of all files with _py3 variants even included if az is always run with Python 3?

It's also not just about the size; install speed is also a thing. It takes 50 seconds to install the single azure-cli package on Azure machine, with its 39 000 files (dpkg-query -L azure-cli | wc -l: 39094), just a little less than installing the twentyish packages that comprise all of awscli's dependencies (that can be used by other software on the system).

[yugangw-msft]

CC @mayurid @troydai @johanste @lmazuel, this is known issue. We did some improvement for windows installer, but same kind of improvement can be done on Linux as well.

[troydai]

Thank you for the feedback. You made a great point. We will definitely look into this. I will keep this issue open and assign it to myself.

[yugangw-msft]

@marstr, could you please prioritize this work and get it done sometime in April?

[marstr]

I'm excited to take care of this :)

[simonbrady]

@marstr Thanks for picking this up, I've just been through another painfully slow upgrade (to 2.0.60 under WSL/Ubuntu) so I can assure you this fix will be welcomed!

[benc-uk]

Geat that this is finally getting looked at!

Every time I update the CLI, I kiss goodbye to my machine for ~20 minutes. I dread updating it.
Bundling Python with the CLI seems like madness

Looking forward to these changes

[marstr]

20 minutes? Yikes. Which platform are you using @benc-uk? Maybe Ubuntu via WSL?

[benc-uk]

Yes.
A very common configuration I find at customers and partners. First it's the actual update and then Windows Defender going absolutely crazy

I've had some snide comments from customers about the CLI in this regard

[marstr]

All the more reason to get this taken care of. Thanks for the honest feedback.

[troydai]

The WSL file system being slow when transfer large number of files were the problem when I looked at it months ago. Windows Defender is more about the cold start time of the command.

[benc-uk]

It's a pipe dream at this point as I can see the amount of work that's gone into this Python version of the CLI, but...

A single executable (i.e. based on golang) would be ideal. I've been working with Kubernetes, Helm and Terraform. Their tooling is written in Golang and it's really nice just having this single static binary you can put anywhere

EDIT: the new AzCopy V10 has gone down this route

[marstr]

If you look into my GitHub profile, you'll see that I'm a huge fan of Go and have spent a non-trivial part of my career on/with it. I love Terraform and k8s, and am envious of their ease of distribution. TBH, there are a lot of benefits to Go, but with a project like ours where we need to support many many contributors across the company, asking people to learn Go is a much heavier lift than asking them to use/learn Python. Not to mention, I don't think this product is at burn-it down and rewrite quality just yet ;)

[benc-uk]

I totally understand, that's why I prefixed my comment with "this is a pipe dream" 😄
Having done some Go myself it's quite the context switch from other languages

Hopefully some tidy up of the packaging is all that is required to get things speedy again

[sptramer]

@marstr Will slimming the .deb package finally remove the bundled Python interpreter and finally rely on the system Python? This will affect install instructions.

[marstr]

Yes! That's what I'm working on right now.

[jiasli]

Perhaps one can use some REST api directly and build some really small utility for specific tasks but not sure how to do that?

This (use some REST api directly) is exactly what we are investigating in the Microsoft Graph migration (#12946). If it turns out to be working well, we may incorporate this method in all future command modules or code-gen'ed modules.

[ringerc]

I was stunned to find that my diagnostic Docker containers grew by over 1GB when I installed the Azure CLI. This is ... astonishing.

[deyvsh]

Appreciate the work going on to slim down the install! In case it's of interest, our use case is to run it inside AWS CloudShell, which has a home dir (the only writable bit) size limit of 1GB, and we only need enough of the CLI to support az login.

[benc-uk]

I don't think there's anyway out of this now, beyond a full rewrite in Go, Rust or Dotnet etc, something that can compile to a single binary

Each week that passes the CLL gets larger and larger, and the chances of a rewrite smaller and smaller

Where does it stop? When the CLI takes up 2GB? 5GB? 50GB?

[sodul]

@benc-uk Have you tried https://github.com/clumio-code/azure-sdk-trim ?

I wrote this a while back to delete older API directories that are obviously superfluous in 99% of cases and the site-packages/azure directory goes from 846.6 MB to 348.1 MB for us. That's not perfect, it is still ridiculously large, but until the Azure developers actually do something about this problem it does help quite a bit.

If you call it from a Dockerfile make sure to have it in the same RUN call as when the Azure cli/sdk is installed or you will still get the overhead in the layers.

[jiasli]

@benc-uk, If you do an ncdu, you will find Azure CLI itself (/cli, 28.6 MB) is pretty small:

It is actually the Azure Python SDK (/mgmt, 676.7 MB) that makes Azure CLI huge. See #7387 (comment) for the explanation.

Rewriting in Go, Rust, .NET won't give too much benefit as those Azure SDKs are equally huge.

[benc-uk]

Agreed, but with a compiled language you ship a binary, you don't ship the SDK with it!

If I wrote a Go app that used the Go SDK for Azure, used a few functions from the SDK, and compiled it - it would be completely standalone executable and only a few megabytes

[usrme]

I still want to see a version of Azure CLI that can be composed of different components more easily, but I've written a blog post that can hopefully help some people out: https://usrme.xyz/posts/how-to-trim-a-container-image-that-includes-azure-cli/. Using the methods described there I was able to slim an image from 1.17GB to 307MB!

[psadi]

I was trying to find a cli mainly for azure devops. Installing the cli literally (via pip) shocked me. My usage is just to manage repositories & pull requests.

I understand the installer itself is pretty small, but having 1G of dependencies is not ideal, at-least for my use case

Isn't there way to decouple modules and use them independently, the current az cli is kind of overkill for my workflow.

[ivanechegaray]

Some people are confusing or don't understand the problem.
The CLI alone can only weigh a few megabytes, but the bad design of the entire CLI, not only the core part, causes it to have dependencies for more than 1.5GB and continues to grow as well commented above. For automated processes that use disposable computing where you can't cache images, downloading the 1.5GB is not optimal.

Even more I see with great surprise that some say that changing to Go we will practically not gain anything regarding the size of the CLI, surely they are the ones who love and support the current development that is a disaster.

[mpender]

quite true on the actual issue of dependency hell, just looking at the pip install execution logs makes ones eyes water in disbelief. Looking at other major cloud providers I can see their equivalent cli tools weigh in differently
AWS ~ 210 MB
GCP ~ 822 MB

Does it add more confusion to request a 'core' cli that is smaller but handles common activities (VMs, Blobs, AD, etc) or is that just masking the fundamental issue. think I would nearly prefer to dynamically add extensions to whatever i need rather than pulling down everything.

[bebound]

I've created this pr to fix the problem. #25801
result:
Ubuntu 22.04 installed size 1,196 MB -> 322 MB
Docker image size 1,300 MB -> 706 MB