Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

publish v2 to PyPI #4947

Open
mattsb42-aws opened this issue Feb 11, 2020 · 74 comments
Open

publish v2 to PyPI #4947

mattsb42-aws opened this issue Feb 11, 2020 · 74 comments
Labels
feature-request A feature should be added or improved. installation p2 This is a standard priority issue v2

Comments

@mattsb42-aws
Copy link
Member

It's great that there's a first-class supported installation mechanism that does not rely on environment specifics, but this is still a normal Python package, and for those customers who do already have an existing compatible environment the installation mechanism is a lot heavier-weight than using the existing Python distribution mechanisms.

Could we get v2 publishing to PyPI so that those customers who have those prerequisites can have a simpler and easier to maintain installation/updating/etc story?

@joguSD
Copy link
Contributor

joguSD commented Feb 11, 2020

Not publishing CLI v2 to PyPI has been a conscious decision and for the time being we have no plans to publish to PyPI. In the long term we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature.

@joguSD joguSD added the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 11, 2020
@sbbowers
Copy link

@joguSD Could you please elaborate on the reasoning behind this conscious decision? Perhaps link us to some dialog around this discussion?

@ssbarnea
Copy link

I think that time may prove if that decision was more towards conscious or in-conscious one. I am really curious to see how this will evolve, I only hope I have enough popcorn 🍿.

On the other hand, other wheels coming from the same source did not get their notoriety without any reasons. boto, boto3 anyone?

At least pypi was 3rd party that guaranteed that a specific version is not altered once uploaded. I guess now we will have to keep saving checksums .

@mattsb42-aws
Copy link
Member Author

mattsb42-aws commented Feb 13, 2020

So pip install git+git://github.com/aws/aws-cli.git#2.0.0 then. :)

@no-response no-response bot removed the closing-soon This issue will automatically close in 4 days unless further comments are made. label Feb 13, 2020
@ssbarnea
Copy link

Sure, we all know that is impossible to force push code to github, not.

@adamchainz
Copy link

Or use a tarballs link for greater speed https://adamj.eu/tech/2019/03/11/pip-install-from-a-git-repository/

@joguSD
Copy link
Contributor

joguSD commented Feb 14, 2020

Just as a reminder, the AWS CLI V1 will continue to receive regular updates
that are published to PyPI and pip will continue to be a supported
installation mechanism for AWS CLI V1.

Here are some reasons why we dropped publishing to PyPI for AWS CLI V2, and are
not supporting pip as an official installation mechanism.

  1. Installing with pip requires Python knowledge. The AWS CLI isn't a tool
    exclusive to Python developers and it's very easy to end up with an invalid
    installation. Often, reported issues end up being due to a misconfigured pip
    install that's getting dependencies from the wrong place, etc. These issues
    happen unexpectedly and are hard to debug which is frustrating for customers.
    It's also possible to inadvertently break other packages in the same
    environment.

  2. Depending on a C extension can mean that at pip install time a C compiler
    and any other required libraries may need to be present for the install to
    succeed. Because of this we've avoided depending on libraries that require C
    extensions as having a compiler wasn't previously a requirement to install the
    AWS CLI. We also do not feel comfortable adding a compiler as a requirement to
    install the AWS CLI V2. For V1 this has limited our ability to implement some features
    as there are no suitable Python native libraries that meet our requirements
    (cryptography, etc).

  3. Supporting pip as an installation mechanism requires supporting large
    dependency ranges for the packages we depend on. While simple in concept, this
    is difficult to execute on reliably as it's not feasible to test all
    permutations of platform, Python version, and package version. When new
    packages we depend on are published V1 customers installing via pip are often
    broken. This leads us to have relatively aggressive version ranges and we end
    up becoming a blocker for the Python community to upgrade packages.
    See upgrade to PyYAML 5.2 #4749 PIP dependency conflict on colorama with AWSCLI #4561 Installation is broken: Requirement.parse('urllib3<1.23,>=1.21.1') #3535

  4. There are some things we have no control of when installing via pip. For
    example, we have no influence on the openssl version or what TLS related
    features are supported as those will depend on how Python was built. This makes
    it difficult to ensure functionality of the AWS CLI.

  5. We cannot include or ship additional binaries (that may not even be written
    in Python) as part of an AWS CLI install. In the future the AWS CLI can be
    partially or even entirely rewritten in another language while maintaining
    backwards compatibility.

Hopefully that helps to shed some light on why we decided not to publish the
AWS CLI V2 to PyPI.

Lastly, we're looking to continue to improve the installation process for AWS
CLI V2 by potentially integrating with homebrew and Linux package managers, and
providing docker images. We'll also be looking to include build instructions
for cases where the pre-built binaries aren't available or suitable for a
particular use case.

@ulidtko
Copy link

ulidtko commented Feb 14, 2020

Those sound as decent and understandable reasons — except perhaps the #1. I'd avoid going into language comparison contests here — but it's hard to refute that Python is among the few must know languages for DevOps folks and the like.

Likely, I'm being ignorant in this judgement; but one can be forgiven for not "really knowing" even Bash — but not Python.

It's the language of humanity's first ever photo of a [shadow of] black hole, remember?


Besides: that's some really wierd reasoning, don't you think?

  • Some users have hard time with pip, while others (most?..) pip users are quite happy and silent.
  • Yet others have hard time with prebuilt binaries, while others (most?..) binariy users are quite happy and silent.
  • Therefore, we'd punish ALL pip users, including the happy&silent ones, and force them to go use binaries.

Edit: sorry, disregard this.


Lastly: pip install https://github.com/aws/aws-cli/archive/2.0.0.tar.gz fails with:

No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0)

What's the route to go to install aws-cli v2 from source?

@quiver
Copy link
Contributor

quiver commented Feb 14, 2020

@ulidtko

Not only aws-cli but also botocore(low-level API) is not published to PyPI for v2, so you need to install botocore beforehand.

For example, on a vanilla A1(ARM-base) Amazon Linux 2 instance, you can install AWS-CLI v2 as follows:

$ sudo yum install -y python3 python3-devel gcc libffi-devel openssl-devel
$ python3 -mpip install  https://github.com/boto/botocore/archive/v2.tar.gz
$ python3 -mpip install  https://github.com/aws/aws-cli/archive/v2.tar.gz
$ aws --version
aws-cli/2.0.0 Python/3.7.4 Linux/4.14.165-131.185.amzn2.aarch64 botocore/2.0.0dev4

@thimslugga
Copy link

thimslugga commented Feb 14, 2020

What about providing an AppImage for Linux vs the zip archive with the installer? I created one using the contents from the dist dir and it seems to be working fine. Just throwing it out there as it may be worthwhile way to distribute the v2 release for Linux. I can share steps I used if there's any interest.

@bristea
Copy link

bristea commented Feb 15, 2020

That like is like gold

No matching distribution found for botocore==2.0.0dev4 (from awscli==2.0.0)

Translates to "We made a release using unreleased dependencies." and "We didn't test if the release artifact can be installed".

Nobody mentioned is that binary code downloaded needs to signed on MacOS, so which means that the so called brew alternative install method will take considerable amount of effort/time.

Let's face it, v2 was released even without a tag being made of github, this should say a lot. AFAIK it could also be made from another repository, another license. The v2 was not presented as a preview and was directly published on https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html as a recommended release when the code repository, didn't had any tag, prerelease or not.

I would think twice before arguing that so many users are not able to use pip, or compile code, when basic software development practices like tagging are not followed.

@sbbowers
Copy link

Installing with pip requires Python knowledge...

I agree. Doing things in a domain requires domain knowledge.

Depending on a C extension can mean that at pip install time a C compiler
and any other required libraries may need to be present...

This is more domain knowledge, sure.

Supporting pip as an installation mechanism requires supporting...

It seems to me that this is your crux. Supporting PIP costs time and money and it sounds like it is generally painful. I suspect this is because, of the people that need support, they don't have the domain knowledge you'd mentioned.

There is a difference between "supporting PIP as the method of install" vs "uploading your package to PyPi". You can keep on delivering your curl/unzip/bash method of deployment as official and supported and also allow, for the people that know and want to, install v2 via pip as an "unsupported" option.

There are some things we have no control of when installing via pip...

You may not have control over it, but you can check for it and make it dependent on a specific version.

We cannot include or ship additional binaries...

Additional binaries... sounds like dependencies, which could be checked for during the build process.

The take-away I'm trying to make here is that supplying your packages via PIP isn't antithetical to your desire to limit support to your official installer. Would you please reconsider your aversion to pypi? Many developers would appreciate it.

@ulidtko
Copy link

ulidtko commented Feb 17, 2020

There are some things we have no control of when installing via pip. For
example, we have no influence on the openssl version or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI.

Don't you think that this is quite intentional?..

To a user of your package, that's an atrocious idea: that installing an API client tool gives control over openssl on my system to the tool author. I'm getting shivers just from thinking of it!

Constructively, there're two things you can do:

  • outright fail pip build, if the given openssl isn't good enough;
  • graceful degradation, where parts of functionality which depend on missing TLS features just get disabled.

We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install. In the future the AWS CLI can be
partially or even entirely rewritten in another language while maintaining
backwards compatibility.

Why, what's wrong with setup(data_files=...) and/or MANIFEST.in, did you look into it?

Thousands of developers do include arbitrary files in their pip installs. I see no problem here.

Oh wait, actually I do. Binaries don't belong in source installs.

If I'd want to rewrite aws-cli in golang or something — I'd make it separate project, in a separate repo, and maintain its interface as "CLI compatible" rather than vaguely "backwards-compatible".

Don't ship binaries where source is expected. No-no. No no no no.

@benbridts
Copy link

benbridts commented Mar 9, 2020

@joguSD Would it make sense to have separate issue for "install with a package manager" or "install with package manager X". Currently It's hard to automate the installation with the bundled installer (because you may not have a directory in your PATH that's suited for not-packaged binaries).

And it could be a good place to consolidate all the related issues like this one, #4926 #4842 #4660 #3553 #3627 #2501 and #727.

With v1 the suggestion was (probably rightly so) to use pip when needing a packaged/automated installation, with v2 this isn't an option anymore

@allenatdecisiv
Copy link

@joguSD what about platforms you don't directly support but language packaging mechanisms (like PyPi) do? For instance, Cygwin?

@jpetazzo
Copy link

jpetazzo commented May 3, 2020

Hi!

I do understand that AWS is trying to make it easier to install the AWS CLI for users who are not familiar with Python. It is regrettable that it is done at the expense of users who are familiar with Python, in a way that prevents them from integrating the AWS CLI within many environments.

For instance, many platforms will let you specify Python dependencies in a requirements.txt or similar file. A package published on PyPI becomes installable on these platforms without any extra or special work. A package installed through a script that requires root does not.

This is the case on Heroku, and it seems to be the case on AWS Lambda too. If I want to run a script using the AWS CLI on Lambda, it looks like I could do it with AWS CLI v1 (since the Lambda documentation explains how to package Python dependencies) but I don't know if it'd be possible with AWS CLI v2.

There are also many platforms that won't allow root access. A package published on PyPI can be installed as an unprivileged user, while the AWS CLI v2 installer requires sudo.

There are ways to have your cake and eat it too: check out Docker Compose installation instructions. Compose is available as a standalone executable for most platforms, and can also be installed with pip install.

I understand the desire to make Python an "implementation detail". But every software is written in some language, and that language has best practices around installation procedures. For instance, C code can leverage e.g. autoconf and automake, and every Linux distribution will have helpers to facilitate packaging software that uses autoconf and automake. Likewise, software written in Python can leverage PyPI.

Finally, if CLI installation is truly a huge point of friction, there are more drastic solutions as well. For instance, it could be ported to Go, to allow producing a standalone binary for every major platform. In 2012, I was working at Docker Inc. when we made the decision to rewrite our container engine from Python to Go, in order to facilitate its adoption by the largest number of people. I do understand that the AWS CLI is a complex piece of software that needs to interface with hundreds of services. But if a 15-person start-up (the headcount of Docker back then) can rewrite a container engine in Go, perhaps a company like AWS can manage to port a CLI as well? ;-)

If porting the CLI to Go seems to big a burden, I'd suggest to at least offer a PyPI installation process. It's fine if it's "not officially supported". At least don't make our job harder.

Thanks,

@titan550
Copy link

titan550 commented May 3, 2020

The installation instructions look like a alpha release note. However, the documentation suggests that v2 is a prod release. The only installation method that seems to be prod ready is Windows MSI (I have not tested).

Please provide options for installing using package managers such as pip etc ASAP.

@borrell
Copy link

borrell commented May 4, 2020

Unfortunately a new core AWS feature (SSO) is a non-starter for us for the moment, until

  1. CLIv2 is available through some sort of versioned repo (pip, distro repos, anything) and not just an installer linked on an AWS documentation page
  2. Core SDK support for basic things like 'use SSO to authenticate' are available (this is purely directed at botocore).

@tmccombs
Copy link

tmccombs commented Jun 2, 2020

Another problem with the current install process is that while there are instructions on how to update the package, there doesn't appear to be any documented way to programmatically check if an update is available.

@bendog
Copy link

bendog commented Jun 5, 2020

I'm having to use the new aws sso login feature a lot, and since i'm working inside virtual environments, which use tools which have awscli as a dependency, I have to exit my virtual environment every time i need to use the sso login (hourly).
Please save me from this! please release to PyPI!!

@vrdhn
Copy link

vrdhn commented Jun 15, 2020

For my own future reference 😄

pip install --user  https://github.com/boto/botocore/archive/v2.zip  https://github.com/aws/aws-cli/archive/v2.zip
## Not working anymore ...
pip install --user  git+git://github.com/boto/botocore.git#v2  git+git://github.com/aws/aws-cli.git#v2

@mcfedr
Copy link

mcfedr commented Jun 17, 2020

Another disadvantage with not publishing to pip, you making me have a second python interpreter installed, which might be fine for a dev machine, but in a docker image isnt good.

This is related to the issues installing on alpine linux in #4685 - if it just used the system python (as in v1) then would be no problem.

tim-bm added a commit to tim-bm/kube-tools that referenced this issue Jul 28, 2020
The reason for installing version one
aws/aws-cli#4947
@NewbiZ
Copy link

NewbiZ commented Aug 11, 2020

Installing with pip requires Python knowledge. The AWS CLI isn't a tool
exclusive to Python developers and it's very easy to end up with an invalid
installation. Often, reported issues end up being due to a misconfigured pip
install that's getting dependencies from the wrong place, etc. These issues
happen unexpectedly and are hard to debug which is frustrating for customers.
It's also possible to inadvertently break other packages in the same
environment.

Not sure I get your point here. Installing with pip does not mean pip has to be the only way to install aws cli.
e.g. There are already RPM and DEB packages of AWS CLI at the moment.

Depending on a C extension can mean that at pip install time a C compiler
and any other required libraries may need to be present for the install to
succeed. Because of this we've avoided depending on libraries that require C
extensions as having a compiler wasn't previously a requirement to install the
AWS CLI. We also do not feel comfortable adding a compiler as a requirement to
install the AWS CLI V2. For V1 this has limited our ability to implement some features
as there are no suitable Python native libraries that meet our requirements
(cryptography, etc).

Again, I do not get the point you are making here. Nobody is forcing AWS CLI to be distributed as a source python package. You can ship pre-built versions of AWS CLI as wheels to pypi, thus have no build-time requirement during installation.

There are some things we have no control of when installing via pip. For
example, we have no influence on the openssl version or what TLS related
features are supported as those will depend on how Python was built. This makes
it difficult to ensure functionality of the AWS CLI.

What are you implying here?
That you will now ship your own python interpreter and openssl inside your new AWS CLI "bundle"?
Considering your main point seem to be the support burden, I fear this decision will just change your support requests from "help me aws cli is broken because of my openssl version" to "please release a new bundle with an updated openssl because of CVE-42".
Will I have to independently update AWS CLI whenever there is a security fix in python/openssl ?

We cannot include or ship additional binaries (that may not even be written
in Python) as part of an AWS CLI install. In the future the AWS CLI can be
partially or even entirely rewritten in another language while maintaining
backwards compatibility.

Of course you can, multiple libraries on pypi do ship with prebuilt dependencies embedded.
You can either statically link your bindings with the depended library during wheel creation, or just directly embed the shared/dynamic library in your package and use that for your cffi calls, there's plenty of ways to do that.


To be honest, this move of getting out of pypi is really getting on my nerve.
The code quality of boto3 and AWS CLI is abysmal, and all I see here is a escape plan from that codebase.
I am pretty sure we will soon have an announcement along the lines of "Sorry, we screwed up in Python, we are now working on porting AWS CLI to Go, we hope it's going to be better <3"

@tmccombs
Copy link

Considering your main point seem to be the support burden, I fear this decision will just change your support requests from "help me aws cli is broken because of my openssl version" to "please release a new bundle with an updated openssl because of CVE-42".

This is especially bad, since there is not a great way to automate upgrading the AWS CLI now.

@ackalker
Copy link

ackalker commented Jan 27, 2021

You can load zipfile as a module on the command line

This is a great tip, but nevertheless misses the point of this issue. pip > install should be a supported way to install awscli2.

Of course you are right, my comment was meant tongue in cheek.
No project should force its users to download and install an entire separate Python environment built from who knows what sources just to run a single tool.
There are shops out there that have very strict rules about what software is allowed to run on devops' machines, usually the need to vet the source code from which that software is built, and no random binary blobs downloaded from the internet, even if they are hatched by Amazon, are ever going to fit that bill.

@henning-schild
Copy link

If you want to sell your services you better bend over backwards to make those tools readily available in a professional manner. Meaning packages in all Linux distros ... skip pypi if you want but things like https://bugs.gentoo.org/767268#c1 are shocking results of a really poor deployment channel for that essential component. sudo funny-script-without-signature-or-checksum ...

@danpilch
Copy link

danpilch commented Feb 8, 2021

Removing the path of least resistance (installing via pip) just makes me not want to upgrade. I'm sure a lot of other people will ignore v2 until v1 is EOL and we're forced to install a python cli tool in a non native way.

@ulidtko
Copy link

ulidtko commented Feb 8, 2021

I'm sure a lot of other people will ignore v2 until v1 is EOL

I'm sorry if this is bitter feedback for some, but having learned that AWS staff has difficulties with basic packaging and versioning so much that they want to make Python an implementation detail of aws-cli — I'm reluctant to use even v1. Who knows what other nonsense they decided to put into there; I don't care enough to review the 70kSLoC myself.

@danvaida
Copy link

$ wget https://awscli.amazonaws.com/AWSCLIV2-2.1.26.pkg
$ open AWSCLIV2-2.1.26.pkg
$ aws --version
aws-cli/2.1.26 Python/3.7.4 Darwin/20.3.0 exe/x86_64 prompt/off

Meanwhile, a pip-installed CLIv1 runs fine on Python 3.9.
Now, I think therein lies a tale... 😄

@vacri
Copy link

vacri commented Feb 18, 2021

I'm clearly late to this conversation... but just to highlight a major issue again: package managers are there for a reason. Throwing away package management altogether is such an incredible antipattern. There is so much that package managers do in the ops space.

Even in the environment of the dev workstation, where there's a human to control everything, what used to be

[package manager] install [package]

is now

  1. hunt for webpage with link
  2. run a command to pull the package down
  3. find a temp dir to extract it to
  4. scan the install script because it's a custom script doing a custom thing and who knows what it might do - hope you're familiar with shell!
  5. run the install
  6. test it

That's the recommended "happy path" with no edge cases. And then repeat that all again when there's an update... if you remember to check for one. This is not the right method for less-skilled admins, plus rather than "doing the work once" by the developers, you're making every admin duplicate the work.

If not pip because you want to go language agnostic, then why not rpm for Amazon Linux and maybe deb because it's the other super-popular package format? Both of these are designed for dependency management and deal with upgrades fluidly, and you don't have to hunt for a link in a webpage for packages in them.

I don't have the venv or docker problems that other users have but I'll keep using v1 anyway, because updating to get new features is a one-liner, rather than a manual zipfile install method from last century (no joke there - rpm's from '97, apt's from '98, cpan '95...)

@danizen
Copy link

danizen commented Feb 20, 2021

The reason this is an anti-pattern for development is that a different version of botocore may be installed for other reasons, and so we Python developers who use virtual environments simply cannot use awscli v2 without using the workarounds above, because it would conflict with the packages used by other tools.

This works against proper development practices.

Some community work-arounds, such as pypi package awscliv2, work for me only at home because within my federal agency, we do not yet have access to Docker or WSL, and python virtual environments are my way to manage all dependency isolation.

@ShadowJonathan
Copy link

@danizen i would recommend keeping working with venvs, because they're so widely used, and so more manageable than complete system environments, just a note.

@mvanbaak
Copy link

we do not yet have access to Docker or WSL, and python virtual environments are my way to manage all dependency isolation.

Also, I dont want to install and run docker just for the awscli. It's not that I cant install it, it's just that I dont use docker, and having to go that route just for a cli tool ... yeah, that's a NOPE

@fff7d1bc
Copy link

Here you have a very well done paper on why the route you've choose with awscli v2 is wrong -- https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packagers-security-nightmare/

@zerthimon
Copy link

Unbelievable. And the worst part is - it's Amazons official. I wouldn't be surprised if it was Alibaba cloud or something, but it's Amazon. Really ? Wow, just wow.

@ShadowJonathan
Copy link

This issue isn't going to progress anything, it's not getting closed nor resolved, but kept open to blackhole discussion, nothing will come from this, and nobody can force amazon to do anything about it. So i suggest finding other ways to forward this issue, other than leaving behind a comment here saying your support.

I've been watching this issue thread for a while now, and I haven't seen a member respond to the latest concerns and calls for python best-practice here, other than suggesting an entirely un-ergonomic way of deployment, not even willing to meet the developers half-way, dismissing releasing on pypi in favour of a vendored appimage blob.

Earlier here it's already been said that amazon considers using python in general as an implementation detail to how the aws cli is run, fine, fair, but then it acts as if it's above any choice it makes in that regard, e.g. not answering to what the community of python expects of their standardised way of publishing a package for a 30-year-old programming language, on a 20-year-old publication site, for a cloud service that has then only existed for 15 years, and then act as if their way is the highway.

Well then, @aws, fucking commit, make your own https://pypi.aws instance, dont just sit around and dabble with this absolute bullshit. You love developers so much? You love your customers so much? Prove it then, your platform has been a hot piece of overpriced garbage for a decade and a half now, so you cant change my mind, but you might change others.

@ulidtko
Copy link

ulidtko commented Feb 22, 2021

Just boycott awscli v2. Don't use it.

If anyone ever wonders why — explain. Hopefully you can be understood.

@danizen
Copy link

danizen commented Feb 25, 2021

I do see the problem with installing in a virtual environment from AWS perspective - there can be as many installations of AWS CLI as there are virtual environments, and together with user knowledge and the issue of including other capabilities maybe not written in Python and controlling the version of SSL (LibreSSL probably on their horizon), it makes sense.

However, I have programs that use boto3 to do things similar to aws sso login, and they rely on boto3 and therefore botocore.

Bottom-lines:

  • If it cannot be published to PyPI then you need to assure that when I invoke the command-line aws from within a virtual environment, it uses its own version of botocore. I will be testing this and reporting back with V2 on Windows and then on Mac.
  • It has to be published in a package format where we can programmatically determine updates. This is part of Software Supply Chain management for developers & DevOps, but also part of IT Asset Management, Users should be able to programmatically determine whether there is a new update available.
  • It needs to be installable without administrative privileges on Windows, due to the many users who use AWS CLI who do not have administrative privileges to their own computer. Git for Windows does this - it is not that hard.

If these three requirements can be met by AWS, I see no problem with the lack of PyPI. Making an image available addresses the first two requirements, but the 3rd requirement is not met because many Windows users still are not allowed to use WSL.

@tmccombs
Copy link

It has to be published in a package format where we can programmatically determine updates. This is part of Software Supply Chain management for developers & DevOps, but also part of IT Asset Management, Users should be able to programmatically determine whether there is a new update available.

This is this biggest thing for me. I'd actually prefer having it in an apt repo to PyPi, but the current only official deployment mechanism is... not good. I also don't love that an entire python runtime is bundled with it. Or that if you upgrade the aws-cli, it leaves the old version lying around.

@zerthimon
Copy link

zerthimon commented Feb 25, 2021

Well, we are at 2021 and rely on a zip file and bash script with couple cps and lns to install python libraries, how cool is that ?
Seriously, I now need to figure out how to install it with a SaltStack on all my ec2 instances, making it idempotent and upgradable.
Thanks for thinking about making it easier for DevOps.

@boltronics
Copy link

It's so easy to upgrade awscli v1.

pipx upgrade awscli

Done. No root privileges required. It's what all the paying AWS customers want.

Instead we get this 20 step process which requires a dedicated installation page to explain (instead of a couple lines of the README file) which discourages upgrading. This probably creates more of a risk than anything else honestly.

I too am probably going to want to write a Salt state to manage this, but how many AWS customers are going to bother with that? Shouldn't AWS be taking on the inconveniences themselves instead of trying to push them onto paying customers?

Installing with pip requires Python knowledge.

Then continue to provide the already existing installation methods too.

Depending on a C extension can mean that at pip install time a C compiler and any other required libraries may need to be present for the install to succeed.

If you're already supplying another installation method, what does it matter? It's the user's decision.

Supporting pip as an installation mechanism requires supporting large dependency ranges for the packages we depend on.... See #4749 #4561 #3535

Just say you won't support anyone who isn't using a dedicated virtualenv environment for awscli. That would prevent you from dealing with all of those issues.

There are some things we have no control of when installing via pip. For example, we have no influence on the openssl version or what TLS related features are supported as those will depend on how Python was built.

Yes but it's not that hard to check and fail installation if your requirements are not met, with the appropriate error message. I think most users wouldn't have a problem, unless you decide to rely on some bleeding edge library feature (which wouldn't be a great idea anyway).

We cannot include or ship additional binaries (that may not even be written in Python) as part of an AWS CLI install.

Then have awscli initiate a check for whatever dependencies there might be (if ever that becomes a thing) at runtime where required. It's not a big deal.

But even if it was, this isn't your typical free software project where people can only hope, beg and perhaps offer a donation to help persuade a change in how things are done. In this case, you are getting paid to write this library because we are all paying for the AWS service. Hence I would expect AWS developers to take on some of the inconvenience to satisfy customers.

Well that's my thoughts on the matter. I'm a bit frustrated because I went to use aws ecr get-login as per the docs it wouldn't work, and I had to investigate why pipx wasn't upgrading my awscli setup, until I stumbled upon this huge issue.

@ShadowJonathan
Copy link

ShadowJonathan commented Feb 26, 2021

@joguSD any response yet? Or are y'all explicitly ignoring this issue, even when it's attracting a lot of attention (even today) because you're ignoring one of the oldest language's best practices? Do you all really think you're so above that to not bother with these "silly practices"?

I re-read your first message, if you're not going to use python for awscli, commit, don't fuck around and say "oh but we might not do that", it's been a year, people are still annoyed at this not being on pypi, put it on pypi.

@ck3mp3r
Copy link

ck3mp3r commented Feb 26, 2021

Ironically I can see a few of the Amazon leadership principles having been neglected and even broken with how they treat this issue. For one I don't see "Customer Obsession" being respected with this approach...

@kellertk
Copy link
Contributor

Hi everyone. First off, thank you all for the frank feedback about this decision, particularly specific feedback from some on this issue about why this matters to you. While @joguSD has already detailed why the team decided to not support pip, the continuing feedback is very important to us and we do use these anecdotes in future planning.

However, I would like to ask that we try to keep this conversation civil and constructive. Just as a friendly reminder, contributions to this repository, including issue comments, are governed by the Amazon Open Source Code of Conduct.

For our part, I can promise that we're reconsidering this decision internally based on the feedback we've received here. We want to consider the implications of a such a change (see @joguSD's comment linked above), so that will take some time. Expect an update from the team on that front by the end of next week.

@ericfrederich
Copy link

ericfrederich commented Mar 4, 2021

The aws cli is not without bugs. There have been times I've had to step my way through a command in a debugger like PyCharm to find the issue. For this, it is very useful to be able to install via pip or pipx.

Even if I clone this project and check out v2 branch I cannot install with pip install -e . because it's depending on some non-published botocore==2.0.0dev97

@allenatdecisiv
Copy link

Even if I clone this project and check out v2 branch I cannot install with pip install -e . because it's depending on some non-published botocore==2.0.0dev97

@ericfrederich,

I am guessing that is by design and intentional? They aren't tagging v2.x.x releases in botocore. To find the "2.0.0dev97 release" (ahem) of botocore you must read the commit log messages from the v2 branch. Then check out out the commit. Then build a wheel like python setup.py bdist_wheel. Then install or upgrade via pip (with the corresponding awscli v2 wheel). YMMV...

@shaunhickson
Copy link

Hi all - I’m the manager of the AWS CLI team, and I wanted to address the conversation that’s been going on here, as well as some broader issues. I appreciate everyone who’s taken the time to comment. Every comment on every issue in this repository is read by multiple members of the team, and we value every anecdote and data point about how our product works or doesn’t work for you. One of our goals for the rest of 2021 is to do a better job of making that communication bi-directional and to more directly engage with the community on longer-term plans and roadmaps. I’ll discuss some of our plans for that, but first I want to address the issue at hand.

We do not plan to publish the AWS CLI project to PyPI at this time. I appreciate that this will not be a popular decision among people following this issue. In Q2, we plan to begin work on a source bundle that is well documented, easily ingestible by package managers, and that can be used to build the CLI for platforms other than those we support out of the box. This source distribution will include botocore along with the CLI. Our hope is that providing an officially-supported source distribution will alleviate many of the frustrations that you have experienced with our current approach. We understand that we need to make our product easily available for those who can’t or don’t want to use one of our pre-packaged binaries or container images. Once we’ve released and validated the source bundle, we’ll open an issue here to prioritize distribution channels that will provide the most value and ensure that maintainers have what they need from us.

Starting in April, we’re going to begin posting monthly roadmap updates as issues on our GitHub repository. These will lay out what we’re actively working on and what’s on the horizon. Our backlog includes a mix of priorities, some driven by the CLI team and the community, some which align with larger AWS goals. Our focus will be on sharing the priorities emerging from our team, which will include Github issues we are planning to work on. We will iterate on the exact format and approach for these updates as we determine what works best, but they will be at least monthly for the rest of 2021.

I’m going to lock this issue now, and we will be closing any new iterations of it as duplicates. I would ask that everyone please keep in mind the Code of Conduct https://aws.github.io/code-of-conduct when using GitHub to communicate with the team.

@aws aws locked and limited conversation to collaborators Mar 9, 2021
@tim-finnigan tim-finnigan added the p2 This is a standard priority issue label Nov 3, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
feature-request A feature should be added or improved. installation p2 This is a standard priority issue v2
Projects
None yet
Development

No branches or pull requests