Allow aws commands to use aws environment variables for auth

[apotek]

Description

Allow aws commands to use aws environment variables for auth
Do not assume credential file creation will have valid credentials.
Begin to move away from deprecated ->io constructs.

Resolves #222

Testing

Open your composer.json file. Find the "repositories" section and add the Usher repository:

        {
            "type": "git",
            "url": "https://github.com/ChromaticHQ/usher.git"
        },

(nb: Make sure you add that comma at the end)

Then require this test branch:

ddev composer require "chromatic/usher:dev-222-database-download-trait-does-not-respect-aws-credentials-in-environment-variables"

Note: if you have already done this previously, and are coming back for more testing be sure to go to vendor/chromatic/usher in your local repo and do a git pull to get latest.

Run commands you normally use in your client project.! Report failures here! These are just examples.

  database:download <site>     
  database:refresh-ddev <site>
  dev:refresh <site>
  frontend:dev-disable               <site>  
  frontend:dev-enable               <site>
  etc...

First, make sure there are no AWS environment variables set in your container. Also remove the .aws/credentials. You should have an authentication error and be prompted to enter your credentials. Enter bad credentials. You should be prompted to re-enter your credentials. This will keep going until a successful authorization or you cancel.

Now, break your .aws/credentials by modifying key or token so that it won't work. You should see a failure again and a prompt to fix your credentials.

Break your .aws/credentials again. Now, however, create a .ddev/.env file and put GOOD aws credentials in it. Restart your ddev project. Now you should see success and not be prompted to enter the aws credentials, even though your ./aws/credentials file contains a bad token or key.

(The environment variables override the credentials file as per spec).

Now, break your env variables by breaking a key or token in your .ddev/.env file and restarting your ddev project. You should see an error about your AWS environment variables being bad and that you need to fix them.

You can then decide to either remove your environment variables and use the ~/.aws/credentials, or (recommended) just use the .env file.

Remember to remove your modifications to composer.json and composer.lock! Don't commit that change.

Expected Results

1. If you have environment variables set up and they are wrong, robo should exit with a message about your environment variables.
2. If you have environment variables set up and they are correct, the download should work and there should be no prompts for anything.
3. If you don't have environment variables and no credentials file, you should have an authentication error and a prompt to configure your credentials.
4. If you don't have environment variables and you have a credentials file but the credentials are wrong, you should have an authentication error and be prompted to configure the credentials.
5. If you don't have environment variables and you have a credentials file and it has valid credentials, there should be no error, no prompt, and it should just download.

[mmatsoo]

@apotek Your instructions all worked a charm until I got to the part to put good credentials into .ddev/.env and restart ddev. I restarted ddev and tried the robo command several times, even switching to a file called env instead of .env and I kept getting prompted to enter AWS credentials. (I tried this on my Imprivata instance.)

ddev robo database:download imprivata

database download.
==================

➜  'imprivata' database_s3_region not set. Defaulting to us-east-1.
➜  'imprivata' S3 bucket: imprivata-drupal-8
➜  'imprivata' S3 Key prefix: 'prod-imprivata'
➜  Connecting to S3...

                                                                                                                        
 [ERROR] HTTP 403 returned for "https://imprivata-drupal-8.s3.us-east-1.amazonaws.com/?list-type=2".                    
                                                                                                                        
         Code:    AccessDenied                                                                                          
         Message: Access Denied                                                                                         
         Type:                                                                                                          
         Detail:                                                                                                        
                                                                                                                        

➜  Unable to authenticate to AWS S3.
            You can either set your credentials as environment variables and try again,
            or you can continue by configuring your AWS credentials file.

 Do you wish to configure your AWS S3 credentials file? (yes/no) [yes]:

So, the part to enter credentials works great, but my .env file didn't seem to get recognized / used.

[apotek]

So, the part to enter credentials works great, but my .env file didn't seem to get recognized / used.

Does your .env file have the correct credentials, and is it formatted like this?

AWS_SECRET_ACCESS_KEY=#############################
AWS_ACCESS_KEY_ID=#####################

(ie, i have made the mistake of copying an ini file to the env file thinking).

If all this checks out, I would be interested to see if your environment variables were actually set, ie:

ddev ssh
env | grep AWS

should output the correct credentials. Is that the case? And they should persist even after reboot. Thank you for testing this. You'll have to do a git pull in vendor/chromatic/usher to get latest should you try again.

[danarbello]

Reviewed this synchronously

[mmatsoo]

@apotek It turns out my .env file needed to be like the example you gave, large caps:

AWS_ACCESS_KEY_ID = XXXXXXXXXXXXXXXXXXXX
AWS_SECRET_ACCESS_KEY = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

What I had before, when it didn't work, looked like this:

[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

I initially removed [default], but it still didn't work, so I made the keys large caps and then it began working.

[danarbello]

@mmatsoo Interesting. It should be noted that Usher copies over the creds in the latter format:

[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

So, even Usher doesn't capitalize the variable keys 🤔

[apotek]

@mmatsoo Interesting. It should be noted that Usher copies over the creds in the latter format:

[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

So, even Usher doesn't capitalize the variable keys 🤔

The AWS credentials file is in a format called an "ini" file, where a configuration section is delineated by a [section] and the variables are defined under it. You'll be familiar with this format from php.ini settings when configuring a lamp stack for example. It's a pretty widespread format and there are even native php functions to read this format parse_ini_file().

Environment variables are a completely different beast. There is, by the way no technical reason environment variables are all upper case; it's a convention, similar to naming constants in upper case too. The only reason the environment vars have to be upper case when doing an environment config, is that the aws CLI as well as the aws-async php library that Usher uses, expects the environment variables to be upper case.

The .env file and the credentials file are entirely different ways to set up your credentials for AWS and can't be interchanged.