S3Client to use Boto3 (#2149)

* AWS_SESSION_Token read from ENV

* add s3client

* s3client using boto3

* removed AWS creds from travis and tox env vars

* importing form contrib.s3 instead of s3 directly

* AWS credentials passed as env vars

* fixed wrong s3 import

* redshift tests passing locally

* test_ecs mocked

* changed all byte instances to strings

* writing to file as bytes

* decoding bytes to strings for python3 tests

* contrib/redshift test reading creds from env vars

* travis build on master

* flake8 redshift_test

* reverted to master redshift_test

* skipping test/redshift_test.py on travis

* flake8 test/redshift_test.py

* skipping test/redshift_test.py on travis

* moved test/redshift_test.py to test/contrib/

* redshift_test using mocked S3Target open

* AWS_SESSION_Token read from ENV

* moved redshift_test to contrib folder and refactored redshiftmanifest tests to mock calls to write

* fixed flake8 errors

* added multipart s3 tests

* add s3client

* removed AWS creds from travis and tox env vars

* importing form contrib.s3 instead of s3 directly

* AWS credentials passed as env vars

* fixed wrong s3 import

* redshift tests passing locally

* test_ecs mocked

* changed all byte instances to strings

* writing to file as bytes

* decoding bytes to strings for python3 tests

* contrib/redshift test reading creds from env vars

* travis build on master

* flake8 redshift_test

* reverted to master redshift_test

* skipping test/redshift_test.py on travis

* flake8 test/redshift_test.py

* skipping test/redshift_test.py on travis

* moved test/redshift_test.py to test/contrib/

* redshift_test using mocked S3Target open

* moved redshift_test to contrib folder and refactored redshiftmanifest tests to mock calls to write

* fixed flake8 errors

* added multipart s3 tests

* travis to trigger builds on master

* skip on travis message made clearer

* add s3client with rebase

* removed arn resolution

* s3client resolving auth alone

* logging boto3 resource call

* fixed as per PR's comments

* fix as per PR comments

* removed blank line for flake8

* removed unused OptionParameter

* add deprecated error to s3.resource

* fixed flake8

* batch tests skip on travis

* fix flake8 errors

* rebase with master

* update listdir docstring

* skip multipart test on travis

* further multipart tests to skip on travis

.travis.yml

@@ -15,6 +15,9 @@ env:
     - BQ_TEST_PROJECT_ID=luigi-travistestenvironment
     - BQ_TEST_INPUT_BUCKET=luigi-bigquery-test
     - GOOGLE_APPLICATION_CREDENTIALS=test/gcloud-credentials.json
+    - AWS_DEFAULT_REGION=us-east-1
+    - AWS_ACCESS_KEY_ID=accesskey
+    - AWS_SECRET_ACCESS_KEY=secretkey
   matrix:
     - TOXENV=flake8
     - TOXENV=docs