Support AWS Signature Version 4 (Refactored)

[windkit]

Brief Description

• Related to issue [leo_s3_libs]Authenticating requests(AWS Signature version4) to be implemented leofs#283 Support aws-sdk-go leofs#373

The flow is similar to Version 2 with two major differences

• Payload SHA-256 is used for Signature
• Chunked Upload

Related Pull Requests

• Support AWS Signature Version 4 (Refactored) leo_gateway#24
• Support AWS Signature Version 4 (Refactored) #2
• Support AWS Signature Version 4 (Added leo_hex:binary_to_hexbin) leo_commons#3

Work in Progress

• Unit Test

Using Authorization Header

Checksum Precomputed

http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

• SHA-256 Checksum Provided in HTTP Header x-amz-content-sha256
• Small Modification

Amazon Chunked Upload

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html

• "Rolling" Checksum of Chunks

Using Query Parameters

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

Using HTTP Post

http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html

[mocchira]

s/binary()/list()/

and basically you need to dialyze for type checking before PL.

[windkit]

? These parameters look binaries to me.
BTW, I will check how to use dialyzer

[mocchira]

? These parameters look binaries to me.

yep.
you are right.

BTW, I will check how to use dialyzer

At first,
issue the below command only once in the repo root. ( basically when you cloned into local )

make build_plt

issue the below command if you want to dialyze. ( basically when you finished to implement before UT )

make dialyzer

[mocchira]

@windkit LGTM. I'm waiting for UT and PR to leo_hex.

[windkit]

Sorry that I forgot to create the PR
leo-project/leo_commons#4

[windkit]

I am now writing the unit test, will work on transfer_deocde and unit test in leo_gateway afterwards

[windkit]

I have fixed the unit test for V2
(as interface of leo_s3_auth:authenticate and leo_s3_auth:get_signature are changed)

Unit Test for V4 Signing is added
http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html

[mocchira]

LGTM.

As you know that @yosukehara asked you benchmarking to compare the performance of v2 and v4,
Please share the result on the leo-project/notes repo.

• note

  The purpose of this benchmark is to inform LeoFS users how the performance is affected when using v4, so the bench should conduct against the whole system with using basho_bench and typical use cases like R:W = 8:2, Write Only, Read Only.

[yosukehara]

Regarding leo_commons, I've merged your request. You could modify the row of leo_commons in the deps as below:

{leo_commons, ".*", {git, "https://github.com/leo-project/leo_commons.git", {branch, "develop"}}},

• Added binary_trim/1 to Trim Space from Binaries leo_commons#4

[yosukehara]

I've merged your request into the 1.4-branch. Again, thanks.