WIP - Adds Google cloud storage support

[francisdb]

Superseded by #1340

fixes #588

This is an initial import of the code we currently have in (pre)production at https://www.waylay.io . Kudos to @brunoballekens

Some questions toward maintainers:

• Can we keep using play-json?
• Can we keep using specs2?
• Is it a good idea to make a difference between Source.empty (404) and Source.single(ByteString.empty) (empty file)? Other solution would be returning Future[Option[Source[ByteString], Metadata]]] which makes things rather ugly
• Do you have a guide on best practices for coming up with a java api?
• Please review TODO's to see if anything is missing

[francisdb]

@ennru mind answering the questions I put in the comment?

[ennru]

Great that you'd like to donate this code to Alpakka.

• Akka Http's spary-json would be a more "natural" choice. Does it provide all you need?
• Scalatest is preferred as we've no other tests in specs2. Is is too much work to migrate?
• I'd say the distinction between not found and empty file is important.
• We don't really have a guide how to offer a Java API. We strive for feature equality with Java-esque interface (Optional, java.util.List).

I'll need to look at the code to give more guidance...

[francisdb]

I'll switch to scalatest and spray-json, should not be that much work

[johanandren]

Would be good if you could also read through the brand new contributor advice https://github.com/akka/alpakka/blob/master/contributor-advice.md and adjust public/internal APIs etc as needed, thanks!

[francisdb]

@ennru @johanandren I would like to share the Session class from the googlecloud-pub-sub project. Any recommendations there?

see also GoogleSession in #764

[ennru]

Yes, sharing sounds good to me. How much code could be shared? Is it "just" the session class?

We'd either need to add a google-cloud-base module or let one of the connectors be the "main" module and let the others depend on it.

[tg44]

Yapp sharing could be good. Centralize the deps, create a "common" way to do things. But on the other hand we are speaking about ~90 line of code. So I still not convinced myself.
The only thing we shouldn't do; is using the official google sdk-s as deps. I walked through some of their code and trust me, you really don't want to depend on that :)

So I think for the start we could implement the same session and token handling for the 3 modules, and we should generalize/centralize this later. (Mostly because if we do a refactor in one PR we need to do that on the other related PRs too. If we have a more "stable/freezed" codebase we could refactor more easily.) Maybe start a new issue with this "google shared code" problem, and we will not forget it.

[francisdb]

@tg44 makes sense to wait until everything has been merged

[johanandren]

I think this break the build on non unixy machines. Use TempDirectory.location() and then combine with a more specific file name that will not exist.

[francisdb]

Switching to Paths.get(System.getProperty("java.io.tmpdir")).resolve("idonotexist.json") as TempDirectory is not accessible

[francisdb]

I don't have the time to work on this any more, might in the end be a better idea to create a wrapper for the official google client that now supports async callbacks?

[francisdb]

on the other hand we have been running this code in production for a while now and it does its job

[ennru]

Sorry, to see you leaving this behind. Welcome back another day.

[l15k4]

We are using it too in production

[jkobejs]

@ennru what needs to be implemented to have this PR ready for merge?
Is there anything else besides:

• Documentation
• Create proper java/scala dsl's
• Add mocked http tests
• Review configuration compared to other google module
• Switch to Host-Level Client-Side API instead of Request-Level? (Maybe)

francisdb mentioned that it might be better idea to implement this in terms of new google client that supports async callbacks, it that better approach?

@francisdb can you point me to the google client that supports async callbacks, I haven't had luck to find documentation about it yet.

[tg44]

I have no idea which one is the "new google client", but last time when I checked out the official java sdk from google, it was really ugly. Personally I would not use that lib in any production system. (It makes a lot of unneeded requests, no reasonable caching, the codebase is hardly readable and not segmented, and if you check the dependencies its really big.) So we coded all the required functionality from the REST documentation after we tried to use it and failed bcs of the performance and memory issues.

[francisdb]

I guess I mixed op Amazon and Google. There is a new async aws client. So forget about that comment.

[ennru]

@josipgrgurica Would be great if you have the time to get this rolling again. We'd like it to follow the structure as described in the contributor advice.

[jkobejs]

@ennru I have time to continue on this, I'll go through this PR and contributor advice to see what is missing and what needs to be changed. Should I fork francisdb:googlestorage branch and work on it?