Implement FileSource in Java

[kkaefer]

I think we should look at what it takes to implement the FileSource object in Java. This would allow us to remove the dependencies on CURL, OpenSSL and libzip, as well as SQLite.

With the FileSource refactor, we can now more easily swap out the storage backends: Either, we replace the FileSource object completely, which leaves us with implementing the FileSource API, which is essentially just a request/cancel method. All the underlying logic, like caching, HTTP requesting and loading from the APK file has to be implemented manually.

We could also replace only part of the API, e.g. only reimplement the HTTP logic in Java. I think we should start with the latter, but potentially move our way up the chain until we have we can get rid of all of these dependencies, especially given that they have Java APIs that use the system's built-in libs.

/cc @ljbade

[mikemorris]

yes yes yes

[ljbade]

Yeah OK I will start looking into what is required on the JNI side to handle two way comms.

[ljbade]

Looks like we should use Volley. @bleege Are you familiar with Volley?

As far as JNI goes main thing is to pay attention to object references with local/global scope and ensuring Java attached to any threads.

[ljbade]

Also taking a look at using JNI with arrays. A good overview is here. However it does not mention these functions that should be faster.

Some JNI tips from IBM.

Interesting way to remove the need to call Java from native: http://nerds-central.blogspot.com/2012/04/extreme-jni-performance.html

Finally I need to read up on ByteBuffer

[bleege]

Looks like we should use Volley. @bleege Are you familiar with Volley?

We looked at Volley for the Android Raster SDK last February and again last Summer. We ultimately decided not to use it as it was a bit murky in terms of documentation / stability (see links).

The Raster SDK currently uses Square's OkHttp for networking. That said we're now leaning towards using the Glide Framework based on some research in January for V4 support. The big reason is that it uses stock Java networking classes but can also make use of Volley and OkHttp if desired and has great caching support.

That said it looks like Google's finally cleaned that those issues with the Tutorial / Overview that you found @ljbade. I'm not also familiar with GL's FileSource object so if you're feeling like Volley would work well for that then it's certainly worth kicking the tires again.

[kkaefer]

The FileSource object's API is just three functions: a request() function that will request the URL, returns a cancelable Request object, and will run the provided callback in the provided loop. A request() function that only takes a URL and a callback, and will call the callback in an arbitrary thread, and finally a cancel() function that takes a Request object, and is supposed to cancel the request. In terms of the underlying implementation, it does not mandate anything, except that the functions must be threadsafe.

[bleege]

@ljbade and I just had a great discussion on the current state of Android GL and what a solution for this problem would need / look like. This was super helpful to understand what type of content was going to be downloaded, if it needed to be cached, etc as many of the Android networking libraries are tuned to specific features (ex: Glide for downloading and caching images) vs more generic HTTP access like OkHttp.

Based on this info, I think it makes the most sense to investigate OkHttp and Volley for use in GL. OkHttp is a much more widely used in the Android world so it's likely many people using the Android GL SDK will also be using it in their apps. This can be good as it helps keep the overall number of dependencies (and app sizes) down, but it can also be bad especially for people who hardwire the Android GL SDK into their apps (see: Eclipse users).

Summary of Discussion:

Current State

• Using native Linux networking logic

Content To Download

• Images
  • PNG and JPEG for Satellite & Maki Icons
• Fonts
• Styles (JSON)
• .pbf (Gzip compress protocol buffer)
  ** Binary files

Caching

• Done in the GL native library so not needed at the Java side

Needs

• Takes URL and returns BLOB of data
• Asynchronous

[ljbade]

@bleege spot on summary,

should also add, must support android 2.x if possible -> #555

[bleege]

Some other possibilities that have turned up during research:

• Square Retrofit for Typesafe REST API client
• ProtBuf Libraries
  • Google's Protobuf
  • Square Wire

Comparisons

• Android Async HTTP Clients: Volley vs Retrofit
• Comparison of Volley, Picasso, Retrofit, and OkHttp

[bleege]

Native API References from @mikemorris.

• file_source.hpp
• Node implementation of FileSource

[bleege]

Images and Font demos are done. Next up is Styles.

[kkaefer]

@bleege any reason we can't just treat everything as opaque binary data? It shouldn't matter to the Java request library what the contents of the files we're requesting are.

[bleege]

@kkaefer Yep, we can do that. In fact the examples I've finished so far all make use of byte arrays and then use native features to make use of them (ImageView and Typeface). The next ones are JSON which is also built in to Android and ProtoBufs, which are probably going to require an additional library (I'm currently leaning towards using Square Wire for this).

That said, the driving reason for looking at these different libraries is that the stock HTTP networking API in Android has some pretty severe limitations with some "highlights" being:

• Synchronous only requests that block
• SSL handshaking / failover issues
• Lack of connection pooling
• No automatic GZIP support

I've settled in on OkHttp from Square to build the examples described in previous comments. It gives us everything we need but is strictly HTTP / SPDY client and can provide data in binary format. This is in contrast to most open source libraries that try to solve these issues by also baking in other common addition features like custom JSON Support, Image processing, and POJO Marshaling. Since the GL Core is going to handle all this stuff we don't need it as addition bit weight in a library.

Anyway, here's the link to the my example app on GitHub called OkGL. I"m going to try to finish up the features tomorrow (including cancel support) and then hopefully refactor it to be cleaner / easier to use. Please feel free to check it out and let me know what you think.

[kkaefer]

ImageView and Typeface

The Java FileSource doesn't need to interpret the data it downloaded at all.

[bleege]

The Java FileSource doesn't need to interpret the data it downloaded at all.

Agreed. But I needed a way to test that things were behaving as they should. :-)

[bleege]

Done with JSON. Investigating ProtoBuf next.

[bleege]

Just finished building the PofC / Demo app using OkHttp. I didn't end us using a ProtoBuf library as this is likely to be processed by the Core GL engine so delivering it in Binary form is all what's needed. Was able to build out all required functionality though so IMHO OkHttp is going to be the library to use for implementing the FileSource in Java. For details please see the app's source code at:

https://github.com/bleege/OkGL

[ljbade]

@bleege what do you mean about not being able to build all functionality?

[bleege]

I've moved the completed OkGL project from my personal GitHub account to Mapbox's. I'd forgotten that the recent permissions refactor now gave me access to do so. Anyway, it'll be waiting there for us when we're ready to implement FileSource for Android GL.

https://github.com/mapbox/OkGL

[bleege]

Just built out the initial version of JavaFileSource for Android GL based on OkHttp. To use:

// Request Resource from Web, Process with custom Callback
JavaFileSource.getInstance().request("http://www.mapbox.com", new Callback() {
            @Override
            public void onFailure(Request request, IOException e) {
            }

            @Override
            public void onResponse(Response response) throws IOException {
            }
     });

// Cancel Request based on URL
JavaFileSource.getInstance().cancel("http://www.mapbox.com");

[ljbade]

Awesome, now you need to create the C++ class and use JNI to glue them together. Let me know if you need help.

[bleege]

Thanks @ljbade! I will happily take you up on your offer for assistance. 👍

[ljbade]

@bleege Done, created #2231 and #2232

[bleege]

Awesome! Thanks @ljbade!

[ljbade]

Reopening as we had to switch back to curl because of crashes (#2295).

Switching back to OkHTTP depends on fixing #2400

[jfirebaugh]

We are back on OkHTTP now.

[bleege]

Yep, OkHTTP is back via #2400 , #2856 , and #2857 .