Allow transforming URLs before requesting #8084

kkaefer · 2017-02-16T17:50:25Z

An alternative to #7485. This implements a way of modifying the URLs that get requested from the internet. Potential use cases are altering URLs or hostnames and supporting use cases as described in #7455

This implementation invokes the supplied callback on the main thread (where the DefaultFileSource:: setResourceTransform happens) to avoid threading issues. I benchmarked the time it takes to transform the URL, and typically got times <1ms roundtrip (for scheduling the callback on the main thread and returning the result to the OnlineFileSource thread), but it can sometimes be higher when the main thread is busy rendering the map.

To do:

Android implementation
Unit tests

Caveats:

Does not allow transforming asset:// and file:// URLs; the transform is only called on requests that go out to the internet
When a request is scheduled, the internet connection goes offline for a long time, and then goes back online, it'll use the URL that was transformed during initialization; it does not attempt to transform the URL again when the request is attempted again

When using this feature, note that requests will still be cached by their canonical path, and not the transformed path. This means that the transformed URL should point to the same resources (e.g. on a different server, or with an time-limited access token) to avoid a caching mismatch. This is up to the server operator and application developer to ensure.

1ec5

This API looks good, and ensuring main thread execution will make it much harder for a developer to mess up. In addition to the comments below, please add a note to the iOS and macOS changelogs, since MGLOfflineStorageDelegate is fashioned as a public API.

1ec5 · 2017-02-16T19:46:19Z

platform/darwin/src/MGLFoundation.mm

+
+/// Initializes the run loop shim that lives on the main thread.
+void MGLInitializeRunLoop() {
+    static mbgl::util::RunLoop mainRunLoop;


If the intent is to ensure that this object gets constructed once per process and lives for the lifetime of the process, consider a load method or C++ static initializer, which doesn’t have to get called explicitly. Of the list on that page, +[MGLOfflineStorage load] or an __attribute__(constructor) function seem to be the most desirable options. (We have an example of a +load method in MGLAccountManager.) Avoid a framework initializer, since running C++ code in a framework initializer is a no-no.

1ec5 · 2017-02-16T19:48:21Z

platform/darwin/src/MGLFoundation_Private.h

@@ -0,0 +1,3 @@
+#import "MGLFoundation.h"
+
+void MGLInitializeRunLoop();


Moving this method closer to the call site would make its purpose clearer. Otherwise, a brief documentation comment would be helpful.

Removed this method

1ec5 · 2017-02-16T19:49:11Z

platform/darwin/src/MGLOfflineStorage.h

+/**
+ The receiver’s delegate.
+
+ An offline storage sends messages to its delegate to allow it to transform URLs


Nit: “An offline storage object”

1ec5 · 2017-02-16T19:50:32Z

platform/darwin/src/MGLOfflineStorage.h

+ @param url The original URL to be transformed.
+ @return A URL that will now be downloaded.
+ */
+- (NSURL*)offlineStorage:(MGLOfflineStorage*)storage


Nit: space before * for consistency.

1ec5 · 2017-02-16T19:51:59Z

platform/darwin/src/MGLOfflineStorage.mm

+                kind = MGLResourceKindSpriteJSON;
+                break;
+            default:
+                kind = MGLResourceKindUnknown;


Initialize kind to MGLResourceKindUnknown where you declare it above, and remove this default case. That way, we’ll get a warning here when a new mbgl::Resource::Kind is added.

1ec5 · 2017-02-16T19:53:43Z

platform/darwin/src/MGLOfflineStorage.mm

+                                                            encoding:NSUTF8StringEncoding]];
+            MGLResourceKind kind;
+            switch (res.kind) {
+            case mbgl::Resource::Kind::Style:


If you have autoindentation turned off in Xcode, you might want to select this code and hit ⌃I to reindent it. Otherwise, anyone touching this code will end up reindenting individual lines haphazardly when they hit : etc.

1ec5 · 2017-02-16T19:57:07Z

platform/darwin/src/MGLTypes.h

@@ -85,6 +85,19 @@ typedef NS_OPTIONS(NSUInteger, MGLMapDebugMaskOptions) {
 #endif
 };

+/** TODO docs */
+typedef NS_ENUM(NSUInteger, MGLResourceKind) {


In general, we keep enumeration definitions close to the classes that use them, which in this case would be in MGLOfflineStorage.h. MGLMapDebugMaskOptions above was moved here in order to avoid duplication between iOS and macOS, while MGLUserTrackingMode is a mistake – it doesn’t even make sense for macOS.

1ec5 · 2017-02-16T19:58:22Z

platform/darwin/src/MGLTypes.h

+typedef NS_ENUM(NSUInteger, MGLResourceKind) {
+    /** Unknown type */
+    MGLResourceKindUnknown,
+    /** TODO: docs */


As you document these values, you might want to link to relevant portions of the style specification (example), since we don’t document these resources anywhere else in the SDK.

kkaefer · 2017-02-17T17:14:38Z

Here's how to transform URLs on Android:

diff --git a/platform/android/MapboxGLAndroidSDKTestApp/src/main/java/com/mapbox/mapboxsdk/testapp/MapboxApplication.java b/platform/android/MapboxGLAndroidSDKTestApp/src/main/java/com/mapbox/mapboxsdk/testapp/MapboxApplication.java
index a10c6ea..80877e0 100644
--- a/platform/android/MapboxGLAndroidSDKTestApp/src/main/java/com/mapbox/mapboxsdk/testapp/MapboxApplication.java
+++ b/platform/android/MapboxGLAndroidSDKTestApp/src/main/java/com/mapbox/mapboxsdk/testapp/MapboxApplication.java
@@ -4,13 +4,16 @@ import android.app.Application;
 import android.os.StrictMode;
 
 import com.mapbox.mapboxsdk.Mapbox;
+import com.mapbox.mapboxsdk.storage.Resource;
+import com.mapbox.mapboxsdk.offline.OfflineManager;
+import com.mapbox.mapboxsdk.offline.OfflineManager.ResourceTransformCallback;
 import com.squareup.leakcanary.LeakCanary;
 
 import timber.log.Timber;
 
 import static timber.log.Timber.DebugTree;
 
-public class MapboxApplication extends Application {
+public class MapboxApplication extends Application implements ResourceTransformCallback {
 
   @Override
   public void onCreate() {
@@ -38,6 +41,11 @@ public class MapboxApplication extends Application {
       .build());
 
     Mapbox.getInstance(getApplicationContext(), getString(R.string.mapbox_access_token));
+    OfflineManager.getInstance(getApplicationContext()).setResourceTransform(this);
+  }
+
+  public String onURL(@Resource.Kind int kind, String url) {
+    return url;
   }
 
   private void initializeLogger() {

and here's how to do it on iOS:

diff --git a/platform/ios/app/MBXAppDelegate.h b/platform/ios/app/MBXAppDelegate.h
index 7ff321b..f70973e 100644
--- a/platform/ios/app/MBXAppDelegate.h
+++ b/platform/ios/app/MBXAppDelegate.h
@@ -1,8 +1,8 @@
-#import <UIKit/UIKit.h>
+#import <Mapbox/Mapbox.h>
 
 extern NSString * const MBXMapboxAccessTokenDefaultsKey;
 
-@interface MBXAppDelegate : UIResponder <UIApplicationDelegate>
+@interface MBXAppDelegate : UIResponder <UIApplicationDelegate, MGLOfflineStorageDelegate>
 
 @property (strong, nonatomic) UIWindow *window;
 
diff --git a/platform/ios/app/MBXAppDelegate.m b/platform/ios/app/MBXAppDelegate.m
index c2834bf..b8cfd56 100644
--- a/platform/ios/app/MBXAppDelegate.m
+++ b/platform/ios/app/MBXAppDelegate.m
@@ -23,7 +23,16 @@ - (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(
         [MGLAccountManager setAccessToken:accessToken];
     }
 
+    [[MGLOfflineStorage sharedOfflineStorage] setDelegate:self];
+
     return YES;
 }
 
+- (NSURL *)offlineStorage:(MGLOfflineStorage *)storage
+    URLForResourceOfKind:(MGLResourceKind)kind
+                 withURL:(NSURL *)url {
+    // Change the URL here if required.
+    return url;
+}
+
 @end

The approach is very similar in both SDKs: implement the delegate/interface and provide a method that gets called whenever a URL needs transforming. The URL is called on the main thread.

kkaefer · 2017-02-17T17:16:29Z

This PR also includes a bare-bones version of #4386 for Android. Long term, we should align the iOS and Android implementations and move the ownership of DefaultFileSource to OfflineManager.

kkaefer · 2017-02-17T17:27:01Z

Instead of passing a Resource&&, we could also pass a const Resource&, and expect a std::string as a return value. This would reflect the fact that the url is the only thing that can be changed (changes to all other values are ignored, since their values have been used prior to this call to construct this URL). I.e. it's not meaningful to change kind, necessity, or tileData, and the user probably shouldn't change priorModified, priorExpires and priorEtag either.

1ec5

iOS and macOS changes look great. Thanks for all the iterations on this feature.

1ec5 · 2017-02-17T18:40:36Z

platform/darwin/src/MGLOfflineStorage.h

+    MGLResourceKindUnknown,
+    /** Style sheet JSON file */
+    MGLResourceKindStyle,
+    /** TileJSON file as specified in https://www.mapbox.com/mapbox-gl-js/style-spec/#root-sources */


Use an <a> tag with the text "Mapbox Style Specification", so that the documentation looks cleaner in the jazzy documentation.

I suppose we could alternatively refer to the TileJSON file as the file specified by the configurationURL parameter in MGLTileSource's initializers.

1ec5 · 2017-02-17T18:43:58Z

platform/darwin/src/MGLOfflineStorage.h

+ @return A URL that will now be downloaded.
+ */
+- (NSURL *)offlineStorage:(MGLOfflineStorage *)storage
+    URLForResourceOfKind:(MGLResourceKind)kind


Nit: adding the space before the asterisk caused these colons to become misaligned. 😅

jfirebaugh

Looks good!

Instead of passing a Resource&&, we could also pass a const Resource&, and expect a std::string as a return value.

This seems like a good idea. The SDK bindings enforce this already, which is good.

…e Internet

…RLs prior to internet requests

ivovandongen

LGTM. Although I'd prefer having the setResourceTransform on a separate class from OfflineManager since it's not specific to offline. But we can pick that up in: #8083

kkaefer · 2017-02-20T15:22:48Z

Longer term, we should follow the iOS model, which has a wrapper class for DefaultFileSource (which is essentially what OfflineManager is). On iOS, it's called MGLOfflineStorage, but it's kind of a misnomer since it's also used in online environments. A better name would've been MGLStorageManager. On Android, we could e.g. rename OfflineManager to StorageManager, and have a Map object take a reference to a StorageManager on construction.

1ec5 · 2017-02-20T21:44:59Z

On iOS, it's called MGLOfflineStorage, but it's kind of a misnomer since it's also used in online environments. A better name would've been MGLStorageManager. On Android, we could e.g. rename OfflineManager to StorageManager, and have a Map object take a reference to a StorageManager on construction.

Reopened #4250 (comment) to track the renaming discussion.

1ec5 added Core The cross-platform C++ core, aka mbgl feature labels Feb 16, 2017

1ec5 suggested changes Feb 16, 2017

View reviewed changes

1ec5 added this to the ios-v3.5.0 milestone Feb 16, 2017

kkaefer force-pushed the url-transform branch from c42b52d to 96c60ca Compare February 17, 2017 10:53

kkaefer requested review from 1ec5, jfirebaugh, ivovandongen and tobrun February 17, 2017 17:24

kkaefer force-pushed the url-transform branch from 0cda439 to 36f9f93 Compare February 17, 2017 17:39

1ec5 approved these changes Feb 17, 2017

View reviewed changes

jfirebaugh approved these changes Feb 17, 2017

View reviewed changes

kkaefer added 2 commits February 20, 2017 11:47

[core] add ability to transform URLs before they're requested from th…

59c2419

…e Internet

[darwin] move RunLoop initialization to static variable

0aa261a

kkaefer force-pushed the url-transform branch from 36f9f93 to 6ac4963 Compare February 20, 2017 11:08

kkaefer added 5 commits February 20, 2017 13:11

[macos,ios] expose callback for transforming URLs

7f9aa40

[android] use shared DefaultFileSource for all maps + OfflineManager

1232f48

[android] OfflineManager#setResourceTransform to allow transforming U…

372a12e

…RLs prior to internet requests

[test] add unit test for DefaultFileSource::setResourceTransform()

53b7c70

[macos,ios] add tests for MGLOfflineStorageDelegate

4c052e7

kkaefer force-pushed the url-transform branch from 6ac4963 to 4c052e7 Compare February 20, 2017 13:03

kkaefer merged commit f8ab327 into master Feb 20, 2017

kkaefer deleted the url-transform branch February 20, 2017 13:36

ivovandongen reviewed Feb 20, 2017

View reviewed changes

This was referenced Feb 20, 2017

[ios] Feature/cachebykey #8041

Closed

Allow setting URL scheme templates #7485

Closed

1ec5 mentioned this pull request Feb 20, 2017

Offline terminology #4250

Closed

jfirebaugh mentioned this pull request Mar 6, 2017

Tiles are cached by url template in the database #7455

Closed

kkaefer mentioned this pull request Mar 8, 2017

Darwin static libs: don't rely on static variable initialization #8321

Merged

kkaefer mentioned this pull request May 23, 2017

Resource transforms: alter URLs before requesting mapbox/mapbox-gl-js#4740

Closed

rinigus mentioned this pull request Oct 5, 2017

[Qt] ResourceTransform interface is missing in Qt bindings #10130

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow transforming URLs before requesting #8084

Allow transforming URLs before requesting #8084

kkaefer commented Feb 16, 2017 •

edited

Loading

1ec5 left a comment

1ec5 Feb 16, 2017

1ec5 Feb 16, 2017 •

edited

Loading

kkaefer Feb 17, 2017

1ec5 Feb 16, 2017

1ec5 Feb 16, 2017

1ec5 Feb 16, 2017

1ec5 Feb 16, 2017

1ec5 Feb 16, 2017

1ec5 Feb 16, 2017

kkaefer commented Feb 17, 2017

kkaefer commented Feb 17, 2017

kkaefer commented Feb 17, 2017

1ec5 left a comment

1ec5 Feb 17, 2017

1ec5 Feb 17, 2017

1ec5 Feb 17, 2017

jfirebaugh left a comment

ivovandongen left a comment

kkaefer commented Feb 20, 2017

1ec5 commented Feb 20, 2017

		@@ -0,0 +1,3 @@
		#import "MGLFoundation.h"

		void MGLInitializeRunLoop();

Allow transforming URLs before requesting #8084

Allow transforming URLs before requesting #8084

Conversation

kkaefer commented Feb 16, 2017 • edited Loading

1ec5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1ec5 Feb 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkaefer commented Feb 17, 2017

kkaefer commented Feb 17, 2017

kkaefer commented Feb 17, 2017

1ec5 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jfirebaugh left a comment

Choose a reason for hiding this comment

ivovandongen left a comment

Choose a reason for hiding this comment

kkaefer commented Feb 20, 2017

1ec5 commented Feb 20, 2017

kkaefer commented Feb 16, 2017 •

edited

Loading

1ec5 Feb 16, 2017 •

edited

Loading