Skip to content
This repository has been archived by the owner on Aug 8, 2023. It is now read-only.

Computed (on-demand) shape source #6940

Closed

Conversation

JesseCrocker
Copy link
Contributor

Implements new API proposed in #6861 for a custom vector source, that queries a class supplied by the client application for the features to display on a tile.

This defines a new class in core, mbgl::style::CustomVectorSource, and a corresponding new source class for iOS/MacOS MGLCustomVectorSourceDataSource. There is also an new interface defined i obj-c, MGLCustomVectorSourceDataSource that defines a single method - (void)getTileForZoom:(NSInteger)zoom x:(NSInteger)x y:(NSInteger)y callback:( void (^)(NSArray<id <MGLFeature>>*) )callback

I see 4 uses for this:

  1. Data that is generated dynamically. A trivial example of this would be a lat/lon grid, that data is very simple, but a grid with a resolution of 0.01 degrees contains 216000 lines.
  2. Data that is too large to fit in memory. If data has a bounded size, it can be used as a GeoJSON sources, but if the data can be arbitrarily large converting it to GeoJSON may not be possible, for example shapefiles.
  3. Data that is fetched from the network, but not as Mapbox Vector Tiles. With this API you could implement an OSM editor, or a WFS client.
  4. Data that already exists in a format with some sort of spatial index, where querying data for a single tile is faster than fetching all of the data. For example, a set of POI data stored in an SQLite database.

My use case for this is displaying GPS traces, which is #2 and #4 from the above list. I develop an application where users frequently have hundreds of GPS traces, but occasionally have tens of thousands, and some of them have hundreds of thousands of points per trace. The data can also be modified, either by the user, or by a background sync process, in either case the data will need to be redisplayed.

@mention-bot
Copy link

@JesseCrocker, thanks for your PR! By analyzing the history of the files in this pull request, we identified @boundsj, @1ec5 and @frederoni to be potential reviewers.

Copy link
Contributor

@1ec5 1ec5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some cursory feedback for starters.

@param x
@param callback A block to call with the data that has been fetched for the tile.
*/
- (void)getTileForZoom:(NSInteger)zoom x:(NSInteger)x y:(NSInteger)y callback:( void (^)(NSArray<id <MGLFeature>>*) )callback;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename "zoom" to "zoomLevel" for consistency with other SDK APIs. Should x and y be unsigned? Rename "callback" to "completionHandler" for consistency.

/**
Request that the source reloads a tile. Tile will only be reloaded if it is currently on screen.
*/
- (void)updateTile:(NSInteger)z x:(NSInteger)x y:(NSInteger)y;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to -setNeedsUpdateAtZoomLevel:x:y:.

extern const MGLGeoJSONSourceOption MGLGeoJSONSourceOptionSimplificationTolerance;


@interface MGLGeoJSONSourceBase : MGLSource
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an unfortunate name for a class. Maybe someone else has an idea for what to call it. Also, it seems like it should declare some initializers and properties related to the options declared above.

impl(static_cast<Impl*>(baseImpl.get())) { }

void CustomVectorSource::setTileData(uint8_t z, uint32_t x, uint32_t y, const mapbox::geojson::geojson& geoJSON) {
impl->setTileData(z, x, y, geoJSON);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indent by four spaces.


- (instancetype)initWithIdentifier:(NSString *)identifier options:(NS_DICTIONARY_OF(NSString *, id) *)options
{
if (self = [super initWithIdentifier:identifier]) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use four spaces per tab stop.

@param zoom
@param y
@param x
@param callback A block to call with the data that has been fetched for the tile.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fill in this documentation and rename callback. Point out that completionHandler must be called. Incidentally, is it absolutely necessary to make this API asynchronous? A synchronous data source method would be far less error-prone.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't need to be async, since it's called on it's own NSOperationQueue. But it definitely would not work to have be it be synchronous on the main queue. Should i change this to be synchronous?

@end

/**
A source for vector data that is fetched 1 tile at a time. Usefull for sources that are
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/Usefull/Useful/

/**
Reload all tiles.
*/
- (void)reload;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to -reloadData for consistency with Cocoa classes like UITableView.

/**
Request that the source reloads a tile. Tile will only be reloaded if it is currently on screen.
*/
- (void)setNeedsUpdateAtZoomLevel:(NSUInteger)z x:(NSUInteger)x y:(NSUInteger)y;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn’t usually obvious to the developer what to pass in as x and y. Would it be possible to rework this method to take an MGLCoordinateBounds, à la -[MKOverlayRenderer setNeedsDisplayInMapRect:zoomScale:]?

proxyType = 1;
remoteGlobalIDString = DA25D5B81CCD9EDE00607828;
remoteInfo = settings;
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this change. The code signing issue was fixed in #6948.

@@ -2061,11 +2078,6 @@
/* End PBXSourcesBuildPhase section */

/* Begin PBXTargetDependency section */
DA25D5C81CCDA0C100607828 /* PBXTargetDependency */ = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this change too.

@@ -53,3 +53,4 @@ FOUNDATION_EXPORT const unsigned char MapboxVersionString[];
#import "NSValue+MGLAdditions.h"
#import "MGLStyleValue.h"
#import "MGLTileSet.h"
#import "MGLCustomVectorSource.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also import MGLGeoJSONSourceBase.h.

/**
Request that the source reloads a tile. Tile will only be reloaded if it is currently on screen.
*/
- (void)setNeedsUpdateAtZoomLevel:(NSUInteger)z x:(NSUInteger)x y:(NSUInteger)y;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, maybe this should be called -reloadTileInCoordinateBounds:zoomLevel:], for consistency with -reloadData.

/**
An object that implements the `MGLCustomVectorSourceDataSource` protocol that will be queried for tile data.
*/
@property (nonatomic, readonly, copy) NSObject<MGLCustomVectorSourceDataSource> *dataSource;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Properties that refer to data sources should be weak to avoid a retain cycle, should not copy, and should be id rather than NSObject. Is there a reason it’s read-only? It’s unusual for a data source to be used on a custom queue.

@JesseCrocker JesseCrocker force-pushed the feature/custom-vector-source branch from a614fe3 to 8cba5ea Compare November 10, 2016 12:46
@1ec5 1ec5 added Core The cross-platform C++ core, aka mbgl feature iOS Mapbox Maps SDK for iOS macOS Mapbox Maps SDK for macOS runtime styling labels Nov 12, 2016
Copy link
Contributor

@1ec5 1ec5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m excited to see this PR come together. Hope you don’t mind my continued waffling on API naming. Trying to get this right so we don’t confuse people who are familiar with Cocoa APIs.

Don’t forget to update the iOS and macOS changelogs and add any new classes to the jazzy configuration files so they show up at the right place in the table of contents.

So far I’ve only looked at the Objective-C side of these changes. @kkaefer, could you have a look at the C++ side? To kick things off, how about some tests? 😄

@param y
@param x
*/
- (NSArray<id <MGLFeature>>*)getTileForZoomLevel:(NSUInteger)zoomLevel x:(NSUInteger)x y:(NSUInteger)y;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this is a simple getter, it should be called -featuresInTileAtX:y:zoomLevel:, without the “get”.

I think it would be just as generally useful to have a -featuresInCoordinateBounds:zoomLevel:, per #6940 (comment), to accommodate those migrating from MKOverlayRenderer. If you think it’s worth keeping a tile coordinate–based API around, perhaps both methods could be optional methods on this protocol. MGLCustomVectorSource would end up calling one or the other, depending on which one is implemented.

@param dataSource An object that implements the `MGLCustomVectorSourceDataSource` protocol that will be queried for tile data.
@param options An `NSDictionary` of options for this source.
*/
- (instancetype)initWithIdentifier:(NSString *)identifier dataSource:(NSObject<MGLCustomVectorSourceDataSource>*)dataSource options:(NS_DICTIONARY_OF(MGLGeoJSONSourceOption, id) *)options NS_DESIGNATED_INITIALIZER;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Cocoa, data sources aren’t normally required; when absent, the class that normally uses the data source assumes no data is available. By following that model here, it’ll be possible to set up a custom vector source before setting up its data source, which may be handy if the data source takes some time to set up. This would entail removing the dataSource: parameter from this method and making the dataSource property nullable.

Fetch data for a tile
@param zoom
@param y
@param x
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When filling in the documentation, make sure to mention that this method is always called on the operation queue in the requestQueue property.

@JesseCrocker
Copy link
Contributor Author

@1ec5 Thanks for working with me on this, I know it's added work for you that wasn't on the roadmap.
So far I'm still at less than half the time I estimated this was going to take to implement. For a codebase of this size, it's been really easy to get started on.

One issue i'm running into is when I run make iproj the mbgl target in the generated workspace doesn't include the new files i've added to core. Do I need to put those somewhere?

@JesseCrocker
Copy link
Contributor Author

@1ec5 to Implement featuresInCoordinateBounds:zoomLevel: It needs to find the bounds for a tile, it looks like mbgl::LatLngBounds already has a constructor to create a LatLngBounds from a CanonicalTileID, but mbgl/tile/tile_id.hpp can't be included in any of the obj-c++ classes, because it includes boost/functional/hash.hpp, and boost isn't in the search path for the iOS targets. Is it OK to add boost to the header search path for the iOS targets?

@1ec5
Copy link
Contributor

1ec5 commented Nov 14, 2016

One issue i'm running into is when I run make iproj the mbgl target in the generated workspace doesn't include the new files i've added to core. Do I need to put those somewhere?

Yes, after adding the files to core, run ./scripts/generate-core-files.sh.

@1ec5
Copy link
Contributor

1ec5 commented Nov 14, 2016

Is it OK to add boost to the header search path for the iOS targets?

Yes, I think it would be OK, although we’ll want to watch the SDK for any significant size increase as a result. To add Boost, add it as a Mason package like we do in core. Be sure to update the macOS configuration too.

@1ec5
Copy link
Contributor

1ec5 commented Nov 22, 2016

The parts of tile_id.hpp that depend on Boost are the three template specializations of hash at the bottom. Maybe they could be split into a separate header; that would enable the SDK to include tile_id.hpp without including Boost.

@1ec5
Copy link
Contributor

1ec5 commented Nov 22, 2016

As I’ve mentioned before, we need better names for MGLCustomVectorSource and MGLGeoJSONSourceBase, especially the latter. Meanwhile, #7160 proposes renaming MGLGeoJSONSource to MGLFeatureSource, since most use cases for MGLGeoJSONSource don’t even involve actual GeoJSON source code.

So, two proposals:

Any other ideas?

/cc @incanus

@JesseCrocker JesseCrocker force-pushed the feature/custom-vector-source branch from 603f2f5 to f159953 Compare November 29, 2016 21:21
@JesseCrocker
Copy link
Contributor Author

I've added a couple of tests for this, both on the obj-c side and the c++ side, but i dont think they are really testing anything useful. Any guidance as to how to test this, especially the core part of it, would be greatly appreciated.

@JesseCrocker
Copy link
Contributor Author

After doing some more testing with this, I turned up memory corruption when a source was removed, and a tile fetch was in progress. My last commit should fix that.

Copy link
Contributor

@1ec5 1ec5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the one nit, my feedback on the SDK side has all been addressed.

@implementation MGLComputedShapeSourceTests

- (void)testInitializer {
MGLComputedShapeSource *source = [[MGLComputedShapeSource alloc] initWithIdentifier:@"id"
Copy link
Contributor

@1ec5 1ec5 Mar 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: indent by four spaces for consistency.

Do not create instances of this class directly, and do not create your own
subclasses of this class. Instead, create instances of `MGLShapeSource` or
`MGLComputedShapeSource`.
*/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, this is totally optional, but we have a mechanism for inserting a Swift code example into a documentation comment via unit tests. Feel free to use MGLShapeSource’s documentation as a model.

@1ec5 1ec5 changed the title Add Custom Vector Source class for iOS/Mac OS Computed (on-demand) shape source Mar 4, 2017
@JesseCrocker
Copy link
Contributor Author

Im still working on fixing a use-after-free issue with this.
How to trigger the problem:
Add an MGLComputedShapeSource to the style, with a dataSource that takes some time to return data.
Call [mapView setStyleUrl:] to change the map style

If any calls to the data source were in progress when the style was changed, they will end up calling setTileData on a mbgl::style::CustomVectorSource that has been deallocated.

Address sanitizer picks up the use after free, but without it turned on this manifests as all sort of random crashes from memory corruption.

I've got a good test case for this at trailbehind@1f5f51e , It calls cycleStyles:on a timer, and adds a MGLComputedShapeSource to the style in mapView:didFinishLoadingStyle:

I see 2 different ways to look at this problem:

  1. dealloc doesn't get called on MGLComputedShapeSource if there are any fetch operations in progress, because the operation has a reference to the MGLComputedShapeSource. If dealloc did get called, the MGLComputedShapeSourceFetchOperation would get canceled, and self.source.rawSource->setTileData would not get called.
  2. MGLComputedShapeSource has no way to know when its mbgl::style::CustomVectorSource has been dealloced, because removeFromMapView: does not get called, and MGLComputedShapeSource only has a raw pointer to the mbgl::style::CustomVectorSource

Fixing based on 2 seems like the better fix, but it seems like a big change that would be beyond the scope of this PR.
2 ways i would think it could be fixed based on 2:

  • Call removeFromMapView: when a style is dealloced.
  • Switch to use shared_ptr instead of unique_ptr for sources

So im guessing fixing based on 1 is the way to go, but im having trouble getting it to work right.

There's also a third option, let this crash, and add a note to the documentation that the source must be removed from the style before the style is removed from the map.

@1ec5 Do you have any suggestion for how to approach this?

@JesseCrocker
Copy link
Contributor Author

I've got the crash mentioned above fixed by removing the reference to the MGLComputedShapeSource from the MGLComputedShapeSourceFetchOperation

Copy link
Member

@kkaefer kkaefer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your pull request! This is definitely something that we want to add, but it's very hard to review this pull request since it's scattered over so many commits. We generally require all commits to only contain self-contained changes, so before merging this, we'll either have to squash this into one commit (or preferably) isolate self-contained changes in this PR and commit them individually.

I realize that you're more familiar with the iOS/macOS side of things, but we also need Android and ideally Qt bindings before adding this change, as well as a lot more changes that also test rendering of generated tiles.

}

void CustomVectorSource::Impl::setTileData(uint8_t z, uint32_t x, uint32_t y, const mapbox::geojson::geojson& geoJSON) {
double scale = util::EXTENT / util::tileSize;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

constexpr

void CustomVectorSource::Impl::setTileData(uint8_t z, uint32_t x, uint32_t y, const mapbox::geojson::geojson& geoJSON) {
double scale = util::EXTENT / util::tileSize;

if(geoJSON.is<FeatureCollection>() && geoJSON.get<FeatureCollection>().size() == 0) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.size() == 0 => !.empty()

if(geoJSON.is<FeatureCollection>() && geoJSON.get<FeatureCollection>().size() == 0) {
for (auto const &item : tiles) {
GeoJSONTile* tile = static_cast<GeoJSONTile*>(item.second.get());
if(tile->id.canonical.z == z && tile->id.canonical.x == x && tile->id.canonical.y == y) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of passing z/x/y separately, can we use CanonicalTileID and then compare the objects directly?


#pragma clang diagnostic push
#pragma clang diagnostic ignored "-Wunused-parameter"
const auto callback = ^void(uint8_t z, uint32_t x, uint32_t y) {};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we instead use C++11 lambdas rather than Apple-Clang specific syntax? You can also remove the z/x/y names to avoid the pragmas. An even better approach would be to pass a null pointer as the callback and check whether callback is invokable before calling it.

};


} // namespace std
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was this moved to a separate file? While not wrong, it doesn't look like it's necessary to include in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was split into a separate file so that tile_id.hpp could be included in objective-c++ files in the iOS/macOS side of things without having to include boost.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MGLPolylineFeature *feature = [MGLPolylineFeature polylineWithCoordinates:coords count:2];
feature.attributes = @{@"value": @(x)};
[features addObject:feature];
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this entire function something that should be in the core code rather than in a binding-specific code?

Copy link
Contributor

@1ec5 1ec5 Mar 7, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean as an implementation of #119? I don’t think it satisfies the requirements for that feature, namely that the graticule would have to be visible before the style even starts to load. But it does make for an interesting demo. (This file is part of macosapp; it isn’t part of any SDK.)

@@ -26,7 +27,10 @@ using SuperclusterPointer = std::unique_ptr<mapbox::supercluster::Supercluster>;

struct GeoJSONOptions {
// GeoJSON-VT options
uint8_t minzoom = 0;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

space

CustomVectorSource(std::string id, GeoJSONOptions options, std::function<void(uint8_t, uint32_t, uint32_t)> fetchTile);

void setTileData(uint8_t, uint32_t, uint32_t, const mapbox::geojson::geojson&);
void updateTile(uint8_t, uint32_t, uint32_t);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the difference between setTileData and updateTile? If they're distinct, can we add some documentation?

@JesseCrocker
Copy link
Contributor Author

@kkaefer Thanks for reviewing this. I think i've addressed all of your comments. Sorry about the large number of commits, i had to do some iteration to make this work, and then there were a bunch of commits renaming things based on review feedback.

Im well versed in Android development and capable of doing the Android bindings for this, but probably wont have time to get to it for a couple months.

@incanus incanus mentioned this pull request Mar 17, 2017
@kkaefer kkaefer mentioned this pull request Mar 20, 2017
9 tasks
@1ec5
Copy link
Contributor

1ec5 commented Mar 20, 2017

#8473 is a squashed, reorganized version of this PR.

@ychescale9
Copy link

Wondering if there's any progress (or interests from other users) on this?
We are currently looking at using MBTiles in Android (offline) and evaluating different options.
Hence we would like to know if adding support for MBTiles source is on the roadmap. Thanks!

@jfirebaugh
Copy link
Contributor

#8473.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Core The cross-platform C++ core, aka mbgl feature iOS Mapbox Maps SDK for iOS macOS Mapbox Maps SDK for macOS runtime styling
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants