Using dynamically-rendered vector tiles (from cartodb)

[chriswhong]

Hello Mapbox Team!

I am experimenting with the undocumented windshaft (cartodb) vector tile feature, and have successfully rendered carto vector tiles in mapboxGL. Here's an example: http://bl.ocks.org/chriswhong/2695b75fd1936bd034df83c91738648d

My question is about dynamically updating the vector tiles using the same workflow that carto.js uses to update raster tiles. This probably sounds like I am doing it wrong , but hear me out.

With carto.js, you can call setSQL() on a raster layer, which triggers leaflet to request new tiles from windshaft. It removes the current raster tile layer and replaces it with new tiles rendered off the new SQL query. I am wondering if it would be possible to update the url of a mapboxGL vector source in the same way, prompting mapboxGL to grab new tiles but leave the layer/style rules in place, essentially refreshing the data much like you can do with geoJSON layers.

I have a lot of bespoke-to-a dataset filtering UI that is essentially generating SQL filters. One solution would be to only have carto serve the vector tiles once with a SELECT *, and then just have my filter UI filter the mapboxGL layers and never talk to Carto. I assume this makes the most benefit of vector tiles, as there is no need to constantly interact with the server.

However, I also have a bunch of charts, each of which is executing SQL against the carto database table, and the SQL is dependent on the filters applied in the filtering UI. With the traditional carto workflow, I can just get new SQL from the filter UI, and use it to quickly update the raster tiles AND each of the charts without too much effort. Everything that is dependent on the new filters does its own AJAX calls and updates itself accordingly, and I can keep everything modular.

I could set a filter on the vector tile source and then use queryRenderedFeatures() on the vector tile layer, filter for unique features, and then use the resulting features to update the charts. I am hesitant to do this for a number of reasons, one being that I don't want to couple the charts too closely with the map's filtering so I can re-use them elsewhere, and another being that I don't seem to get the same totals (SELECT count(*) in the database is usually higher than getUnique(queryRenderedFeatures()).length for some reason, possibly because vector tiles are hiding some features that should not be rendered at certain zoom levels?) So, hitting the DB is best way for me to be 100% sure I have everything when doing aggregations.

Basically, I am trying to do my filtering server-side, as I would with dynamically-generated raster tiles, so that I can update the data behind the vector tile layer on my mapboxGL map. Is this possible? Is this reasonable? Any thoughts or suggestions are much appreciated!

This issue is similar to #1946

[chriswhong]

I have attempted the "download the data as GeoJSON once do as much processing as possible clientside" suggestion in #1946. The performance boost from using carto-served vector tiles for the same data was huge, and I don't think we can go back to "one big geojson" at this point. I also just really want to keep everything speaking SQL/PostGIS so we can work in lots of other custom DB queries around the filters at some point in the future.

[indus]

Why don't you just grab the code from here and hook it into you app? For a specific case the wiring should be simple and would look something like this:

var sql = new SQL();
sql.execute("<query>",vars,options, function(resp){
    GeoJSONLayerRef.setData(resp);
})

To write a more generic plugin that adds this functionality to a layer would propably be a bit more challenging but a benefit for otheres. But I would wait for #281 https://github.com/mapbox/mapbox-gl-js/projects/2 before implementing it.

Oh, sorry. I thought you where talking about a GeoJSON layer.

[lucaswoj]

It sounds like you're blocked on the ability to force GL JS to refresh its vector tile cache. Is that correct? #2633.

[chriswhong]

@lucaswoj I wouldn't even say I am blocked, my current workflow is simply to remove the vector tile source and associated layers, and add new ones every time the dataset updates. It works just fine, the only hiccup is that there is a brief "flash" between when the old layers are removed and the new ones are rendered.

What I want is the ability to update the tile URL of a vector source, having mapbox GL fetch new tiles behind the scenes but leaving the layers in place... the equivalent of how .setData() works for geoJSON.

[mollymerp]

@chriswhong Would something like this work for you?

var newStyle = map.getStyle();
newStyle.sources.<yourSourceId>.tiles = <newSourceUrl>;
map.setStyle(newStyle);

we recently merged in a feature that allows for a smooth transition between styles (no 'flash') #3621, but changing the source URL might cause a brief flash with just the background layer bc we have to request/load all the new tiles again as opposed to just changing layout/paint properties.

[chriswhong]

@mollymerp I think this is exactly what I need! Thanks, I will give it a try and report back.

[mollymerp]

great! closing here as I don't see any next actions. please re-open if you continue having issues.

[javisantana]

You might want to use vector tiles: http://bl.ocks.org/rochoa/eca69273cd2b6cf86673

[areichman]

@mollymerp In my case, I have several time-based weather layers (e.g. radar, temperature) with URLs of the form:

/mylayer/{time}/{z}/{x}/{y}.mvt (or .png)

Every few minutes, I need to update the timestamp in those URLs to fetch the most recent data. I would like to keep the logic required to update each source independent from the other sources, but each update process could obtain a stale copy of the map style that way.

So, I agree with @chriswhong that the equivalent of .setData() for the other source types would be useful.

[asheemmamoowala]

@areichman Have you considered using the transformRequest option on the Map to modify the URLs?

let currentTime = "tuesday-10am";
let map = new mapboxgl.Map({
  container: 'map',
  transformRequest: (url) => { return url.replace("TIME", currentTime); }
});    
map.addSource('vector', {
        "type": "vector",
        "tiles": ["https://mydata.com/TIME/{z}/{x}/{y}.mvt"]
    });

Note that we reuse the fetched tile until it expires, so you'll have to ensure that the tile server sends the appropriate 'Expires' header in response to the tile requests.

[nbulaj]

@asheemmamoowala wait a second.. So if Mapbox will cache the tile for time "10:00", it will render it again even with time "20:00" because cache is not expired (but the tile was changed in fact)? Why not just to change cache behavior for using some conditions (maybe from custom callback) if transformRequest is used? Another way it really a bad design (will not show actual data) and will send many requests to the server (if we set a small cache time)..