Generated tileset.json doesn't include queries key for each layer

[polastre]

In v0.7.2-rc1, the generated tileset.json doesn't include the queries key anymore in the vector_tiles section.

We were using this key for post-processing activities. This is a serious problem for us.

Looks like #658 removed this data from the generated tileset.json file.

I can see how this might be a security vulnerability if that is exposed in production, however it is really useful for intermediate processing in development.

Can we bring it back, please? cc @bchapuis @Perdjesk

[polastre]

It looks like the production concern was already addressed prior to this change, so bringing back the queries and database information in some format in dev would be really helpful.

https://github.com/apache/incubator-baremaps/pull/658/files#diff-94ecab9df3004fae4fd05c3c610d8ef629f81cb775f72a53a19070c0293bc27cL60-L65

[Perdjesk]

In your use case, do I understand correctly that you built around getting the SQL queries out of baremaps server by requesting the tiles.json and were using the SQL queries ad-hoc outside of baremaps and directly against the database?

Going forward I see two possible paths for baremaps to provide an API with the SQL queries per layer:

• Serve a tile.json which is a combination of per specification fields with addition of custom fields (e.g queries). The current JSON schemas of TileJSON are permissive for this as there is no usage of "additionalProperties": false. See https://github.com/mapbox/tilejson-spec/blob/master/. This would match the previous behavior with the difference being that every additions would require vetting before addition into the TileJSON POJO
• Serve a tile.json with only fields from specification and have another endpoint such as tile-extended.json which would be the base tile.json with the additions of custom fields. Alternatively the newly endpoint could serve data without following TileJSON and only contain data tailored to various needs.

[polastre]

Yes @Perdjesk, your understanding is exactly right. I built my own tile exporter that performs much faster than the baremaps exporter (we plan to open source it in the near future). It relies on getting the SQL queries for the tileset out of the baremaps tiles.json.

Either option would be completely fine for our use case.

[bchapuis]

@polastre I'm looking forward to learn more about this. Our exporter has been left a bit aside to focus on the basemap and it would be great if we can integrate your optimizations at some point.

[bchapuis]

@polastre The security concern was a concern reported by @Perdjesk and another user. #722 exposes the SQL queries in dev mode and keep them hidden in production.

[polastre]

Thanks, #772 will help out a lot. We really need this, and I think the dev-mode-only is a reasonable compromise.

[polastre]

Note that our exporter is not built in Java, so I'm not sure if it will be compatible with this Apache project. I'll post a link to it once we open source it.

[bchapuis]

That would be awesome, as long as the license is compatible with Apache (MIT, BSD, ASL, etc.), we can probably port and integrate the optimization to Java.

[polastre]

@bchapuis here is the baremaps-exporter that we wrote to speed things up:
https://github.com/flightaware/baremaps-exporter

[bchapuis]

This is awesome!

The sql unioning looks really straight forward. The original PostgresTileStore was devised to make use of CTEs. I guess that with recent versions of postgis, this kind of optimization is not necessary anymore. I will try to update our code accordingly and see if we can implement a similar batching strategy.

Thanks a lot for sharing.

[bchapuis]

@polastre I started aligning the queries generated by the PostgresTileStore to the queries generated by the baremaps-exporter in #792.