Look into tiling ArcGis feature service requests

[rowanwins]

Currently we just attempt to grab all features from an Esri Feature Service, which

1. Is very slow because we don't limit requests to the extent of the map
   • eg if you're looking in Syd you'll get data for Canberra
2. With older versions of ArcGIS server I think you get a subset of features
3. Sometimes fails with newer versions of ArcGIS server if you exceed the feature limit
   • need to specify returnExceededLimitFeatures: true as a query param then you at least get something
     https://developers.arcgis.com/rest/services-reference/query-feature-service-layer-.htm

We could move to a tiling approach which would enable us to be smarter about which features we request.

Basically you send query params like

returnGeometry: true
spatialRel: esriSpatialRelIntersects
maxAllowableOffset: 19567.879241000017 <---- allows simplifying the geom
geometry {
    "xmin":10018754.171389025,
    "ymin":-5009377.08569302,
    "xmax":15028131.257085029,
    "ymax":0.000002983957529067993,
    "spatialReference":{"wkid":102100,"latestWkid":3857}
}
geometryType: esriGeometryEnvelope
inSR: 102100
outFields: *
outSR: 102100
resultType: tile
}

And with newer versions of ArcGIS Server you can also use quantizationParameters which give you pixel coords rather than geom coords which are typically smaller in data size

quantizationParameters: {
   "mode":"view",
   "originPosition":"upperLeft",
   "tolerance":19567.879241000017,
   "extent":{
      "xmin":10018754.171389025,
      "ymin":-10018754.171389025, "xmax":20037508.342781033,
      "ymax":0.000002983957529067993,
       "spatialReference":{"wkid":102100,"latestWkid":3857}
    }
}

[rowanwins]

See leaflet-virtual-grid for how Esri-Leaflet is handling this under the hood for feature layers. I'm not to sure if Cesium has equivalent APIs or if it's too hard because of the 3D-ness making it hard to define a bounding box...

[rowanwins]

@kring this is one that I'd appreciate some thought on.

This stack overflow post has some similar elements to what I'm thinking. Once you've got the tiles being viewed I'm wondering if you could do some distance compute from the camera to the _tilesToRender to work out some discard rule to throw out those tiles that are way off in the distance and then retrieve data according to the tile bbox...

I see relevance for Esri Feature Services & WFS.

Any thoughts on whether this is a pipedream or would something be achievable with effort?

[kring]

@rowanwins this is doable but is a decent amount of effort. I didn't realize FeatureServer had a simplification parameter - that's very cool. With that, it should be possible to hook it up to Cesium's level-of-detail systems. Using the set of rendered terrain tiles and then adding a heuristic to discard far away ones (or maybe even not) would mostly work. Cesium's terrain engine also uses QuadtreePrimitive, which in theory can be used to do this sort of LOD selection on non-terrain stuff. And the most modern approach would probably be to hook it into the Cesium 3D Tiles rendering engine. Currently the 3D Tiles rendering is kinda locked into tileset.json and other transport format stuff, but this is something I'd love to change (and I know the Cesium team would too).

[techmavengeospatial]

Why not just use geojson-vt Newest feature service 10.7 supports MVT AS Output

…

On Mon, Sep 9, 2019, 8:58 PM Kevin Ring ***@***.***> wrote: @rowanwins <https://github.com/rowanwins> this is doable but is a decent amount of effort. I didn't realize FeatureServer had a simplification parameter - that's very cool. With that, it should be possible to hook it up to Cesium's level-of-detail systems. Using the set of rendered terrain tiles and then adding a heuristic to discard far away ones (or maybe even not) would mostly work. Cesium's terrain engine also uses QuadtreePrimitive, which in theory can be used to do this sort of LOD selection on non-terrain stuff. And the most modern approach would probably be to hook it into the Cesium 3D Tiles rendering engine. Currently the 3D Tiles rendering is kinda locked into tileset.json and other transport format stuff, but this is something I'd love to change (and I know the Cesium team would too). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3661?email_source=notifications&email_token=AFH3BPMIH6X75BCWEVTBZYLQI3WLFA5CNFSM4IO2TN22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6JO5HQ#issuecomment-529723038>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFH3BPM6GV35G3HECX7RCWDQI3WLFANCNFSM4IO2TN2Q> .

[kring]

Yeah geojson-vt would work when it's reasonable to get all the vector data once (it isn't always). We even have a prototype of that around somewhere. There's still the rendering side to consider, though. Our region mapping system and mvt catalog item type work by rasterizing the polygons in vector tiles to images and then draping the images on terrain. This works well, but has limitations. The biggest one is that it won't work with data that is not meant to be on the terrain (e.g. underground pipes, road networks that include bridges, etc.).

[rowanwins]

Thanks for the feedback @kring - yeah I suspected it wouldn't be trivial but it think it would payoff in terms of relieving some pain points in consuming with larger datasets via feature services.

I have thought it would be interesting to see what the Esri JS API does with feature services in 3d mode... maybe I'll take a quick poke around

[AnaBelgun]

@rowanwins once you have a look, please move this to current sprint if action will be taken

[AnaBelgun]

since we decided to move ahead with this, it's now included in next sprint

[nf-s]

Example of tiling requests

• https://www.arcgis.com/apps/mapviewer/index.html?url=https://services-ap1.arcgis.com/ypkPEy1AmwPKGNNv/ArcGIS/rest/services/ABS_Socio_Economic_Indexes_for_Areas_SEIFA_by_2021_SA2/FeatureServer&source=sd

[nf-s]

One way to tackle this is to create a new ArcGisPbfSource in ProtomapsImageryProvider.ts (probably somewhat similar to GeojsonSource that makes requests and converts features to ProtomapsFeatures.

Can possibly make use of these

• https://github.com/rowanwins/arcgis-pbf-parser
• https://github.com/rowanwins/mapbox-gl-arcgis-featureserver

[nf-s]

There is also this https://esri.github.io/arcgis-rest-js/api/feature-layer/queryFeatures/

[AnaBelgun]

See #6765 (comment)