Add flexibility to ArcGIS Feature Services connector

[fvankrieken]

Part of #1004, and touches a bit on #923

Thought updating ArcGIS Feature Service datasets for GFT was going to be simple. Had a bit of a wrench thrown in when multiple attempts failed

I broke out the commits so that the first commit has a lot of the test data (big part of the added lines in this PR), so commit-by-commit will be easier.

Context

As a refresher for terms

• Feature Service is what this general platform is called
• a Feature Server is essentially a dataset collection (a data product if you will), which generally supports a map view
• a Feature Server Layer is a classic ESRI layer - one dataset

So our library datasets refer to a single layer of a given Feature Server - in some cases, we have a couple datasets per feature server. We identified these in our templates with a layer_id (or assumed it to be 0.) Turns out layer ids aren't necessarily constant for a given feature server. I think ideally they are supposed to be, but for a handful of layers, the maintainers seemed to delete the previous layer and simply create a new one (with a new id).

Given that, I thought it made sense to

1. not specify id when possible, and grab the only layer there (and throw an error if there's an unexpected layer on the server). Seems to be cleanest for these Feature Servers that are tightly coupled to a single dataset
2. specify name otherwise, and attempt to look up layer by name.

There's a strong argument that generally, it's safer to specify id rather than name... but I don't know if that's something we can rely on here. I would hope so, but it's hard to tell.

Summary of changes

Separated out logic of models a little bit. Before, we just had a not quite accurately named FeatureServer object, shared across library and the connector. I separated things out so we have

• FeatureServer (not just a layer)
• FeatureServerLayer (has name and id, and could contain more metadata if we want)
• FeatureServerLayerDefinition (library-specific - optional name and id)

Separating these out made the different utilities a lot cleaner - not having to worry about if the layer object had its id and name resolved yet.

Beyond that, added and expanded FeatureService utils and added tests, which hadn't existed before

Test run

Passing run of calling library for all datasets

[codecov]

Codecov Report

Attention: Patch coverage is 98.86364% with 1 line in your changes missing coverage. Please review.

Project coverage is 56.55%. Comparing base (0f8b8dc) to head (09b5e52).

Files	Patch %	Lines
dcpy/models/connectors/esri.py	94.73%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1054      +/-   ##
==========================================
+ Coverage   55.48%   56.55%   +1.06%     
==========================================
  Files          97       97              
  Lines        4837     4905      +68     
  Branches      760      775      +15     
==========================================
+ Hits         2684     2774      +90     
+ Misses       2018     1998      -20     
+ Partials      135      133       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[fvankrieken]

Just rebased to fix clunky language in a docstring, no code changes since the linked action running all of these

[fvankrieken]

a tad unrelated to this PR but included because I was touching library tests with arcigs fs stuff - this was making live calls to socrata before, now it's using a mocked api response

[fvankrieken]

These I looked at the feature server linked in the info section to make sure the changes being made were accurate

[fvankrieken]

Build is failing due to duplicate rows in a source dataset for templatedb (failing a dbt check). This is still ready to go!

[fvankrieken]

I refactored this just slightly - seemed to be a bit cleaner with a match ... case. See latest commit

[alexrichey]

LGTM!

[fvankrieken]

Annoyed at this

python/mypy#12364

(see final comments especially)

[damonmcc]

@fvankrieken got some builds failing and seems related to this

GFT and FacDB have the same error while planning the build:

ValidationError: 1 validation error for Config
dataset.source.arcgis_feature_server.layer
  Extra inputs are not permitted [type=extra_forbidden, input_value=0, 
input_type=int]

for GFT, it's due to dcp_cscl_commonplace. for FacDB, it's due to nysdec_lands

[fvankrieken]

@fvankrieken got some builds failing and seems related to this

GFT and FacDB have the same error while planning the build:

ValidationError: 1 validation error for Config
dataset.source.arcgis_feature_server.layer
  Extra inputs are not permitted [type=extra_forbidden, input_value=0, 
input_type=int]

for GFT, it's due to dcp_cscl_commonplace. for FacDB, it's due to nysdec_lands

Oh shoot. Blanked about mutating existing structures in those config objects. I'll get a fix

[damonmcc]

(note from in-person chat) this action run to archive all ArcGIS Feature Server datasets may fix it