Develop Extract for JSON format

[jbrown-xentity]

User Story

In order to harvest DCAT-US catalog JSON files, data.gov admins want:

• an extract utility to load and validate a root JSON file
• a function to then parse that file into individual dataset records identified by a UUID

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN a JSON URL is supplied to the python code (such as https://data.doi.gov/data.json)
  WHEN the JSON is downloaded
  THEN it will pass a validation utility to confirm the file can be opened and extracted

• GIVEN the JSON is valid
  THEN a new UUID "identifier" will be generated for the record

• GIVEN we have a valid JSON record with an identifier
  THEN we save that JSON record to S3 with the filename identifier with prefix as <feature>/<sourceId>/<jobId>/<recordId>

Background

Old CKAN process.

This is the first functional step of the harvesting pipeline. See the Harvest Job Lifecycle for pre- and post- steps. Also see the Controller Outline for the inputs and output data types and data structures. There will be future work related to extracting different types of file formats.

It is key not to confuse the file formats with the metadata schemas (i.e. JSON vs. DCAT-US).

We would like to be as abstract in this extract function definition so it can be easily extensible and scalable with future capabilities.

FYI: In order to create a repeatable test, you can create a local test server like we did for catalog. See test files here and mock server here and here.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Confirm file at URL is valid JSON format
• Parse root JSON into individual records
  • This should be done outside the root JSON validation
• Generate a new UUID for each potential record to store it in S3 and track it within the pipeline
• Save the new record to S3 with the filename identifier with prefix as <feature>/<sourceId>/<jobId>/<recordId>

[jbrown-xentity]

Replace current github action logic with poetry action setup

[rshewitt]

Waiting for discussion on error handling, functional vs OOP, env var formats, and docstrings.

[rshewitt]

refactored approach

• download file ( xml, json, waf-html ) THIS WILL ALWAYS BE THE CATALOG
• schema_specific check ( e.g. is_dcatus ( json ) or iso ( xml ) )
• parse catalog ( this is schema specific )
• upload to s3

additions

• add tests which address exceptions
  • 404, 403, .json() doesn't work