Configure S3 to store Harvester Records

[btylerburton]

User Story

In order to perform operations on the data records reliably, datagov wants an interface to interact with S3 or the localstack equivalent.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

sourceId = UUID generated by controller on creation of new harvest source
jobId = UUID generated by controller when a new job is intiated
recordId = UUID generated by extract service to track status of record within pipeline

• GIVEN I would like to save the extracted record from a harvest source
  AND I have a prefix defined by the schema <feature>/<sourceId>/<jobId>/<recordId>
  THEN I want a utility to PUT that object using that prefix

• GIVEN I would like to retrieve a previously saved record from S3
  AND I have a prefix defined by the schema <feature>/<sourceId>/<jobId>/<recordId>
  THEN I want a utility to GET the object associated with that prefix

• GIVEN I would like to delete a previously saved record
  AND I have a prefix defined by the schema <feature>/<sourceId>/<jobId>/<recordId>
  THEN I want a utility to DELETE the object associated with that prefix

• GIVEN I would like to query the count of added/updated/deleted records with a <jobId>
  AND I have a prefix defined by the schema <feature>/<sourceId>/<jobId>
  THEN I want a utility to GET the objects associated with that <jobId> and return them.

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Data.gov would like all Boto / S3 references contained within a single module so that any upgrades to the service would happen simultaneously.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Create a module to interact with S3 client
• Create helper methods to abstract away all details except name (<feature>/<sourceId>/<jobId>/<recordId>) and value (the record to store)

[rshewitt]

the PUT object is currently included in the work near completion in the extract ticket #4257

[robert-bryson]

Moving back to new dev as work superseded by Airflow work.