Harvester 2.0: [1/4d] Create a DCATUS dataset on CKAN

[rshewitt]

User Story

datagov wants the ability to create a DCATUS dataset using the CKAN API in the new harvesting repo

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN a valid DCATUS catalog
  AND a data.gov catalog-like CKAN system is running independently
  AND a CKAN dataset schema reflecting the information on a dataset dataset page
  WHEN the "create dataset"-equivalent function is called
  THEN the DCATUS catalog will be transformed into the CKAN-compatible schema
  THEN the transformed DCATUS catalog will be added to CKAN

Background

• the current harvesting solution is being replaced. in the short-term, the CKAN catalog is intended to persist so the "0.1" harvester replacement needs to offer CKAN dataset creation.
• we have a transformation function which can convert a dcatus catalog to a ckan format. The intention is to start fresh and make an improved version.
  • examples of unnecessary or overly custom code:
    • checking if a parent package id starts with a specific string. link
    • are we using schema version 1.0? link
      • it appears there are 15 packages which use 1.0. do we upgrade these packages to 1.1?
    • do we need save_object_error?
    • do we still need MAPPING and SKIP when we have MAPPING_V1_1 and SKIP_V1_1?

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• create a function which creates a DCATUS dataset via CKAN API
• verify dataset exists
• Document any bugs or strangeness with the dataset display on the UI, or any differences in the data via the API.

Resources

• create

[rshewitt]

package_create route is disabled. is it possible to allow this for dev only?

[rshewitt]

branch

[rshewitt]

an mvp ckan dataset only contains information that is present on the dataset page. here's a list of fields which appear to be on that dataset page. this list excludes any values calculated by CKAN ( e.g. modified date, schema information, harvest source id, source hash, etc... )

• title
• owner_org
• notes
• accessLevel
• license_id
• publisher
  • value
• maintainer
• resources
  • description
  • url
  • name
• tags
• type
• identifier
• modified
• bureauCode
• programCode
• publisher_hierarchy

[rshewitt]

ckan catalog dev test dcatus catalog

prod ckan dataset

comparing the pictures indicates a similarity between the two. the catalog dev dataset is created using a dcatus-to-ckan conversion function in this branch

[rshewitt]

include the remaining information that needs to be added to the transformed dataset ( dcatus-to-ckan )

[rshewitt]

ckan schemas