refactor drush command and update docs

composer-manifest.yaml

@@ -56,7 +56,7 @@ packages:
     drupal/bulk_update_fields: 2.0.0-alpha6
     drupal/cache_control_override: 1.1.0
     drupal/cer: 5.0.0-beta3
-    drupal/coder: 8.3.21
+    drupal/coder: 8.3.22
     drupal/color_field: 2.5.0
     drupal/components: 2.4.0
     drupal/config_direct_save: 2.1.0
@@ -248,7 +248,7 @@ packages:
     phpoffice/phpword: 0.18.3
     phpseclib/phpseclib: 3.0.23
     phpspec/prophecy: v1.17.0
-    phpstan/phpdoc-parser: 1.24.1
+    phpstan/phpdoc-parser: 1.24.2
     phpstan/phpstan: 1.10.35
     phpstan/phpstan-deprecation-rules: 1.1.4
     phpunit/php-code-coverage: 9.2.29
@@ -322,7 +322,7 @@ packages:
     simplesamlphp/twig-configurable-i18n: v2.3.5
     simshaun/recurr: v5.0.1
     sirbrillig/phpcs-variable-analysis: v2.11.17
-    slevomat/coding-standard: 8.13.4
+    slevomat/coding-standard: 8.14.1
     solarium/solarium: 6.3.1
     squizlabs/php_codesniffer: 3.7.2
     stack/builder: v1.0.6

config/sfgov_api.settings.yml

@@ -0,0 +1,8 @@
+username: admin
+password: admin
+host_ip: host.docker.internal
+port: '8000'
+wag_parent_en: '2'
+wag_parent_es: '3'
+wag_parent_fil: '4'
+wag_parent_zh_hant: '5'

web/.gitignore

@@ -3,3 +3,4 @@
 /README.txt
 /example.gitignore
 /README.md
+/modules/custom/sfgov_api/src/Drush/Errors

web/modules/custom/sfgov_api/README.MD

@@ -1,34 +1,60 @@
 # SF.gov API
-This module builds and exposes json data for the wagtail site to consume.
+This module builds and exposes json data from the Drupal side and then provides means of pushing that data into Wagtail.
 
-## Calling the API
+# Setup
+## Local setup
+1. This setup assumes you're hosting your Drupal site through lando.
+2. Install the wagtail site locally from sfgov's github
+3. Force wagtail to accept external connections by adding the following to your local_settings.py in the Wagtail repo.
+```
+ALLOWED_HOSTS = ['localhost', 'host.docker.internal', '127.0.0.1']
+```
+
+### Tips
+- Make sure you've run a config import so that all of the settings at `/admin/config/system/wagtail-api-credentials` are properly set up.
+- You're likely to have to reset wagtail many times during testing. Copy and paste the following into your terminal to do so.
+```
+dropdb ds_platform &&
+createdb ds_platform &&
+./manage.py migrate &&
+./manage.py createsuperuser &&
+./manage.py setup_locales &&
+./manage.py loaddata home_translations &&
+./manage.py runserver
+```
+
+## Live Setup
+WIP
+
+# Viewing API Data
 There is currently one route that provides entity data. You must provide the entity type and bundle
-as arguments and it will return all of the entities of that type.
+as arguments and it will return some entities of that type displayed in the format that will be pushed to Wagtail.
+This is very helpful for debugging. Note: you can also provide an entity id as an argument.
 
 Examples:
- - all step_by_step nodes: `/sfgov-api/entity/node/en/step_by_step/`
- - all process_step paragraphs: `/sfgov-api/entity/paragraph/process_step`
+ - all step_by_step nodes in english: `sfgov-api/entity/node/en/step_by_step`
+ - all step_by_step nodes in spanish: `/sfgov-api/entity/es/node/step_by_step`
+ - all process_step paragraphs in english: `/sfgov-api/entity/en/paragraph/process_step`
 
-Optional arguments for language and specific entity ids
- - all step_by_step nodes in spanish: `/sfgov-api/entity/node/es/step_by_step`
- - step_by_step node:610 in english: `/sfgov-api/entity/node/en/step_by_step/en/610`
+Optional arguments for specific entity ids
+ - step_by_step node:610 in english: `/sfgov-api/entity/en/node/step_by_step/610`
 
 **For an entity type to work it has to have a corresponding plugin in sfgov_api/src/Plugin/SfgApi**
 
 ## API structure
 This module uses Drupal's plugin system to create a flexible and extendable variety of api endpoints.
 Most of the work happens in `sfgov_api/src/SfgApiPluginBase.php` which pulls information from annotations/url arguments/plugins.
 It then uses this data to build a list of entities (and translate them if necessary), then make representations
-of them that can be sent as a json object.
+of them that can be viewed as a json object.
 
 The json object consists of three parts for each entity, and the data is set in different places
   - **drupal_data**: This is basic data to use for organizational purposes (id, entity type, bundle). Set by `SfgApiPluginBase`
   - **base_data**: This is where we would add in any metadata that wagtail expects. Set by the entity base plugin like `sfgov_api/src/SfgApiNodePluginBase.php`
   - **custom_data**: This is where we manipulate the data for the individual field values of the entity. Set by the bundle plugin like `sfgov_api/src/Plugin/SfgApi/Node/StepByStep.php`
 
 ## Building on the API
-With this structure, every entity that needs to be exposed for the API will need its own plugin. At time of writing
-there are example plugins in place for Nodes and Paragraphs.
+With this structure, every entity that needs to be exposed for the API will need its own plugin. There are example
+plugins in place for Nodes and Paragraphs.
 
 You can easily generate new plugins with the command `drush generate sfgov:api-plugin`
 
@@ -48,41 +74,77 @@ The part that processes the drupal data can simply pull it from the entity:
 Or it can be more elaborate and wrap the data in some kind of function that processes the data further:
 `$this->getReferencedData($entity->get('field_process_steps')->referencedEntities())`
 
-Any helper functions that are created to process the data specifically for wagtail should be added to `sfgov_api/src/Plugin/SfgApi/ApiFieldHelperTrait.php`
+Any helper functions that are created to process the data specifically for wagtail
+should be added to `sfgov_api/src/Plugin/SfgApi/ApiFieldHelperTrait.php`
 so that they can be used in other plugins.
 
 Note: The aforementioned `getReferencedData` function relies on there being a plugin for the entity type it is referencing. It
 uses said plugin to map out the fields and data.
 
-## Pushing to Wagtail
-At time of writing there are two custom drush commands for pushing nodes into the Wagtail API. It only supports pushing to a
-local version of the wagtail site.
-
-**This approach might change in future iterations and is currently very rough and limited**
-
-### Requirements
-1. This setup assumes you're hosting your Drupal site through lando.
-2. Install the wagtail site locally from sfgov's github
-3. Force wagtail to accept external connections by adding the following to your local_settings.py in the Wagtail repo.
-```
-ALLOWED_HOSTS = ['localhost', 'host.docker.internal', '127.0.0.1']
-```
-
-### Usage
-There are currently two functions, each one for sending individual entities/pages to the Wagtail API. The only difference between the two is how they gather their respective json payloads.
-**pushLiteralData** Lets you set the literal field values being pushed. This is mainly for experimentation purposes.
-**pushEntityById** Hooks into the larger plugin infrastructure built in this module to generate that json payload.
-
-## Todo
-### Features
-- add proper pagination to `SfgApiPluginBase::prepareData()` (currently limited to 20 results)
-- add proper security to the route(s)
-- add interfaces
-
-### Plugins
-- Figure out correct values for baseData in `SfgApiNodePluginBase`
-- Figure out correct values for baseData in `sfgov_api/src/Plugin/SfgApi/Media/Image.php`
-- Figure out correct field names for custom data in `sfgov_api/src/Plugin/SfgApi/Paragraph/ProcessStep.php`
-  and `sfgov_api/src/Plugin/SfgApi/Paragraph/ProcessStep.php`. currently using the Drupal field names for
-  proof of concept.
-- Add plugins for all required entity types
+# Pushing to Wagtail
+
+## Migration Strategy
+A successful push to the Wagtail API returns the wagtail id of the node created. Those ids then get stored in the
+`drupal_wagtail_id_map` table next to their Drupal ID. This allows Drupal to have some idea of what has been migrated and where
+The plan is to migrate all nodes in stub form first then update them to have their actual content. It has to be done this
+way to make the entity relationships work.
+
+### Entity Relationships
+Wagtail is designed around hierarchical relationships, while relationships in Drupal are flat.
+The current way around this are fields like related_content_agency which can store references to other
+nodes but also need to be updated in the API by pushing to `/api/cms/sf.RelatedContentAgency`
+- This is still WIP
+
+### Translations
+Wagtail treats each language of a node as a separate entity while Drupal does not. To keep track
+of these ids there is one entry per language in the `drupal_wagtail_id_map` table.
+
+Also, Wagtail differentiates translations by giving them a different parent page. Those parent pages
+are automatically created as part of the wagtail setup with the `./manage.py setup_locales` and
+`./manage.py loaddata home_translations` commands. Those parent pages have to be manually identified at
+`/admin/config/system/wagtail-api-credentials`. Nodes will handle this translation in `SfgApiNodeBase`.
+
+## Commands
+There are two custom drush commands for pushing nodes into the Wagtail API.
+
+- `sfgov_api:push_entity `(alias: pe): pushes a single entity into Wagtail.
+e.g. `pe node step_by_step en 610` (push step-by-step node 610)
+- `sfgov_api:push_entity_by_bundle` (alias: peb): pushes every entity of a provided bundle type.
+e.g. `drush peb node step_by_step es` (push every spanish step by step)
+
+**Note: At time of writing these are only designed to support pushing nodes and their attached paragraphs**
+
+Both of these commands have the following optional parameters
+- `--print`: Use this for debugging, it will print some useful error data to the console and add two files to
+`web/modules/custom/sfgov_api/src/Drush/Errors`. One file is an exact print of the curl command used for the push, the
+other is an html file of what went wrong in Wagtail.
+- `--stub`: Use to push just the stub data of an entity (title and slug)
+- `--update`: Update the data of an existing node in Wagtail
+
+Both of these basically route to the `pushToWagtail` function which does the heavy lifting.
+
+## Error handling
+There are three primary types of errors that can come from this process. This list might expand
+in the future.
+
+**No Translation:** This simply means that there was no translation of the node available. Not really an error, but the
+error logs and tables look confusing without explicitly calling this out.
+**Wagtail API:** These are json errors that are returned from the API directly (e.g. missing required fields)
+**Wagtail Errors:** Sometimes when you push it breaks Wagtail in a non-api way. Wagtail's response is to generate
+an html page with the full error message (e.g. pushing a slug that already exists)
+
+Every node that that is pushed registers data to the `drupal_wagtail_id_map` table. If it has an error then it will
+record the error id which can then be looked up in the `drupal_wagtail_errors` table.
+
+# Miscellaneous
+**Unused Code**
+There are some pieces of code that are currently not being used but might be useful in the future and/or have
+been useful in previous iterations including:
+  - Plugin's ability to set a wag_bundle
+  - Most of the data in the `drupal_data` array when rendering entities
+  - `getWagtailTime` and `generateRandomSlug` in the `ApiFieldHelperTrait`
+
+**Shape of Wagtail Data**
+If you need to see the shape of data in Wagtail, create that page, then go to its
+corresponding API page. Ie, if you make a StepByStep of ID 8, you can see the shape
+of the data by going to `http://127.0.0.1:8000/api/cms/sf.StepByStep/8`

web/modules/custom/sfgov_api/config/schema/sfgov_api.schema.yml

@@ -0,0 +1,29 @@
+# Schema for the configuration files of the sfgov_api module.
+sfgov_api.settings:
+  type: config_object
+  label: 'sfgov_api settings'
+  mapping:
+    username:
+      type: string
+      label: 'Username'
+    password:
+      type: string
+      label: 'Password'
+    host_ip:
+      type: string
+      label: 'Host IP'
+    port:
+      type: string
+      label: 'Port'
+    wag_parent_en:
+      type: string
+      label: 'WAG Parent EN'
+    wag_parent_es:
+      type: string
+      label: 'WAG Parent ES'
+    wag_parent_zh:
+      type: string
+      label: 'WAG Parent ZH'
+    wag_parent_fil:
+      type: string
+      label: 'WAG Parent FIL'