Add docs for deployment and workflow response

docs/source/prefect.rst

@@ -14,15 +14,6 @@ In HPC, you can use the following command to create a work pool.
 
 If the prefect server is running, but the workpool is not available, then you can create a workpool by going to the website where the prefect server is hosted. Go to `Work Pools` tab and create a workpool with the name `workpool`. This is the name of the workpool that is defined in [prefect.yaml > definitions > work_pools > name](https://github.com/niaid/image_portal_workflows/pull/353/files#diff-b49a6f022232810a70f1a0c2feffbbe84d018b2418a7996e52430c6063ada3a3R23) file.
 
-Deploying workflows
--------------------
-
-Once the prefect server is running and workpool is created. You can login to respective *BigSky* instance (dev, qa, prod) and deploy the workflows.
-
-.. code-block::
-
-   prefect deploy --all
-
 Continuous Deployment (dev to qa to prod)
 -----------------------------------------
 
@@ -41,3 +32,45 @@ Afterwards, we can deploy the aws infrastructure for `qa` as,
    SPACES_SOLUTION_ENV=qa spaces task -f hedwig.spaces-solution.yaml build-deploy
 
 This can then again be applied for the `qa` to `prod` changes.
+
+Managing Prefect Worker
+=======================
+
+Deploying workflows
+-------------------
+
+Once the prefect server is running and workpool is created. You can login to respective *BigSky* instance (dev, qa, prod) and deploy the workflows.
+
+Make sure the configurations are correct:
+
+1. Update prefect.yaml to change the user and/or directory names
+
+   .. code-block::
+
+      # by default pull.directory setting is set for prod environment
+      directory: /gs1/home/hedwig_prod/image_portal_workflows
+
+      # change it to dev or qa, based on your environment
+
+2. Check prefect config with view
+
+   .. code-block::
+
+      prefect config view
+
+      # Update config iff required
+      export PREFECT_API_KEY=xyz
+      export PREFECT_API_URL=abc.com
+
+3. Deploy flows with prefect deploy
+
+   .. code-block::
+
+      prefect deploy
+      # Or deploy all based on prefect.yaml using the following setting
+      # However, this will also deploy pytest_runner workflow in other envs (where it's not needed)
+      # prefect deploy --all
+
+4. Run worker (properly via the helper_scripts/.service file)
+
+   The service files should restarts the worker when killed. Normally, we would need to do this step

docs/source/usage.rst

@@ -8,7 +8,7 @@ Workflow servers are deployed in `development <https://prefect2.hedwig-workflow-
 
 In order to submit a workflow job, you will need deployment IDs of each of the workflows, as shown:
 
-.. list-table:: Deployment IDs (**Updated: 12/11/2023**)
+.. list-table:: Deployment IDs (**Updated: 01/26/2023**)
    :widths: 20 25 25 25
    :header-rows: 1
 
@@ -302,6 +302,18 @@ The pipeline parameters can be also be observed programatically using following
        | jq '.description,.id,.parameter_openapi_schema' > schema.json
 
 
+Pipeline Response
+-----------------
+
+All pipelines responses follow the same schema. Currently, all schema are documented in yaml files as such:
+
+.. literalinclude:: ../../api_schema/PipelineCallback.yaml
+   :language: yaml
+   :emphasize-lines: 1-5,13-21
+   :linenos:
+
+You can explore more on the yaml files in the `Github <https://github.com/niaid/image_portal_workflows/tree/main/api_schema>`_ repo.
+
 CLI/SDK Submission
 ------------------