Bugfix for autodeploy job

[ChaamC]

The new code added with this merge created a new bug for the autodeploy job.

From the autodeploy job's log :

triggerdeploy START_TIME=2021-05-13T14:00:03+0000
Error: DEPLOY_DATA_JOB_SCHEDULE not set

If the AUTODEPLOY_NOTEBOOK_FREQUENCY variable is not set in the env.local file, it would create the error above.
The variable is set in the default.env file, in case it is not defined in the env.local, and is then used for the new env file from pavics-jupyter-base here.
The error happens because the default.env was not called in the triggerdeploy.sh script, and the variable was not set when running the env.local.

Solution was tested in a test environment and the cronjob seems to be fixed now.

I tried to see if the same situation could be found anywhere else. From what I see, default.env seems to be called in a consistent way before the env.local.
There is just here, where the default.env doesn't seem to be called. The env.local is called multiple times in that file, but only the first time is the default.env not called.
@tlvu Do you think it could be problematic too? I am not very familiar with that script, how can I verify this?

[tlvu]

LGTM, just use the already available var COMPOSE_DIR please.

For testing please test that the autodeploy can fully trigger (ie not just check but actually perform a deployment).

[tlvu]

$COMPOSE_DIR/default.env

[ChaamC]

For testing, i used the daccs-iac, with the birdhouse stack, and I checked the log of the autodeploy cronjob and the job was running without errors. I planned to check also after the merge on a staging setup.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/546/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : fix/autodeploy-job
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-4.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/466/

NOTEBOOK TEST RESULTS

    
[2021-06-02T14:49:42.680Z] ============================= test session starts ==============================
[2021-06-02T14:49:42.680Z] platform linux -- Python 3.7.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
[2021-06-02T14:49:42.680Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-06-02T14:49:42.680Z] plugins: anyio-2.2.0, dash-1.20.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-06-02T14:49:42.680Z] collected 143 items
[2021-06-02T14:49:42.680Z] 
[2021-06-02T14:49:42.680Z] notebooks/hummingbird.ipynb ............                                 [  8%]
[2021-06-02T14:49:50.560Z] notebooks-auth/test_thredds.ipynb ..........                             [ 15%]
[2021-06-02T14:50:13.048Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 20%]
[2021-06-02T14:50:28.387Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 24%]
[2021-06-02T14:50:40.289Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 30%]
[2021-06-02T14:50:40.858Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 37%]
[2021-06-02T14:50:55.763Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 37%]
[2021-06-02T14:50:55.763Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 38%]
[2021-06-02T14:51:02.003Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2021-06-02T14:51:08.074Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 46%]
[2021-06-02T14:55:28.631Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 57%]
[2021-06-02T14:56:21.738Z] ...............                                                          [ 67%]
[2021-06-02T14:56:25.426Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 70%]
[2021-06-02T14:56:32.295Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 74%]
[2021-06-02T14:56:45.341Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 81%]
[2021-06-02T14:56:53.100Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 86%]
[2021-06-02T14:57:37.209Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [100%]
[2021-06-02T14:57:37.209Z] 
[2021-06-02T14:57:37.209Z] ======================= 143 passed in 478.93s (0:07:58) ========================
    
  

[tlvu]

There is just here, where the default.env doesn't seem to be called. The env.local is called multiple times in that file, but only the first time is the default.env not called.

Yeah, probably defensively also add a . $COMPOSE_DIR/default.env there also. Should technically not cause any problems since default.env is much simpler than env.local (no sourcing of other files).

By the way, great PR description, please copy/paste into merge commit description. In fact always do this copy/paste for all your PR, thanks.

[crim-jenkins-bot]

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/547/
Result : success

BIRDHOUSE_DEPLOY_BRANCH : fix/autodeploy-job
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-4.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/467/

NOTEBOOK TEST RESULTS

    
[2021-06-02T15:16:14.336Z] ============================= test session starts ==============================
[2021-06-02T15:16:14.336Z] platform linux -- Python 3.7.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
[2021-06-02T15:16:14.337Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2021-06-02T15:16:14.337Z] plugins: anyio-2.2.0, dash-1.20.0, nbval-0.9.6, tornasync-0.6.0.post2
[2021-06-02T15:16:14.337Z] collected 143 items
[2021-06-02T15:16:14.337Z] 
[2021-06-02T15:16:16.367Z] notebooks/hummingbird.ipynb ............                                 [  8%]
[2021-06-02T15:16:24.737Z] notebooks-auth/test_thredds.ipynb ..........                             [ 15%]
[2021-06-02T15:16:31.643Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 20%]
[2021-06-02T15:16:40.012Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 24%]
[2021-06-02T15:16:48.577Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ........       [ 30%]
[2021-06-02T15:16:50.609Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 37%]
[2021-06-02T15:17:05.522Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .                 [ 37%]
[2021-06-02T15:17:05.522Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 38%]
[2021-06-02T15:17:12.769Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 43%]
[2021-06-02T15:17:18.801Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 46%]
[2021-06-02T15:20:13.843Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 57%]
[2021-06-02T15:21:02.564Z] ...............                                                          [ 67%]
[2021-06-02T15:21:06.699Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 70%]
[2021-06-02T15:21:12.113Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 74%]
[2021-06-02T15:21:24.818Z] finch-master/docs/source/notebooks/dap_subset.ipynb ..........           [ 81%]
[2021-06-02T15:21:32.621Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 86%]
[2021-06-02T15:22:17.052Z] finch-master/docs/source/notebooks/subset.ipynb ....................     [100%]
[2021-06-02T15:22:17.052Z] 
[2021-06-02T15:22:17.052Z] ======================= 143 passed in 362.91s (0:06:02) ========================