Stand up VA CMS on Lagoon Saas Demo environment

[timcosgrove]

Description

In order to evaluate Lagoon as a viable hosting platform for VA.gov CMS, we need to set up the CMS on an Amazee.io (Lagoon Saas) demo environment so that we can evaluate Lagoon features and potential shortcomings.

Acceptance Criteria

• VA.gov CMS is functional on Amazee.io Saas
  • User can Login
  • User can create content
  • Content build runs successfully with Demo CMS instance as a content API endpoint

CMS Team

Please check the team(s) that will do this work.

• Program
• Platform CMS Team
• Sitewide Crew
• ⭐️ Sitewide CMS
• ⭐️ Public Websites
• ⭐️ Facilities
• ⭐️ User support

[EWashb]

@olivereri @ElijahLynn do you have an update on this one?

[olivereri]

@olivereri @ElijahLynn do you have an update on this one?

Still in the process of setting this up on Lagoon SaaS. Working with Lagoon support. Possible last technical hurdle to having a working CMS copy on Lagoon is importing the database.

[olivereri]

One thing I may have mentioned out loud but not in writing is that this issue isn't part of the Pour Lagoon Foundation epic and is blocking this issue #12009

[EWashb]

@olivereri thanks for the update. I'm going to pull this into Sprint 77. Looks like this needs to be done first. Also, added it as a blocker for #12009.

[ElijahLynn]

I'm still working on this and have made decent progress refactoring our .env files so that composer install works correctly. This is arguably work that could have been post-poned, after the PoC, but I felt compelled to get it done correctly so we weren't left with tech debt if we go with Lagoon.

I think we are close to getting the ACs met within the next 1-2 days. This will have to come over to next sprint.

[ElijahLynn]

Working branch is https://github.com/department-of-veterans-affairs/va.gov-cms/tree/VACMS-12291-lagoon-poc

[ElijahLynn]

Eric just got AC #1 / User Login across the finish line:

[ElijahLynn]

Not all of the styles are not loading because the styles.css is 404ing because it wasn't generated. And that is because composer va:theme:compile isn't able to run (manually for now). This can be another issue but details are below:

nginx.main.demo-vago…st.us2.amazee.io/:1 Refused to apply style from 'https://nginx.main.demo-vagov-cmstest.us2.amazee.io/themes/custom/vagovclaro/dist/styles.css?rph09d' because its MIME type ('text/html') is not a supported stylesheet MIME type, and strict MIME checking is enabled.

yarn is installed on the images so we can maybe update this to be /usr/bin/yarn

va.gov-cms/composer.json

Line 587 in 4a3104e

"cd bin && ln -sf ../docroot/libraries/yarn/bin/yarn yarn",

.

[ElijahLynn]

AC 2: Create content - is failing with this on https://nginx.main.demo-vagov-cmstest.us2.amazee.io/node/add/page.

The website encountered an unexpected error. Please try again later.

Twig\Error\LoaderError: Template "@cmsds/alert.twig" is not defined in "themes/custom/vagovclaro/templates/misc/status-messages.html.twig" at line 29. in Twig\Loader\ChainLoader->getCacheKey() (line 98 of vendor/twig/twig/src/Loader/ChainLoader.php).

[ElijahLynn]

RE: AC3 (content build): The GQL query hung after like 15m or so. I'm also building this from my local, fwiw. I tested it by adding --drupal-address=https://nginx.main.demo-vagov-cmstest.us2.amazee.io to the end of

va.gov-cms/composer.json

Line 699 in 4582306

"cd web; INSTALL_HOOKS=no NODE_ENV=production yarn build --pull-drupal --no-drupal-proxy --buildtype=vagovdev --api=https://dev-api.va.gov",

then running ddev composer va:web:build.

[olivereri]

Testing out AC3 too. I synced CMS and CMS-Test, then updated the database incase this might have been the problem. I did a web build as well, and it got as far as you did @ElijahLynn however I let it go and error:

| GetNodePage__slice3 | 8s | 25 pages |
Error: HTTP error: 500: Internal Server Error
    at Object.query (/var/www/html/docroot/vendor/va-gov/content-build/src/site/stages/build/drupal/api.js:118:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async RetryOperation._fn (/var/www/html/docroot/vendor/va-gov/content-build/node_modules/p-retry/index.js:50:12)
Failed to pipe Drupal content into Metalsmith!
/var/www/html/docroot/vendor/va-gov/content-build/src/site/stages/build/plugins/download-assets.js:11
  throw up;
  ^

Error: HTTP error: 500: Internal Server Error
    at Object.query (/var/www/html/docroot/vendor/va-gov/content-build/src/site/stages/build/drupal/api.js:118:13)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async RetryOperation._fn (/var/www/html/docroot/vendor/va-gov/content-build/node_modules/p-retry/index.js:50:12) {
  attemptNumber: 11,
  retriesLeft: 0
}
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Script cd web; INSTALL_HOOKS=no NODE_ENV=production yarn build --pull-drupal --no-drupal-proxy --buildtype=vagovdev --api=https://dev-api.va.gov --drupal-address=https://nginx.main.demo-vagov-cmstest.us2.amazee.io handling the va:web:build event returned with error code 1
composer [va:web:build] failed, composer command failed: exit status 1. stderr=

[ElijahLynn]

I'm having a weird nostalgia on this one:

Failed to pipe Drupal content into Metalsmith!
500 Internal Server Error

Something seems familiar but not sure what yet 🤷

I think we might want to look at ALB logs for the next step.

[olivereri]

AC 3 Complete.

Raw troubleshooting in slack here:
https://dsva.slack.com/archives/CT4GZBM8F/p1675726049369389

The cause was a PHP out of memory error uncovered by using Kibana provided by Lagoon. Inspecting the Nginx container log revealed:

NOTICE: PHP message: PHP Fatal error:  Allowed memory size of 419430400 bytes exhausted (tried to allocate 163840 bytes) in /app/docroot/core/lib/Drupal/Core/Cache/Cache.php on line 53

Temporary solution was to add ini_set('memory_limit', '4G'); to settings.lagoon.php and redeploy. After that PHP reported the new limit:

and finally made it passed where it last errored:

| GetNodeLeadershipListingPages | 191s | 138 pages |
Finished 758 queries in 208s with 21028 pages
https://nginx.main.demo-vagov-cmstest.us2.amazee.io/ response time: : 3:29.239 (m:ss.mmm)
Total time to load content from GraphQL: 3:33.352 (m:ss.mmm)
Drupal successfully loaded!

[ElijahLynn]

Good sleuthing @olivereri and @ndouglas!

[ElijahLynn]

AC2 works with https://nginx.main.demo-vagov-cmstest.us2.amazee.io/node/add/checklist

[olivereri]

Working with AC2 revealed some issues that we'll need to tackle. In certain instances the site will throw the below when it should be showing a pretty error:

The website encountered an unexpected error. Please try again later.

Twig\Error\LoaderError: Template "@cmsds/alert.twig" is not defined in "themes/custom/vagovclaro/templates/misc/status-messages.html.twig" at line 29. in Twig\Loader\ChainLoader->getCacheKey() (line 98 of vendor/twig/twig/src/Loader/ChainLoader.php).

[EWashb]

Working with AC2 revealed some issues that we'll need to tackle. In certain instances the site will throw the below when it should be showing a pretty error:

The website encountered an unexpected error. Please try again later.

Twig\Error\LoaderError: Template "@cmsds/alert.twig" is not defined in "themes/custom/vagovclaro/templates/misc/status-messages.html.twig" at line 29. in Twig\Loader\ChainLoader->getCacheKey() (line 98 of vendor/twig/twig/src/Loader/ChainLoader.php).

Is this something we need a new ticket for, or does it completely block the demo environment?

[olivereri]

Working with AC2 revealed some issues that we'll need to tackle. In certain instances the site will throw the below when it should be showing a pretty error:

The website encountered an unexpected error. Please try again later.

Twig\Error\LoaderError: Template "@cmsds/alert.twig" is not defined in "themes/custom/vagovclaro/templates/misc/status-messages.html.twig" at line 29. in Twig\Loader\ChainLoader->getCacheKey() (line 98 of vendor/twig/twig/src/Loader/ChainLoader.php).

Is this something we need a new ticket for, or does it completely block the demo environment?

We did satisfy AC2 because we could create content on the CMS so not blocked. However, we'll need a new ticket to track this. There's a good chance we'll find more things like this too but we can keep moving forward.

[EWashb]

@olivereri That sounds fair. I assume we will find these things as we move along too. We can create new tickets when those come up. Thanks for the update!

[ElijahLynn]

This is done. Closing.