Add Leader-Follower paradigm to Solr-on-ECS

[nickumia-reisys]

User Story

In order to serve more traffic load, the Data.gov SSB Team wants to implement a leader-follower paradigm for standalone Solr configurations.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN [a contextual precondition]
  [AND optionally another precondition]
  WHEN [a triggering event] happens
  THEN [a verifiable outcome]
  [AND optionally another verifiable outcome]

Background

Parent Issue:

• [Tiny EPIC] Support SOLR standalone on ECS #3826

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

In order to implement this design, we would...

• Leave all existing code as is
• Create an n-count of ECS task definitions/services to start up each replica (very similar to current ecs.tf)
• Create a secondary LB with target groups pointing to each ECS solr replica service (small subset of lb.tf)
• Create a subdomain on the parent solr-[id number].ssb.data.gov domain and point it to the secondary LB just created
• Duplicate admin.tf to initialize the admin user/password for each of the replicas by way of the same n-count parameter
  • Alternatively, modify the admin.tf service to optionally update the admin password for each of the replicas as well. (Be careful not to break existing functionality)
• Output the secondary replica domain/user/pass for the user.
  • Presumably, all replicas will share the same admin user/pass combination
• Update the bind methodology to add the new binding to every solr
  • In this case, an end user can connect with either the admin solr or the replica solr with the same password. If this is undesirable, complicated logical would need to be written to choose between binding to the admin solr or the replica solr -- fair warning, this may not be possible in CSB mentality
• Make all of the above conditional on the replica parameter being > 0
• Create a different startup script for a solr follower solr.xml or solrconfig.xml
• Solr startup process that boots up from AWS backup vault (may not be possible?) or resyncs from SOLR leader

[hkdctol]

we will discuss but logs show we can proceed with Leader for now.

[nickumia-reisys]

The easiest-to-implement, but more resource-intensive design is to replicate exactly what we have on FCS which is two version of the catalog application: (1) Public for end-users, (2) Admin console for harvesting

In order to implement this design, we would...

• Leave all existing code as is
• Create an n-count of EFS volumes for each solr replica desired (very similar to current efs.tf)
• Create an n-count of ECS task definitions/services to start up each replica (very similar to current ecs.tf)
• Reference the specific n-count of the EFS volumes by count_id for each of the n-count task definitions
• Create a secondary LB with target groups pointing to each ECS solr replica service (small subset of lb.tf)
• Create a subdomain on the parent solr-[id number].ssb.data.gov domain and point it to the secondary LB just created
• Duplicate admin.tf to initialize the admin user/password for each of the replicas by way of the same n-count parameter
  • Alternatively, modify the admin.tf service to optionally update the admin password for each of the replicas as well. (Be careful not to break existing functionality)
• Output the secondary replica domain/user/pass for the user.
  • Presumably, all replicas will share the same admin user/pass combination
• Update the bind methodology to add the new binding to every solr
  • In this case, an end user can connect with either the admin solr or the replica solr with the same password. If this is undesirable, complicated logical would need to be written to choose between binding to the admin solr or the replica solr -- fair warning, this may not be possible in CSB mentality
• Make all of the above conditional on the replica parameter being > 0
• Create a different startup script for a solr follower solr.xml or solrconfig.xml

PRs are needed for both the datagov-brokerpak-solr repo and the catalog.data.gov repo.

I am against converting the existing solr-on-ecs plan to be the leader-follower plan. However, if we wanted to update existing Solrs with the new functionality this is the only way. I believe this leader/follower paradigm should be a separate CSB plan. In which case, we would provision a separate service to replace the existing one. If we never believe we'll have a use case for the two plans, updating the existing one is acceptable.

To ensure the above paragraph does not cause confusion, I'm not suggesting a new repository; "a separate CSB plan" is merely a new service directory in the existing datagov-brokerpak-solr repo. It is a point of discussion though.

[nickumia-reisys]

The above outline is just the setup for Solr. Effort would need to be put into modifying the manifest.yml to duplicate all of the configuration and code to support a secondary catalog.

• The manifest.yml
• The .profile
• The vars.[everything.yml
• The startup scripts
• The no need for a proxy (but can still use the same proxy if necessary)
• The create-cloudgov-services.sh to create a secondary DB/Redis (not solr)
• ...et cetera

[nickumia-reisys]

Future work (maybe): https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_definition_parameters.html#sd-networkconfiguration

If your service's tasks take a while to start and respond to health checks, you can specify a health check grace period of up to 2,147,483,647 seconds during which the ECS service scheduler ignores the health check status. This grace period can prevent the ECS service scheduler from marking tasks as unhealthy and stopping them before they have time to come up.

[FuhuXia]

Changes to solrconfig.xml to enable replication.
https://solr.apache.org/guide/8_7/index-replication.html

[nickumia-reisys]

This should be sufficiently complete now. To help categorize the above work for future reference, (the sub-bullets are mostly minor, sometimes breaking, fixes to the parent bullet)

Infrastructure Changes:

• Optionally Disable EFS GSA-TTS/datagov-brokerpak-solr#45
  • Put ephemeral_storage in the right definition GSA-TTS/datagov-brokerpak-solr#46
• Solr on ECS - Leader + Followers GSA-TTS/datagov-brokerpak-solr#47
  • lb name need to be 32 or less GSA-TTS/datagov-brokerpak-solr#49
  • Bad load balancer name GSA-TTS/datagov-brokerpak-solr#51
  • solr_follower_individual_urls needs a value GSA-TTS/datagov-brokerpak-solr#52

Solr/CKAN Configuration Changes:

• Solr replication catalog.data.gov#496
• Solr on ECS - Solr Replication Setup GSA-TTS/datagov-brokerpak-solr#48
• Use different solr url for web/harvesting catalog.data.gov#497

PR that created appropriate tests for the above changes:

• Update test architecture GSA-TTS/datagov-brokerpak-solr#50

Each PR has specific information about what needed to be changed or added to get us further. The highest level design consists of:

• One Solr Leader + (EFS/Ephemeral) Storage
• Zero or more Solr Followers + (EFS/Ephemeral) Storage
• One access point to communicate with leader
• One access point that distributes traffic among followers
• One access point per follower to communicate with a specific follower
• Custom configuration to setup Solr Leader
• Custom configuration to setup Solr Follower
• Separate EFS Volume configuration options for Leader/Follower

[hkdctol]

@nickumia-reisys is the diagram above available for re-use? @mogul wanted to use for SSB documentation

[nickumia-reisys]

Unfortunately, it just the screenshot. I made it with a random online flowchart tool and didn't have a way of saving it without making an account. If it can be used as is, I see no problem adding it to the SSB documentation.

[mogul]

No worries, I think I can recreate it pretty easily.