[research: 1d] Determine the impact of latency to Solr for an average CKAN request

[adborden]

User Story

In order to understand the acceptable latency between CKAN in GovCloud and brokered Solr, the data.gov team wants to understand the impact of CKAN->Solr latency on the user experience.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• We can make a judgment call about the urgency of brokering Solr out of US West (Oregon) instead of US-East.

Background

We know that having Solr in a difference region from the apps will ad latency to back-end Solr requests. We want to know the impact so that we can decide if we care enough to broker Solr out of US West (Oregon) right away, or we can hold off and do this later.

Security Considerations (required)

None, this is to validate that latencies with our brokered solr are reasonable.

Sketch

We're making a spreadsheet for this data

• Collect data about the existing Solr transaction costs for our top 95% of requests
• Calculate the additional time added to each request type based on measured latency between GovCloud and US-East
• Measure the actual latency and enter it in the sheet.
  • Looks like between US-East-1 (N. Virginia) and US-West-2 (Oregon, aka GovCloud-West) is around 75.5ms one way.
  • We should be measuring this ourselves!
• Review the predicted impact on the Apdex relative to FCS
• Make a call as to how we prioritize brokering services out of US West (Oregon)

[adborden]

I added some gotchas about SolrCloud and the image won't include our pre-installed configs. We might want to temporarily update the broker's docker registry to pull from our docker.io/gsa/catalog-solr:inventory_2.8 tag to work around this. Then split out additional issues for how to get back to the upstream image containers/registry.