Forward User-Agent from AWS CloudFront to catalog app

[FuhuXia]

User Story

In order to better understand the origin of requests coming to catalog app, data.gov team wants see User-Agent header info in catalog app logs.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN all catalog.data.gov requests go thru CloudFront -> catalog-proxy -> catalog-web
  AND request to /api/action/status_show is not cached
  WHEN I run command curl -A "my-test-user-agent" https://catalog.data.gov/api/action/status_show
  THEN the access logs capture this request
  AND my-test-user-agent is found when searching the log
• GIVEN command curl -I https://cata.data.gov/dataset is run
  WHEN the command is run for the 2nd time with random string in user-agent
  curl -s -I -A "!@#$%^" https://catalog.data.gov/dataset | grep x-cache
  THEN x-cache: Hit from cloudfront as output to show cached result is hit.

Background

We want the original user-agent header info in the request instead of all requests are marked from "Amazon CloudFront", but we don't want user-agent as a cache-key to be cached against.

Security Considerations (required)

No

Sketch

Follow this AWS instruction to set up an Origin Request policy.

[nickumia-reisys]

Conflicting documentation 😠 😢 💢 💢
https://aws.amazon.com/blogs/networking-and-content-delivery/amazon-cloudfront-announces-cache-and-origin-request-policies/

Forwarding information such as the User-Agent to the origin for analytics/logging but without serving different content variants based on device type (now you can forward the user-agent header and exclude it from the cache-key)

https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorCustomOrigin.html#request-custom-headers-behavior

Whether you can configure CloudFront to cache objects based on header values for that header.
You can configure CloudFront to cache objects based on values in the Date and User-Agent headers, but we don't recommend it. These headers have many possible values, and caching based on their values would cause CloudFront to forward significantly more requests to your origin.

[FuhuXia]

I think this is the Origin request policy we need:
Managed-UserAgentRefererHeaders

[nickumia-reisys]

We can't use that because it drops all Query Strings:

But we made a custom one that was similar to it (if it's too much we can just do the user-agent and referer:

[nickumia-reisys]

I think our custom solution was working, it was just the config on catalog-proxy that was causing it to break

[FuhuXia]

We can modify the nginx config. Detecting "Amazon CloudFront" was a convenient but temporary solution.
https://github.com/GSA/catalog.data.gov/blob/fedf828f33985c691007065ca0de646abc01e1a0/proxy/nginx-cloudfront.conf#L16-L23

[nickumia-reisys]

@Jin-Sun-tts was able to create our domains on cloud.gov with the revived command:

cf create-private-domain gsa-datagov catalog-dev.data.gov
cf create-private-domain gsa-datagov catalog-stage.data.gov

Interim report:

• The solution to this issue is creating a custom Cache Policy and a custom Origin Request Policy (with the specifications above).
  • General Steps in Cloudfront on AWS Console:
    • Create Cache Policy
      • Go to Policies in the menu on the left.
      • Click on Create Cache Policy in the Custom Policies Section under the Cache Tab.
        • NOTE: Policy only needs to be created once per account.
      • Fill out the details.
        • Note: TTL specs vary based on catalog route (api vs. reports vs. sitemap vs default)
          • We should document this somewhere...
      • For Cache Key Settings, choose either Host or no keys at all.
      • For Compression Support, leave default?
      • Save.
    • Create Origin Request Policy
      • Go to Policies in the menu on the left.
      • Click on Create Origin Request Policy in the Custom Policies Section under the Origin Request Tab.
        • NOTE: Policy only needs to be created once per account.
      • Fill out the details.
      • For Headers, include at least Referer, User-Agent, Host and CloudFront-Forwarded-Proto (but we should be able to select All Viewer Headers
      • For Query Strings, select All.
      • For Cookies, select None.
      • Save.
    • Use the Policies
      • Go to a Cloudfront Distribution
      • Go the to Behaviors Tab.
      • Edit a specific behavior.
      • Under the Cache key and origin requests section, select Cache policy and origin request policy (recommended)
      • Select the custom cache and origin request policies you just created.
      • Save changes.
• Forwarding the user-agent is solved. However, the nginx config on catalog-proxy needs to be updated. So this will be blocked until that issue is solved, Limit catalog-proxy traffic to only from Cloudfront #4413

[FuhuXia]

One Origin Request Policy CatalogProdOriginPolicy was created with the following settings.

Headers - Include the following headers - Referer; User-Agent; Host
Cookies - None
Query strings - All

Three Cache Policies were created

• catalogProdCachePolicy as default

Minimum TTL (seconds) 300
Maximum TTL (seconds) 31536000
Default TTL (seconds) 86400

Headers - None
Cookies - None
Query strings - All

Gzip Enabled
Brotli Enabled

• catalogProdCachePolicy-REPORT for /report/*

Minimum TTL (seconds) 86400
Maximum TTL (seconds) 86400
Default TTL (seconds) 86400

Headers - None
Cookies - None
Query strings - All

Gzip Enabled
Brotli Enabled

• catalogProdCachePolicy-NOCACHE for /api/* and /sitemap/*

Minimum TTL (seconds) 0
Maximum TTL (seconds) 0
Default TTL (seconds) 0

Headers - None
Cookies - None
Query strings - None

Gzip Disabled
Brotli Disabled

Staging and Development have follow prod and have same settings.

[nickumia-reisys]

Unblocked by completion of

• Limit catalog-proxy traffic to only from Cloudfront #4413

User-agents are in the logs of catalog-proxy app.