O+M 2024-09-23

[FuhuXia]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• Check Production State/Actions

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• Check auto generated O&M tickets from no status column
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• DB-Solr Sync
• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/change tracking update from nightly job to weekly #4345])
• Check Catalog Solr
• Catalog Dupe Check
• Check user management requests
• Check Snyk Dashboard and cleanup

Monthly Checklist

• Invicti Scan

ad-hoc checklist

• audit/review applications on cloud foundry and determine what can be stopped and/or deleted.

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[FuhuXia]

facebook bots have made 3 million request to /_tracking and populated catalog popular views since 3 weeks ago. A PR is created for exclude facebook traffic from the stats.

+++++++++++++++++++++++++++++++

UPDATE:

PR deployed. Facebook bots disappeared from "non-bot _tracking by user agent" dashboard on NewRelic.

Before: facebook bot counts for 70% of /_tracking traffic.

After: top /_tracking traffic are from normal user agents.

[FuhuXia]

Harvester /harvest/nist/ continues to be stuck. Need to investigate.

Update:

GitHub ticket created.

Update 2:

Issue fixed. Now the job can complete, not getting stuck.

[FuhuXia]

Source /harvest/education-json failed for the past 7 days. The source http://www.ed.gov/data.json is 404.

Update:

ed.gov site went thru some renovation. The working url is https://data.ed.gov/data.json.

Update 2

Now original http://www.ed.gov/data.json is correctly redirected to https://data.ed.gov/data.json. But we leave the source url as https://data.ed.gov/data.json.

[FuhuXia]

Facebook bots are crawling catalog in the silly manner, blindly following all possible found links and making requests to various tags combinations.
The requests rate seems to be stabilized at 12 requests per seconds.

[FuhuXia]

We locked down DB for the weekend for postgresql version bump works, during which there is no harvesting, no DB writing activity, therefore no solr replicating. To my surprise, SOLR followers' memory usage stayed flat for the whole time.

This shows that solr replicating activity is the one this is directly associated with solr ECS memory leak.

Update

based on previous clue, we removed the solr traffic 60%-30%-10% allocation on AWS balance group console and gave each solr follower equal amount the traffic.