O+M 2023-11-3

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• Check Production State/Actions

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/change tracking update from nightly job to weekly #4345])
• Check Catalog Solr
• Catalog Dupe Check

Monthly Checklist

• Invicti Scan

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[robert-bryson]

[[2023-10-30]]

• Harvest emails ok - normal identifier errors
• db solr sync ok, see tomorrow for notes on stage issue
• NR alert: https://gsa-tts.slack.com/archives/C4RGAM1Q8/p1698674545708059
  • https://gsa-tts.slack.com/archives/C4RGAM1Q8/p1698695541436739

[[2023-10-31]]

• db solr sync failed on staging: timed out (360mins) last night
  • although same as yesterdays:
  • prod ok:
• Github actions ok save ^^
• NR alerts:
  • Inventory prod: open, close
  • Catalog web prod: open, closed
• 1 count DMARC SPF fail from source IP 96.47.7.130, all other pass SPF/DKIM

[robert-bryson]

[[2023-11-01]]

• Solr follow memory alert
• Inventory deploy action fail
• User updates from Crystal:
  • https://github.com/GSA/datagov-account-management/issues/552 - messed this one up, deleted user but have recreated
  • https://github.com/GSA/datagov-account-management/issues/553
  • https://github.com/GSA/datagov-account-management/issues/554
  • https://github.com/GSA/datagov-account-management/issues/559
• Harvest emails:
  • Issue with ISO conversion due to unable to access url
  • Otherwise look ok
• DMARC preview report has fails, but also email saying issue with GSA IT emails might be related?
• NR alerts
  • Catalog web prod: open, close
• DB solr sync ok
  • 

[robert-bryson]

[[2023-11-02]]

• NR alert
  • Catalog web prod: open, close
• Catalog prod fetch deploy timed out (Action logs), success on another run. Thanks Fuhu.
• DB solr sync ok, but manual delete number slowly rising:
  • 
• Harvest emails ok. Seems like there are a couple more failures to transform ISO than normal, but hard to tell.
• DMARC report passes
• Stuck jobs failure is now passing