O+M 2023-06-02

[Jin-Sun-tts]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Production State/Actions

• README

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Tracking Update
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[Jin-Sun-tts]

Tuesday 05/30

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync: packages need to be updated/added to Solr

• Tracking Update: 92671 package indexes to be rebuilt starting from 2023-05-28 00:00:00

Check Harvesting Emails

• Catalog:
• no harvesting job reported yet at 11am EST
• 10 harvesting job errors reported by 5pm EST, most of them are identifier missing related.

Other

• New Relic Alerts Triaged

• Triage DMARC Report from Google:

[Jin-Sun-tts]

Wednesday 05/31

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  1 packages need to be removed from Solr
  8291 packages need to be updated/added to Solr

• Tracking Update: 104112 package indexes to be rebuilt starting from 2023-05-28 00:00:00

Check Harvesting Emails

• Catalog:
• no harvesting job reported yet at 11am EST

Other

• New Relic Alerts

• Triage DMARC Report from Google:

[Jin-Sun-tts]

Thursday 06/01

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  176 packages need to be removed from Solr
  8188 packages need to be updated/added to Solr

• Tracking Update: 90867 package indexes to be rebuilt starting from 2023-05-29 00:00:00

Check Harvesting Emails

• Catalog:
• no harvesting job reported yet at 10am EST

Other

• New Relic Alerts

• Triage DMARC Report from Google:

[Jin-Sun-tts]

Friday 06/02

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  27 packages need to be removed from Solr
  23911 packages need to be updated/added to Solr

• Tracking Update:
  88294 package indexes to be rebuilt starting from 2023-05-30 00:00:00

Check Harvesting Emails

• Catalog:
• 19 harvesting job errors reported yet at 9am E
  
  ST
  

Other

• New Relic Alerts

• Triage DMARC Report from Google:

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Check Catalog Solr
• Catalog Dupe Check