Skip to content

Latest commit

 

History

History
86 lines (82 loc) · 4.36 KB

README.md

File metadata and controls

86 lines (82 loc) · 4.36 KB
Raw

Mass Murder Canada Website

https://www.massmurdercanada.org/

GOALS

  1. ✅ Show this data in a more complete format: https://bit.ly/canadian-multiple-murders
  2. Partial ✅ Be able to search and show summaries: by province, by City, same tabs as in the spreadsheet
  3. Show graphs of data across time.
  4. ✅ Grab web page data and store in DB.
  5. Summaries of web pages by OpenAI.
  6. Summarize ALL of the news articles for a particular record.
  7. ✅ Store in Postgres SQLite.
  8. ✅ Served by Cloudflare.
  9. ✅ Import spreadsheet via CSV.
  10. ✅ Use some of the new Golang patterns I've been learning.

TODO

  • Add logger to config
  • Add some debug logging
  • Be able to insert fake data via curl.
  • Redis: Make sure to add Record to list - so we can grab groups of them.
  • Switch to spf13/cobra
  • Split out the CLI.
  • Be able to insert Faked data into Repository via CLI
  • Had to turn off client side caching for miniredis tests - can we ONLY do that for tests? https://github.com/redis/rueidis#disable-client-side-caching
  • Redis Implementation of NewsStory
  • Link NewsStory to Record
  • Redis: Make sure to add NewsStory to list - so we can see groups of them.
  • Import via line of CSV.
  • Add GetAll() to interface.
  • Create GetAll() service endpoint.
  • Display GetAll() as simple table
  • Move JSON enpoints to /api/v1
  • From / click to display individual records
  • Display Record on the Web, with stories gotten from related keys.
  • Remove flags from config - just set default config.
  • Add flags as neccessary to cmd - then run the required WithOpt config func.
  • Allow actual files to be downloaded if there's no route.
  • Better HTML templates - some sort of CSS framework.
  • Prometheus endpoint for /metrics
  • Includes for HTML headers and things?
  • Need to deal with nullable booleans - there's lots of data we don't have.
  • e.Use(middleware.RequestID())
  • Super hacky: Be able to see different types of Records: firearms only, mass shooting only, OIC firearms only, licensed mass shootings
  • Add totals to group pages?
  • Add tests for Group pages - get test coverage back to +80%
  • ACLs for /api/ endpoints - protected by JWT requirement.
  • Dockerfile so we can run it in Docker.
  • Docker Compose file
  • Add autoTLS - https://echo.labstack.com/cookbook/auto-tls/
  • Do we need a page cache? https://github.com/victorspringer/http-cache - added https://github.com/SporkHubr/echo-http-cache
  • Too many Redis connections - need to close connections as much as possible.
  • Import with TLS - needs to handle new ports and domain name and HTTPS
  • For Records and News Stories - more efficient way to download them all at once when using Redis. Right now we're opening and closing a new connection for each one.
  • Be able to enable / disable traces and profiling using flags.
  • Monitoring for uptime.
  • Add groups by Province and City?
  • Be able to disabled caching by flag.
  • SQLite Implementation of Record
  • SQLite Implementation of NewsStory
  • SQLite file creation if it doesn't exist.
  • SQLite file migration if it isn't already done.
  • Cross compile for Linux w/SQLite3?
  • Deploy to domain name? massmurdercanada.org
  • Put Behind Cloudflare?
  • Will need to run migrations later on - how do we do that? bin/ff migrate?
  • SQLite stream to storage?
  • Integrate News Stories with Web downloading.
  • Get CRUD working for Record
  • Get CRUD working for NewsStory
  • Integrate News Stories with AI Summaries.
  • Need some additional JWT work: actually check the claims - how to get them generated and asssigned to me?
  • Postgres Implementation of Record
  • Postgres Implementation of NewsStory
  • skaffold + k8s files?
  • Do I need to add some contexts to track requests?
  • Add /healthz which tests for health of DB?
  • Some sort of Analytics
  • Integrate StatsCan homicide records.
  • Add Feature Flag Interface - use in-memory / augmented by written file - https://openfeature.dev/ - https://github.com/open-feature/flagd
  • Better Groups/Tags interface.
  • Make it actually be organized/designed?
  • Add Search?
  • https://echo.labstack.com/cookbook/graceful-shutdown/
  • Might need to do some more performance tuning once we have more traffic.