feat: report daily participants to public stats

[bajtos]

Create a new spark_stats table to keep track of daily participants:

SELECT day::TEXT, COUNT(DISTINCT participant_id) as count
FROM daily_participants GROUP BY day;

This table allows us to correctly calculate monthly participants too:

SELECT
  date_trunc('month', day)::DATE::TEXT as month,
  COUNT(DISTINCT participant_id) as count
FROM daily_participants
GROUP BY month;

TODO:

• add tests

Links:

• More evaluation stats & visualisations & alerts #79

[juliangruber]

..otherwise LGTM!

[juliangruber]

Performance wise, are we ok with running 2 SQL queries for every participant, in series? And if it fails midway, will we have inconsistent data? Playing devil's advocate here

[bajtos]

Great questions! 💯

I'll think about this tomorrow.

[bajtos]

Yeah, I was focused so much on storage efficiency and the querying side, that I completely neglected the writing side.

With ~2k participants per round, my current implementation would run 4k SQL queries. That would take forever to complete.

Thanks for flagging this early! 😍

[bajtos]

Cool, I found a neat trick: We can use SELECT UNNEST($1::TEXT[]) to build a query that accepts a JavaScript array parameter and converts the items into result rows.

Then I can change INSERT...VALUES to INSERT...SELECT to leverage this mechanism.

The changes were a bit more involved (see 7a2d14d), but I expect the performance to be excellent.

[juliangruber]

Great work 😍