Experiment results analysis generation #44
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## experiment-results #44 +/- ##
======================================================
- Coverage 82.14% 81.19% -0.96%
======================================================
Files 69 69
Lines 6005 6067 +62
======================================================
- Hits 4933 4926 -7
- Misses 1072 1141 +69 ☔ View full report in Codecov by Sentry. |
= Overview of scope of this PR Inside of this PR I implement an Experiment Results analysis approach based on the analysis tables. The basic idea is to take the analysis keys that are generated by comparing an individual observation with ground truth data. Through a very large set of rules we are able to assign individual blocking, down and ok rules based on how confident we are in that particular signal being a sign for censorship. We then take all the scores pertaining to a particular observation group relevant to a measurement and generate a `MeasurementExperimentResult` which should be backward compatible with out existing PR. Based on this we add support for generating the experiment results based on the analysis inside of the `mkanalysis` command and a simple web interface for inspecting them. In terms of performance some cursory benchmarks were run the dataset from 2023-09-01 - 2023-11-01 and it was processing data at a rate of ~7k observations per second scaling on 34 cores. = Summary of changes * Implement Experiment Result generation based on the analysis tables * Implement minimal UI for MeasurementExperimentResult * Add support for generating MeasurementExperimentResult as part of mkanalysis cli command * Add more tests for all of the above
* Make utcnow() calls timezone aware * Implement workardound for clickhouse bug mymarilyn/clickhouse-driver#388 * Implement more tests for range deletions * Refactoring of get_prev_range functions * Fix problem in experiment result generation
|
Performed some clean up of the git commit history to make it possible to merge this without needing to rely on the github squash and merge function. The reason to do that is that I want to preserve the history of certain changes in a dedicated commit (the ones about deleting dead code and the one about the clickhouse bug). This is so that in the future if we need to recover the dead code it's easy and so it's possible to easily see all the changes that are needed in order to address the clickhouse bug. |
Yeah, this seems like a good choice 💯 |
To make it easier to review I am separating all the analysis diff into a separate PR that targets the experiment_results branch.
This way it's possible to view the diff more cleanly and make it easier to understand what is going on.
Overview of scope of this PR
Inside of this PR I implement an Experiment Results analysis approach based on the analysis tables.
The basic idea is to take the analysis keys that are generated by comparing an individual observation with ground truth data. Through a very large set of rules we are able to assign individual blocking, down and ok rules based on how confident we are in that particular signal being a sign for censorship.
We then take all the scores pertaining to a particular observation group relevant to a measurement and generate a
MeasurementExperimentResultwhich should be backward compatible with out existing PR.Based on this we add support for generating the experiment results based on the analysis inside of the
mkanalysiscommand and a simple web interface for inspecting them.In terms of performance some cursory benchmarks were run the dataset from 2023-09-01 - 2023-11-01 and it was processing data at a rate of ~7k observations per second scaling on 34 cores.
Summary of changes