Extended multiple correction for group sequential, added doc for multiple correction. #179

daryadedik · 2018-01-09T17:05:44Z

completed section for multiple correction
multiple correction for group sequential

… it to false by default, added docs for multiple correction

coveralls · 2018-01-09T17:14:48Z

Coverage increased (+0.03%) to 92.179% when pulling 4b03d11 on daryadedik into f924f30 on dev.

shansfolder

Looks great! Thanks a lot!
I just have a few small suggestions for wordings.

shansfolder · 2018-01-09T17:09:23Z

docs/tutorial.rst

@@ -98,26 +98,28 @@ You may also find the description in our :ref:`API <modindex>` page.
 	* ``min_observations=20``: Minimum number of observations needed.
 	* ``nruns=10000``: Only used if assume normal is false.
 	* ``relative=False``: If relative==True, then the values will be returned as distances below and above the mean, respectively, rather than the absolute values.
+	* ``multi_test_correction=True``: Initiate multiple correction (Bonferroni correction is supported).


multi_test_correction=False by default?

ah, sure! Will correct that!

shansfolder · 2018-01-09T17:09:46Z

docs/tutorial.rst


 *group_sequential* is a frequentist approach for early stopping:

 	* ``spending_function='obrien_fleming'``: Currently we support only Obrient-Fleming alpha spending function for the frequentist early stopping decision.
 	* ``estimated_sample_size=None``: Sample size to be achieved towards the end of experiment. In other words, the actual size of data should be always smaller than estimated_sample_size.
 	* ``alpha=0.05``: Type-I error rate.
 	* ``cap=8``: Upper bound of the adapted z-score.
+	* ``multi_test_correction=True``: Initiate multiple correction (Bonferroni correction is supported).


Same here. multi_test_correction=False by default?

shansfolder · 2018-01-09T17:13:04Z

expan/core/statistics.py

@@ -57,7 +58,8 @@ def delta(x, y, assume_normal=True, percentiles=[2.5, 97.5],
            the weighted mean and confidence intervals, which is equivalent
            to the overall metric. This weighted approach is only relevant
            for ratios.
-        num_tests: number of tests or reported kpis
+        multi_test_correction (boolean): correct the confidence intervals (multiple correction problem)


how about "flag of whether the correction for multiple testing is needed" to be consistent with the docstring below?

shansfolder · 2018-01-09T17:16:02Z

expan/core/statistics.py

@@ -429,6 +430,8 @@ def normal_sample_difference(x, y, percentiles=[2.5, 97.5], relative=False):
            absolute values. In this case, the interval is mean-ret_val[0] to
            mean+ret_val[1]. This is more useful in many situations because it
            corresponds with the sem() and std() functions.
+        multi_test_correction (boolean): True or False whether the correction for multiple testing is needed.


how about "flag of whether the correction for multiple testing is needed"?

shansfolder · 2018-01-09T17:16:16Z

expan/core/statistics.py

@@ -473,6 +477,8 @@ def normal_difference(mean1, std1, n1, mean2, std2, n2, percentiles=[2.5, 97.5],
            absolute values. In	this case, the interval is mean-ret_val[0] to
            mean+ret_val[1]. This is more useful in many situations because it
            corresponds with the sem() and std() functions.
+        multi_test_correction (boolean): True or False whether the correction for multiple testing is needed.


how about "flag of whether the correction for multiple testing is needed"?

shansfolder · 2018-01-09T17:22:49Z

expan/core/early_stopping.py

@@ -56,7 +59,9 @@ def group_sequential(x,
            the end of experiment
        alpha: type-I error rate
        cap: upper bound of the adapted z-score
-
+        multi_test_correction: multiple correction flag
+        num_tests: number of tests or reported kpis used for multiple correction (default: 1, no correction is done)


how about "num_tests (integer): number of tests or reported kpis used for multiple correction. This value is only used if multi_test_correction=True"?

yes, if multi_test_correction is True I add worker_args['num_tests'] = len(self.report_kpi_names) in _delta. I made that because I don't want to make num_tests as a parameter of the method, because it's basically the number of reported kpis, which also added in parameters.

I see. 👍

shansfolder · 2018-01-09T17:23:20Z

expan/core/statistics.py

@@ -57,7 +58,8 @@ def delta(x, y, assume_normal=True, percentiles=[2.5, 97.5],
            the weighted mean and confidence intervals, which is equivalent
            to the overall metric. This weighted approach is only relevant
            for ratios.
-        num_tests: number of tests or reported kpis
+        multi_test_correction (boolean): correct the confidence intervals (multiple correction problem)


how about "flag of whether the correction for multiple testing is needed" to be consistent?

shansfolder · 2018-01-09T17:23:48Z

expan/core/early_stopping.py

@@ -56,7 +59,9 @@ def group_sequential(x,
            the end of experiment
        alpha: type-I error rate
        cap: upper bound of the adapted z-score
-
+        multi_test_correction: multiple correction flag


how about "flag of whether the correction for multiple testing is needed" to be consistent?

could more clear, thanks!

shansfolder · 2018-01-09T17:24:04Z

expan/core/experiment.py

        """
        Perform subgroup analysis on date partitioning each day from start day till end date. Produces non-cumulative
        delta and CIs for each subgroup.
+        Args:
+            multi_test_correction (boolean): True or False whether the correction for multiple testing is needed.


how about "flag of whether the correction for multiple testing is needed"?

shansfolder · 2018-01-09T17:24:29Z

expan/core/experiment.py

        """
        Perform subgroup analysis.
        Args:
            feature_name_to_bins (dict): a dict of feature name (key) to list of Bin objects (value). 
                                      This dict defines how and on which column to perform the subgroup split.
+            multi_test_correction (boolean): True or False whether the correction for multiple testing is needed.


how about "flag of whether the correction for multiple testing is needed"?

sure, will make everywhere the same.

mkolarek · 2018-01-09T17:38:02Z

docs/glossary.rst

@@ -83,11 +83,26 @@ You can find links to our detailed documentations for

 Subgroup analysis
 ------------------------------------
-Subgroup analysis in ExaAn will select subgroup (which is a segment of data) based on the input argument, and then perform a regular delta analysis per subgroup as described before. 
+Subgroup analysis in ExaAn will select subgroup (which is a segment of data) based on the input argument, and then perform a regular delta analysis per subgroup as described before.


sorry there's just a typo here (ExpAn)

ah, ok, will fix =)

coveralls · 2018-01-09T17:58:45Z

Coverage increased (+0.03%) to 92.179% when pulling 7d3231e on daryadedik into f924f30 on dev.

mkolarek

looks great

daryadedik · 2018-01-10T10:16:33Z

added multiple correction for bootstrap (also False by default) since it was there before and it also make sense to have it for bootstrap.

coveralls · 2018-01-10T10:24:03Z

Coverage increased (+0.05%) to 92.201% when pulling 2d918e2 on daryadedik into f924f30 on dev.

added multiple correction for sga, delta and group sequential and set…

4b03d11

… it to false by default, added docs for multiple correction

shansfolder approved these changes Jan 9, 2018

View reviewed changes

mkolarek reviewed Jan 9, 2018

View reviewed changes

changed wording and default for multi_test_correction flag

7d3231e

daryadedik requested review from gbordyugov, shansfolder and mkolarek January 9, 2018 17:54

mkolarek approved these changes Jan 10, 2018

View reviewed changes

shansfolder approved these changes Jan 10, 2018

View reviewed changes

added multiple correction for the bootstrap

2d918e2

daryadedik merged commit bb85bd5 into dev Jan 10, 2018

daryadedik deleted the daryadedik branch January 10, 2018 11:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extended multiple correction for group sequential, added doc for multiple correction. #179

Extended multiple correction for group sequential, added doc for multiple correction. #179

daryadedik commented Jan 9, 2018 •

edited

Loading

coveralls commented Jan 9, 2018 •

edited

Loading

shansfolder left a comment

shansfolder Jan 9, 2018

daryadedik Jan 9, 2018

shansfolder Jan 9, 2018

shansfolder Jan 9, 2018

shansfolder Jan 9, 2018

shansfolder Jan 9, 2018

shansfolder Jan 9, 2018

daryadedik Jan 9, 2018 •

edited

Loading

shansfolder Jan 9, 2018

shansfolder Jan 9, 2018

shansfolder Jan 9, 2018

daryadedik Jan 9, 2018

shansfolder Jan 9, 2018

shansfolder Jan 9, 2018

daryadedik Jan 9, 2018

mkolarek Jan 9, 2018

daryadedik Jan 9, 2018

coveralls commented Jan 9, 2018 •

edited

Loading

mkolarek left a comment

daryadedik commented Jan 10, 2018

coveralls commented Jan 10, 2018 •

edited

Loading

Extended multiple correction for group sequential, added doc for multiple correction. #179

Extended multiple correction for group sequential, added doc for multiple correction. #179

Conversation

daryadedik commented Jan 9, 2018 • edited Loading

coveralls commented Jan 9, 2018 • edited Loading

shansfolder left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daryadedik Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Jan 9, 2018 • edited Loading

mkolarek left a comment

Choose a reason for hiding this comment

daryadedik commented Jan 10, 2018

coveralls commented Jan 10, 2018 • edited Loading

daryadedik commented Jan 9, 2018 •

edited

Loading

coveralls commented Jan 9, 2018 •

edited

Loading

daryadedik Jan 9, 2018 •

edited

Loading

coveralls commented Jan 9, 2018 •

edited

Loading

coveralls commented Jan 10, 2018 •

edited

Loading