some general changes to the various forest() functions: argument header is now TRUE by default, the y-axis is now created with yaxs="i", and the y-axis limits have been tweaked slightly in accordance
forest.rma() and the various addpoly() functions now provides multiple styles for drawing the prediction interval via the predstyle argument
forest.rma() and addpoly.rma() now write out the default label (instead of an abbreviation) for the model results; as before, the label can be changed via the mlab argument
permutest.rma.uni() gains a btt argument and permutest.rma.ls() gains btt and att arguments
selmodel() gains a subset argument (to specify a subset of studies to which the selection function should apply)
nobs() now just returns the number of estimates, not the effective number of observations
added measure "PRZ" to escalc()
added measures "PRZ", "CLES", "AUC", "CLESN", and "AUCN" to escalc()
escalc() gains a flip argument
fixed predict() ignoring the level argument for robust.rma objects obtained with clubSandwich=TRUE
some tweaks were made to vcalc() to speed up the calculations (by James Pustejovsky)
added transformation function transf.dtoovl() for transforming standardized mean differences to overlapping coefficient values
added transformation function transf.dtoovl() (for transforming standardized mean differences to overlapping coefficient values) and `transf.dtocliffd() (for transforming standardized mean differences to Cliff’s delta values)
escalc() gains a correct argument (to specify whether a bias correction should be applied)
better handling of outlist="minimal"
added more tests
a character string to specify which effect size or outcome measure should be calculated. See ‘Details’ for possible options and how the data needed to compute the selected effect size or outcome measure should then be specified (i.e., which of the following arguments need to be used).
a character string to specify which effect size or outcome measure should be calculated (e.g., "SMD", "ZCOR", "OR"). See ‘Details’ for possible options and how the data needed to compute the selected effect size or outcome measure should then be specified (i.e., which of the following arguments need to be used).
These arguments pertain to data input:
+ +vector with the \(2 \times 2\) table frequencies (upper left cell).
optional (logical or numeric) vector to specify the subset of studies for which the measure should be calculated. See the ‘Value’ section for more details.
These arguments pertain to handling of zero cells/counts/frequencies:
+ +a non-negative number to specify the amount to add to zero cells, counts, or frequencies. See ‘Details’.
logical to specify whether studies with no cases/events (or only cases) in both groups should be dropped when calculating the observed effect sizes or outcomes. See ‘Details’.
a character string to specify the type of sampling variances to calculate. See ‘Details’.
These arguments pertain to the computations:
+ +a character string to specify the type of sampling variances to calculate. Can also be a vector. See ‘Details’.
logical to specify whether a bias correction should be applied to the effect sizes or outcomes (the default is TRUE).
These arguments pertain to the formatting of the returned data frame:
+ +character vector with two elements to specify the name of the variable for the observed effect sizes or outcomes and the name of the variable for the corresponding sampling variances (the defaults are "yi" and "vi").
Before a meta-analysis can be conducted, the relevant results from each study must be quantified in such a way that the resulting values can be further aggregated and compared. Depending on (a) the goals of the meta-analysis, (b) the design and types of studies included, and (c) the information provided therein, one of the various effect sizes or outcome measures described below may be appropriate for the meta-analysis and can be computed with the escalc function.
The measure argument is a character string to specify the outcome measure that should be calculated (see below for the various options), arguments ai through ni are then used to specify the information needed to calculate the various measures (depending on the chosen outcome measure, different arguments need to be specified), and data can be used to specify a data frame containing the variables given to the previous arguments. The add, to, and drop00 arguments may be needed when dealing with frequency or count data that may need special handling when some of the frequencies or counts are equal to zero (see below for details). Finally, the vtype argument is used to specify how the sampling variances should be estimated (again, see below for details).
The measure argument is a character string to specify the outcome measure that should be calculated (see below for the various options), arguments ai through ni are then used to specify the information needed to calculate the various measures (depending on the chosen outcome measure, different arguments need to be specified), and data can be used to specify a data frame containing the variables given to the previous arguments. The add, to, and drop00 arguments may be needed when dealing with frequency or count data that needs special handling when some of the frequencies or counts are equal to zero (see below for details). Finally, the vtype argument is used to specify how the sampling variances should be computed (again, see below for details).
To provide a structure to the various effect sizes or outcome measures that can be calculated with the escalc function, we can distinguish between measures that are used to:
| (1) | contrast two independent (either experimentally created or naturally occurring) groups, | |
| (2) | describe the direction and strength of the association between two variables, | |
| (3) | summarize some characteristic or attribute of individual groups, or | |
| (4) | quantify change within a single group or the difference between two matched/paired samples. |
Furthermore, where appropriate, we can further distinguish between measures that are applicable when the characteristic, response, or dependent variable assessed within the individual studies is:
| (a) | a quantitative variable (e.g., amount of depression as assessed by a rating scale), | |
| (b) | a dichotomous (binary) variable (e.g., remission versus no remission), | |
| (c) | a count of events per time unit (e.g., number of migraines per year), or | |
| (d) | a mix of the types above. |
Below, these number and letter codes are used (also in combination) to make it easier to quickly find a measure suitable for a particular meta-analysis (e.g., search for (1b) to find measures that describe the difference between two groups with respect to a dichotomous variable or (2a) for measures that quantify the association between two quantitative variables).
In many meta-analyses, the goal is to synthesize the results from studies that compare or contrast two groups. The groups may be experimentally defined (e.g., a treatment and a control group created via random assignment) or may occur naturally (e.g., men and women, employees working under high- versus low-stress conditions, people/animals/plants exposed to some environmental risk factor versus those not exposed).
+In many meta-analyses, the goal is to synthesize the results from studies that compare or contrast two groups. The groups may be experimentally defined (e.g., a treatment and a control group created via random assignment) or may occur naturally (e.g., men and women, employees working under high- versus low-stress conditions, people/animals/plants exposed to some environmental risk factor versus those not exposed, patients versus controls).
When the response or dependent variable assessed within the individual studies is measured on a quantitative scale, it is customary to report certain summary statistics, such as the mean and standard deviation of the observations within the two groups (in case medians, min/max values, and quartiles are reported, see conv.fivenum for a function that can be used to estimate means and standard deviations from such statistics). The data layout for a study comparing two groups with respect to such a variable is then of the form:
| mean | standard deviation | group size | ||||
| group 1 | m1i | sd1i | n1i | |||
| group 2 | m2i | sd2i | n2i |
where m1i and m2i are the observed means of the two groups, sd1i and sd2i are the observed standard deviations, and n1i and n2i denote the number of individuals in each group. The raw mean difference, the standardized mean difference, and the (log transformed) ratio of means (also called the log ‘response ratio’) are useful outcome measures when meta-analyzing studies of this type.
When the response or dependent variable assessed within the individual studies is measured on a quantitative scale, it is customary to report certain summary statistics, such as the mean and standard deviation of the observations within the two groups (in case medians, min/max values, and quartiles are reported, see conv.fivenum for a function that can be used to estimate means and standard deviations from such statistics). The data layout for a study comparing two groups with respect to such a variable is then of the form:
| mean | standard deviation | group size | ||||
| group 1 | m1i | sd1i | n1i | |||
| group 2 | m2i | sd2i | n2i |
where m1i and m2i are the observed means of the two groups, sd1i and sd2i are the observed standard deviations, and n1i and n2i denote the number of individuals in each group.
Measures for Differences in Central Tendency
+Often, interest is focused on differences between the two groups with respect to their central tendency. The raw mean difference, the standardized mean difference, and the (log transformed) ratio of means (also called the log ‘response ratio’) are useful outcome measures when meta-analyzing studies of this type.
The options for the measure argument are then:
"MD" for the raw mean difference (e.g., Borenstein, 2009),
"SMD" for the standardized mean difference (Hedges, 1981),
"SMD1" for the standardized mean difference where the mean difference is divided by the standard deviation of the second group (and "SMD1H" for the same but with heteroscedastic population variances),
"ROM" for the log transformed ratio of means (Hedges et al., 1999; Lajeunesse, 2011).
The raw mean difference is simply \((\textrm{m1i}-\textrm{m2i})\), while the standardized mean difference is given by \((\textrm{m1i}-\textrm{m2i})/\textrm{sdi}\). For measure="SMD", \(\textrm{sdi} = \sqrt{\frac{(\textrm{n1i}-1)\textrm{sd1i}^2 + (\textrm{n2i}-1)\textrm{sd2i}^2}{\textrm{n1i}+\textrm{n2i}-2}}\) is the pooled standard deviation of the two groups (assuming homoscedasticity of the population variances). For measure="SMDH", \(\textrm{sdi} = \sqrt{\frac{\textrm{sd1i}^2 + \textrm{sd2i}^2}{2}}\) is the square root of the average variance (allowing for heteroscedastic population variances). Finally, for measure="SMD1" and measure="SMD1H", \(\textrm{sdi} = \textrm{sd2i}\) (note: for measure="SMD1", only sd2i needs to be specified and sd1i is ignored).
For measure="SMD", the positive bias in the standardized mean difference (i.e., in a Cohen's d value) is automatically corrected for within the function, yielding Hedges' g (Hedges, 1981). Similarly, the analogous bias correction is applied for measure="SMDH" (Bonett, 2009), measure="SMD1" (Hedges, 1981), and measure="SMD1H".
For measure="SMD", the positive bias in the standardized mean difference (i.e., in a Cohen's d value) is automatically corrected for within the function, yielding Hedges' g (Hedges, 1981). Similarly, the analogous bias correction is applied for measure="SMDH" (Bonett, 2009), measure="SMD1" (Hedges, 1981), and measure="SMD1H". With correct=FALSE, these bias corrections can be switched off.
For measure="ROM", the log is taken of the ratio of means (i.e., \(\log(\textrm{m1i}/\textrm{m2i})\)), which makes this outcome measure symmetric around 0 and results in a sampling distribution that is closer to normality. Hence, this measure cannot be computed when m1i and m2i have opposite signs (in fact, this measure is only meant to be used for ratio scale measurements, where both means should be positive anyway).
For measure="SMD", if the means and standard deviations are unknown for some studies, but the standardized mean differences (Cohen's d values) are directly available (e.g., if they are reported in those studies), then these can be specified via argument di. Also, if the t-statistics from an independent samples (Student's) t-test are available for some studies, one can specify those values via argument ti, which are then transformed into the corresponding standardized mean differences within the function (the sign of the t-statistics is then taken to be the sign of the standardized mean differences). If only the (two-sided) p-values corresponding to the t-tests are known, one can specify those values via argument pi (which are then transformed into the t-statistics and then further into the standardized mean differences). However, since a two-sided p-value does not carry information about the sign of the test statistic (and hence neither about the standardized mean difference), the sign of the p-values (which can be negative) is used as the sign of the standardized mean differences (e.g., escalc(measure="SMD", pi=-0.018, n1i=20, n2i=20) yields a negative standardized mean difference of -0.7664). See here for a more detailed illustration of using the ti and pi arguments.
For measure="MD", one can choose between vtype="LS" (the default) and vtype="HO". The former computes the sampling variances without assuming homoscedasticity (i.e., that the true variances of the measurements are the same in group 1 and group 2 within each study), while the latter assumes homoscedasticity (equations 12.5 and 12.3 in Borenstein, 2009, respectively). For measure="SMD", one can choose between vtype="LS" (the default) for the usual large-sample approximation to compute the sampling variances (equation 8 in Hedges, 1982), vtype="LS2" to compute the sampling variances as described in Borenstein (2009; equation 12.17), vtype="UB" to compute unbiased estimates of the sampling variances (equation 9 in Hedges, 1983), and vtype="AV" to compute the sampling variances with the usual large-sample approximation but plugging the sample-size weighted average of the Hedges' g values into the equation. The same choices also apply to measure="SMD1". For measure="ROM", one can choose between vtype="LS" (the default) for the usual large-sample approximation to compute the sampling variances (equation 1 in Hedges et al., 1999), vtype="HO" to compute the sampling variances assuming homoscedasticity (the unnumbered equation after equation 1 in Hedges et al., 1999), vtype="AV" to compute the sampling variances assuming homoscedasticity of the coefficient of variation within each group across studies, and vtype="AVHO" to compute the sampling variances assuming homoscedasticity of the coefficient of variation for both groups across studies (see Nakagawa et al., 2023, for details on the latter two options and why they are interesting).
For measure="SMD", if the means and standard deviations are unknown for some studies, but the standardized mean differences (Cohen's d values) are directly available (e.g., if they are reported in those studies), then these can be specified via argument di. Also, if the t-statistics from an independent samples (Student's) t-test are available for some studies, one can specify those values via argument ti, which are then transformed into the corresponding standardized mean differences within the function (the sign of the t-statistics is then taken to be the sign of the standardized mean differences). If only the two-sided p-values corresponding to the t-tests are known, one can specify those values via argument pi (which are then transformed into the t-statistics and then further into the standardized mean differences). However, since a two-sided p-value does not carry information about the sign of the test statistic (and hence neither about the standardized mean difference), the sign of the p-values (which can be negative) is used as the sign of the standardized mean differences (e.g., escalc(measure="SMD", pi=-0.018, n1i=20, n2i=20) yields a negative standardized mean difference of -0.7664). See here for a more detailed illustration of using the ti and pi arguments.
For measure="MD", one can choose between vtype="LS" (the default) and vtype="HO". The former computes the sampling variances without assuming homoscedasticity (i.e., that the true variances of the measurements are the same in group 1 and group 2 within each study), while the latter assumes homoscedasticity (equations 12.5 and 12.3 in Borenstein, 2009, respectively). For measure="SMD", one can choose between vtype="LS" (the default) for the usual large-sample approximation to compute the sampling variances (equation 8 in Hedges, 1982), vtype="LS2" to compute the sampling variances as described in Borenstein (2009; equation 12.17), vtype="UB" to compute unbiased estimates of the sampling variances (equation 9 in Hedges, 1983), and vtype="AV" to compute the sampling variances with the usual large-sample approximation but plugging the sample-size weighted average of the Hedges' g values into the equation. The same choices also apply to measure="SMD1". For measure="ROM", one can choose between vtype="LS" (the default) for the usual large-sample approximation to compute the sampling variances (equation 1 in Hedges et al., 1999), vtype="HO" to compute the sampling variances assuming homoscedasticity (the unnumbered equation after equation 1 in Hedges et al., 1999), vtype="AV" to compute the sampling variances assuming homoscedasticity of the coefficient of variation within each group across studies, and vtype="AVHO" to compute the sampling variances assuming homoscedasticity of the coefficient of variation for both groups across studies (see Nakagawa et al., 2023, for details on the latter two options and why they can be advantageous).
Datasets corresponding to data of this type are provided in dat.normand1999, dat.curtis1998, and dat.gibson2002.
Measures for Variability Differences
Interest may also be focused on differences between the two groups with respect to their variability. Here, the (log transformed) ratio of the coefficient of variation of the two groups (also called the coefficient of variation ratio) can be a useful measure (Nakagawa et al., 2015). If focus is solely on the variability of the measurements within the two groups, then the (log transformed) ratio of the standard deviations (also called the variability ratio) can be used (Nakagawa et al., 2015). For the latter, one only needs to specify sd1i, sd2i, n1i, and n2i. The options for the measure argument are:
"CVR" for the log transformed coefficient of variation ratio,
"VR" for the log transformed variability ratio.
Measure "CVR" is computed with \(\log\mathopen{}\left(\left(\textrm{sd1i}/\textrm{m1i}\right) \middle/ \left(\textrm{sd2i}/\textrm{m2i}\right) \right)\mathclose{}\), while "VR" is simply \(\log(\textrm{sd1i}/\textrm{sd2i})\), but note that a slight bias correction is applied for both of these measures (Nakagawa et al., 2015). Also, the sampling variance for measure="CVR" is computed as given by equation 12 in Nakagawa et al. (2015), but without the ‘\(-2 \rho \ldots\)’ terms, since for normally distributed data (which we assume here) the mean and variance (and transformations thereof) are independent.
Measure "CVR" is computed with \(\log\mathopen{}\left(\left(\textrm{sd1i}/\textrm{m1i}\right) \middle/ \left(\textrm{sd2i}/\textrm{m2i}\right) \right)\mathclose{}\), while "VR" is simply \(\log(\textrm{sd1i}/\textrm{sd2i})\), but note that a slight bias correction is applied for both of these measures (Nakagawa et al., 2015) unless correct=FALSE. Also, the sampling variances for measure="CVR" are computed as given by equation 12 in Nakagawa et al. (2015), but without the ‘\(-2 \rho \ldots\)’ terms, since for normally distributed data (which we assume here) the mean and variance (and transformations thereof) are independent.
Measures for Stochastic Superiority
+Another way to quantify the difference between two groups is in terms of the ‘common language effect size’ (CLES) (McGraw & Wong, 1992). This measure provides an estimate of \(P(X > Y)\), that is, the probability that a randomly chosen person from the first group has a larger value on the response variable than a randomly chosen person from the second group (or in case \(X\) and \(Y\) values can be tied, we define the measure as \(P(X > Y) + \frac{1}{2} P(X = Y)\)). This measure is identical to the area under the curve (AUC) under the receiver operating characteristic (ROC) curve (e.g., for a diagnostic test or more broadly for a binary classifier) and the ‘concordance probability’ (or c-statistic) and is directly related to the \(U\) statistic from the Mann-Whitney U test (i.e., \(\textrm{CLES} = U / (n_1 \times n_2)\)).
+If the CLES/AUC values with corresponding sampling variances (or standard errors) are known, they can be directly meta-analyzed for example using the rma.uni function. However, in practice, one is likely to encounter studies that only report CLES/AUC values and the group sizes. In this case, one can specify these values via the ai, n1i, and n2i arguments and set measure="CLES" (or equivalently measure="AUC"). If vtype="LS" (the default), the sampling variances are then computed based on Newcombe (2006) (method 4), but using (n1i-1)(n2i-1) in the denominator as suggested by Cho et al. (2019). If vtype="LS2", the sampling variances are computed based on Hanley and McNeil (1982; equations 1 and 2), again using (n1i-1)(n2i-1) in the denominator (and in the unlikely case that the proportion of tied values is known, this can be specified via argument mi, in which case the adjustment as described by Cho et al. (2019) is also applied).
Under the assumption that the data within the two groups are normally distributed (the so-called binormal model), one can also estimate the CLES/AUC values from the means and standard deviations of the two groups. For this, one sets measure="CLESN" (or equivalently measure="AUCN") and specifies the means via arguments m1i and m2i, the standard deviations via arguments sd1i and sd2i, and the group sizes via arguments n1i and n2i. If vtype="LS" (the default), the sampling variances are then computed based on the large-sample approximation derived via the delta method (equation 3 (with a correction, since the plus sign in front of last term in braces should be a multiplication sign) and equation 4 in Goddard & Hinberg, 1990, but using n1i-1 and n2i-1 in the denominators). Computing the CLES/AUC values and corresponding sampling variances does not assume homoscedasticity of the variances in the two groups. However, when vtype="HO", then homoscedasticity is assumed (this will also affect the calculation of the CLES/AUC values themselves). As for measure="SMD", one can also specify standardized mean differences via argument di, t-statistics from an independent samples t-test via argument ti, and/or signed two-sided p-values corresponding to the t-tests via argument pi, which all can be converted into CLES/AUC values (note that this automatically assumes homoscedasticity). One can also directly specify binormal model CLES/AUC values via argument ai (but unless the corresponding sd1i and sd2i values are also specified, the sampling variances are then computed under the assumption of homoscedasticity).
"COR" for the raw correlation coefficient,
"UCOR" for the raw correlation coefficient corrected for its slight negative bias (based on equation 2.3 in Olkin & Pratt, 1958),
"ZCOR" for Fisher's r-to-z transformed correlation coefficient (Fisher, 1921).
If the correlation coefficient is unknown for some studies, but the t-statistics (i.e., \(t_i = r_i \sqrt{n_i - 2} / \sqrt{1 - r_i^2}\)) are available for those studies (for the standard test of \(\text{H}_0{:}\; \rho_i = 0\)), one can specify those values via argument ti, which are then transformed into the corresponding correlation coefficients within the function (the sign of the t-statistics is then taken to be the sign of the correlations). If only the (two-sided) p-values corresponding to the t-tests are known, one can specify those values via argument pi. However, since a two-sided p-value does not carry information about the sign of the test statistic (and hence neither about the correlation), the sign of the p-values (which can be negative) is used as the sign of the correlation coefficients (e.g., escalc(measure="COR", pi=-0.07, ni=30) yields a negative correlation of -0.3354).
If the correlation coefficient is unknown for some studies, but the t-statistics (i.e., \(t_i = r_i \sqrt{n_i - 2} / \sqrt{1 - r_i^2}\)) are available for those studies (for the standard test of \(\text{H}_0{:}\; \rho_i = 0\)), one can specify those values via argument ti, which are then transformed into the corresponding correlation coefficients within the function (the sign of the t-statistics is then taken to be the sign of the correlations). If only the two-sided p-values corresponding to the t-tests are known, one can specify those values via argument pi. However, since a two-sided p-value does not carry information about the sign of the test statistic (and hence neither about the correlation), the sign of the p-values (which can be negative) is used as the sign of the correlation coefficients (e.g., escalc(measure="COR", pi=-0.07, ni=30) yields a negative correlation of -0.3354).
For measure="COR" and measure="UCOR", one can choose between vtype="LS" (the default) for the usual large-sample approximation to compute the sampling variances (i.e., plugging the (biased-corrected) correlation coefficients into equation 12.27 in Borenstein, 2009) and vtype="AV" to compute the sampling variances with the usual large-sample approximation but plugging the sample-size weighted average of the (bias-corrected) correlation coefficients into the equation. For measure="UCOR", one can also choose vtype="UB" to compute unbiased estimates of the sampling variances (see Hedges, 1989, but using the exact equation instead of the approximation).
Datasets corresponding to data of this type are provided in dat.mcdaniel1994 and dat.molloy2014.
For meta-analyses involving multiple (dependent) correlation coefficients extracted from the same sample, see also the rcalc function.
"RPB" for the point-biserial correlation coefficient,
"RBIS" for the biserial correlation coefficient.
There are also measures "ZPB" and "ZBIS" for applying Fisher's r-to-z transformation to these measures. This may be useful when combining these with other types of correlation coefficients that were r-to-z transformed. However, note that the r-to-z transformation is not a variance-stabilizing transformation for these measures.
If the means and standard deviations are unknown for some studies, one can also use arguments di, ti, or pi to specify standardized mean differences (Cohen's d values), t-statistics from an independent samples t-test, or (signed) p-values for the t-test, respectively, as described earlier under (1a) (together with the group sizes, these are sufficient statistics for computing the (point-)biserial correlation coefficients).
If the means and standard deviations are unknown for some studies, one can also use arguments di, ti, or pi to specify standardized mean differences (Cohen's d values), t-statistics from an independent samples t-test, or signed p-values for the t-test, respectively, as described earlier under (1a) (together with the group sizes, these are sufficient statistics for computing the (point-)biserial correlation coefficients).
For measure="RPB", one must indicate via vtype="ST" or vtype="CS" whether the data for the studies were obtained using stratified or cross-sectional (i.e., multinomial) sampling, respectively (it is also possible to specify an entire vector for the vtype argument in case the sampling scheme differed for the various studies).
"MNLN" for the log transformed mean,
"CVLN" for the log transformed coefficient of variation,
"SDLN" for the log transformed standard deviation.
Note that sdi is used to specify the standard deviations of the observed values of the response, characteristic, or dependent variable and not the standard errors of the means. Also, the sampling variance for measure="CVLN" is computed as given by equation 27 in Nakagawa et al. (2015), but without the ‘\(-2 \rho \ldots\)’ term, since for normally distributed data (which we assume here) the mean and variance (and transformations thereof) are independent.
Note that sdi is used to specify the standard deviations of the observed values of the response, characteristic, or dependent variable and not the standard errors of the means. Also, the sampling variances for measure="CVLN" are computed as given by equation 27 in Nakagawa et al. (2015), but without the ‘\(-2 \rho \ldots\)’ term, since for normally distributed data (which we assume here) the mean and variance (and transformations thereof) are independent.
"SMCRPH" for the standardized mean change using raw score standardization with pooled standard deviations and heteroscedastic population variances at the two measurement occasions (Bonett, 2008),
"ROMC" for the log transformed ratio of means (Lajeunesse, 2011).
The raw mean change is simply \(\textrm{m1i}-\textrm{m2i}\), while the standardized mean change is given by \((\textrm{m1i}-\textrm{m2i})/\textrm{sdi}\). For measure="SMCC", \(\textrm{sdi} = \sqrt{\textrm{sd1i}^2 + \textrm{sd2i}^2 - 2\times\textrm{ri}\times\textrm{sd1i}\times\textrm{sd2i}}\) is the standard deviation of the change scores, for measure="SMCR" and measure="SMCRH", \(\textrm{sdi} = \textrm{sd1i}\), and for measure="SMCRP" and measure="SMCRPH", \(\textrm{sdi} = \sqrt{\frac{\textrm{sd1i}^2 + \textrm{sd2i}^2}{2}}\) is the square root of the average variance. See also Morris and DeShon (2002) for a thorough discussion of the difference between the "SMCC" and "SMCR" change score measures. All of these measures are also applicable for matched pairs designs (subscripts 1 and 2 then simply denote the first and second group that are formed by the matching).
In practice, one often has a mix of information available from the individual studies to compute these measures. In particular, if m1i and m2i are unknown, but the raw mean change is directly reported in a particular study, then one can set m1i to that value and m2i to 0 (making sure that the raw mean change was computed as m1i-m2i within that study and not the other way around). Also, for measures "MC" and "SMCC", if sd1i, sd2i, and ri are unknown, but the standard deviation of the change scores is directly reported, then one can set sd1i to that value and both sd2i and ri to 0. For measure "SMCR", argument sd2i is actually not needed, as the standardization is only based on sd1i (Becker, 1988; Morris, 2000), which is usually the pre-test standard deviation (if the post-test standard deviation should be used, then set sd1i to that). Finally, for measure="SMCC", one can also directly specify standardized mean change values via argument di or the t-statistics from a paired samples t-test or the corresponding (two-sided) p-values via argument ti or pi, respectively (which are then transformed into the corresponding standardized mean change values within the function). The sign of the p-values (which can be negative) is used as the sign of the standardized mean change values (e.g., escalc(measure="SMCC", pi=-0.018, ni=50) yields a negative standardized mean change value of -0.3408).
In practice, one often has a mix of information available from the individual studies to compute these measures. In particular, if m1i and m2i are unknown, but the raw mean change is directly reported in a particular study, then one can set m1i to that value and m2i to 0 (making sure that the raw mean change was computed as m1i-m2i within that study and not the other way around). Also, for measures "MC" and "SMCC", if sd1i, sd2i, and ri are unknown, but the standard deviation of the change scores is directly reported, then one can set sd1i to that value and both sd2i and ri to 0. For measure "SMCR", argument sd2i is actually not needed, as the standardization is only based on sd1i (Becker, 1988; Morris, 2000), which is usually the pre-test standard deviation (if the post-test standard deviation should be used, then set sd1i to that). Finally, for measure="SMCC", one can also directly specify standardized mean change values via argument di or the t-statistics from a paired samples t-test or the corresponding two-sided p-values via argument ti or pi, respectively (which are then transformed into the corresponding standardized mean change values within the function). The sign of the p-values (which can be negative) is used as the sign of the standardized mean change values (e.g., escalc(measure="SMCC", pi=-0.018, ni=50) yields a negative standardized mean change value of -0.3408).
Finally, interest may also be focused on differences in the variability of the measurements at the two measurement occasions (or between the two matched groups). Here, the (log transformed) ratio of the coefficient of variation (also called the coefficient of variation ratio) can be a useful measure (Nakagawa et al., 2015). If focus is solely on the variability of the measurements, then the (log transformed) ratio of the standard deviations (also called the variability ratio) can be used (Nakagawa et al., 2015). For the latter, one only needs to specify sd1i, sd2i, ni, and ri. The options for the measure argument are:
"CVRC" for the log transformed coefficient of variation ratio,
"VRC" for the log transformed variability ratio.
"MPORC" for the conditional log odds ratio,
"MPPETO" for the conditional log odds ratio estimated with Peto's method.
See Curtin et al. (2002) and Zou (2007) for discussions of these measures.
-If only marginal tables are available, then another possibility is to compute the marginal log odds ratios based on these table directly. However, for the correct computation of the sampling variances, the correlations (phi coefficients) from the paired tables must be known (or ‘guestimated’). To use this approach, set measure="MPORM" and use argument ri to specify the correlation coefficients. Instead of specifying ri, one can use argument pi to specify the proportions (or ‘guestimates’ thereof) of individuals (or pairs) that experienced the outcome of interest (i.e., ‘outcome1’ in the paired \(2 \times 2\) table) under both treatments (i.e., pi=ai/(ai+bi+ci+di)). Based on these proportions, the correlation coefficients are then back-calculated and used to calculate the correct sampling variances. Note that the values in the marginal tables put constraints on the possible values for ri and pi. If a specified value for ri or pi is not feasible under a given table, the corresponding sampling variance will be NA.
If only marginal tables are available, then another possibility is to compute the marginal log odds ratios based on these table directly. However, for the correct computation of the sampling variances, the correlations (phi coefficients) from the paired tables must be known (or ‘guestimated’). To use this approach, set measure="MPORM" and use argument ri to specify the correlation coefficients. Instead of specifying ri, one can use argument pi to specify the proportions (or ‘guestimates’ thereof) of individuals (or pairs) that experienced the outcome of interest (i.e., ‘outcome1’ in the paired \(2 \times 2\) table) under both treatments (i.e., pi=ai/(ai+bi+ci+di)). Based on these proportions, the correlation coefficients are then back-calculated and used to compute the correct sampling variances. Note that the values in the marginal tables put constraints on the possible values for ri and pi. If a specified value for ri or pi is not feasible under a given table, the corresponding sampling variance will be NA.
"SPCOR" for the semi-partial correlation coefficient,
"ZSPCOR" for Fisher's r-to-z transformed semi-partial correlation coefficient.
Note that the signs of the (semi-)partial correlation coefficients is determined based on the signs of the values specified via the ti argument. Also, while the Fisher transformation can be applied to both measures, it is only a variance-stabilizing transformation for partial correlation coefficients.
If the test statistic (i.e., t-test) of the regression coefficient of interest is unknown for some studies, but the (two-sided) p-values corresponding to the t-tests are known, one can specify those values via argument pi. However, since a two-sided p-value does not carry information about the sign of the test statistic (and hence neither about the correlation), the sign of the p-values (which can be negative) is used as the sign of the correlation coefficients (e.g., escalc(measure="PCOR", pi=-0.07, mi=5, ni=30) yields a negative partial correlation of -0.3610).
If the test statistic (i.e., t-test) of the regression coefficient of interest is unknown for some studies, but the two-sided p-values corresponding to the t-tests are known, one can specify those values via argument pi. However, since a two-sided p-value does not carry information about the sign of the test statistic (and hence neither about the correlation), the sign of the p-values (which can be negative) is used as the sign of the correlation coefficients (e.g., escalc(measure="PCOR", pi=-0.07, mi=5, ni=30) yields a negative partial correlation of -0.3610).
In the rare case that the (semi-)partial correlations are known for some of the studies, then these can be directly specified via the ri argument. This can be useful, for example, when \(\eta^2_p\) (i.e., partial eta squared) is known for the regression coefficient of interest, since the square root thereof is identical to the absolute value of the partial correlation (although the correct sign then still needs to be reconstructed based on other information).
A dataset corresponding to data of this type is provided in dat.aloe2013.
Bonett, D. G. (2010). Varying coefficient meta-analytic methods for alpha reliability. Psychological Methods, 15(4), 368–385. https://doi.org/10.1037/a0020142
Borenstein, M. (2009). Effect sizes for continuous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 221–235). New York: Russell Sage Foundation.
Chinn, S. (2000). A simple method for converting an odds ratio to effect size for use in meta-analysis. Statistics in Medicine, 19(22), 3127–3131. https://doi.org/10.1002/1097-0258(20001130)19:22<3127::aid-sim784>3.0.co;2-m
+Cho, H., Matthews, G. J., & Harel, O. (2019). Confidence intervals for the area under the receiver operating characteristic curve in the presence of ignorable missing data. International Statistical Review, 87(1), 152–177. https://doi.org/10.1111/insr.12277
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
Cousineau, D. (2020). Approximating the distribution of Cohen's d_p in within-subject designs. The Quantitative Methods for Psychology, 16(4), 418–421. https://doi.org/10.20982/tqmp.16.4.p418
Cox, D. R., & Snell, E. J. (1989). Analysis of binary data (2nd ed.). London: Chapman & Hall.
@@ -613,7 +637,9 @@Fleiss, J. L., & Berlin, J. (2009). Effect sizes for dichotomous data. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 237–253). New York: Russell Sage Foundation.
Freeman, M. F., & Tukey, J. W. (1950). Transformations related to the angular and the square root. Annals of Mathematical Statistics, 21(4), 607–611. https://doi.org/10.1214/aoms/1177729756
Gibbons, R. D., Hedeker, D. R., & Davis, J. M. (1993). Estimation of effect size from a series of experiments involving paired comparisons. Journal of Educational Statistics, 18(3), 271–279. https://doi.org/10.3102/10769986018003271
+Goddard, M. J., & Hinberg, I. (1990). Receiver operator characteristic (ROC) curves and non‐normal data: An empirical study. Statistics in Medicine, 9(3), 325–337. https://doi.org/10.1002/sim.4780090315
Hakstian, A. R., & Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41(2), 219–231. https://doi.org/10.1007/BF02291840
+Hanley, J. A., & McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. https://doi.org/10.1148/radiology.143.1.7063747
Hasselblad, V., & Hedges, L. V. (1995). Meta-analysis of screening and diagnostic tests. Psychological Bulletin, 117(1), 167–178. https://doi.org/10.1037/0033-2909.117.1.167
Hedges, L. V. (1981). Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics, 6(2), 107–128. https://doi.org/10.3102/10769986006002107
Hedges, L. V. (1982). Estimation of effect size from a series of independent experiments. Psychological Bulletin, 92(2), 490–499. https://doi.org/10.1037/0033-2909.92.2.490
@@ -626,11 +652,13 @@Kirk, D. B. (1973). On the numerical approximation of the bivariate normal (tetrachoric) correlation coefficient. Psychometrika, 38(2), 259–268. https://doi.org/10.1007/BF02291118
Lajeunesse, M. J. (2011). On the meta-analysis of response ratios for studies with correlated and multi-group designs. Ecology, 92(11), 2049–2055. https://doi.org/10.1890/11-0423.1
May, W. L., & Johnson, W. D. (1997). Confidence intervals for differences in correlated binary proportions. Statistics in Medicine, 16(18), 2127–2136. https://doi.org/10.1002/(SICI)1097-0258(19970930)16:18<2127::AID-SIM633>3.0.CO;2-W
+McGraw, K. O., & Wong, S. P. (1992). A common language effect size statistic. Psychological Bulletin, 111(2), 361–365. https://doi.org/10.1037/0033-2909.111.2.361
Morris, S. B. (2000). Distribution of the standardized mean change effect size for meta-analysis on repeated measures. British Journal of Mathematical and Statistical Psychology, 53(1), 17–29. https://doi.org/10.1348/000711000159150
Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7(1), 105–125. https://doi.org/10.1037/1082-989x.7.1.105
Nakagawa, S., Poulin, R., Mengersen, K., Reinhold, K., Engqvist, L., Lagisz, M., & Senior, A. M. (2015). Meta-analysis of variation: Ecological and evolutionary applications and beyond. Methods in Ecology and Evolution, 6(2), 143–152. https://doi.org/10.1111/2041-210x.12309
Nakagawa, S., Noble, D. W. A., Lagisz, M., Spake, R., Viechtbauer, W., & Senior, A. M. (2023). A robust and readily implementable method for the meta-analysis of response ratios with and without missing standard deviations. Ecology Letters, 26(2), 232–244. https://doi.org/10.1111/ele.14144
Newcombe, R. G. (1998). Improved confidence intervals for the difference between binomial proportions based on paired data. Statistics in Medicine, 17(22), 2635–2650. https://doi.org/10.1002/(SICI)1097-0258(19981130)17:22<2635::AID-SIM954>3.0.CO;2-C
+Newcombe, R. G. (2006). Confidence intervals for an effect size measure based on the Mann-Whitney statistic. Part 2: Asymptotic methods and evaluation. Statistics in Medicine, 25(4), 559–573. https://doi.org/10.1002/sim.2324
Olkin, I., & Finn, J. D. (1995). Correlations redux. Psychological Bulletin, 118(1), 155–164. https://doi.org/10.1037/0033-2909.118.1.155
Olkin, I., & Pratt, J. W. (1958). Unbiased estimation of certain correlation coefficients. Annals of Mathematical Statistics, 29(1), 201–211. https://doi.org/10.1214/aoms/1177706717
Pearson, K. (1900). Mathematical contributions to the theory of evolution. VII. On the correlation of characters not quantitatively measurable. Philosophical Transactions of the Royal Society of London, Series A, 195, 1–47. https://doi.org/10.1098/rsta.1900.0022
diff --git a/docs/reference/fitstats.html b/docs/reference/fitstats.html index 77b9a34a..c84996a9 100644 --- a/docs/reference/fitstats.html +++ b/docs/reference/fitstats.html @@ -29,7 +29,7 @@ diff --git a/docs/reference/fitted.rma.html b/docs/reference/fitted.rma.html index 8d31cf0e..c844d57d 100644 --- a/docs/reference/fitted.rma.html +++ b/docs/reference/fitted.rma.html @@ -29,7 +29,7 @@ diff --git a/docs/reference/forest.cumul.rma.html b/docs/reference/forest.cumul.rma.html index ce5cb7f1..acc617cd 100644 --- a/docs/reference/forest.cumul.rma.html +++ b/docs/reference/forest.cumul.rma.html @@ -29,7 +29,7 @@ diff --git a/docs/reference/forest.default.html b/docs/reference/forest.default.html index e54abde4..e24176a2 100644 --- a/docs/reference/forest.default.html +++ b/docs/reference/forest.default.html @@ -29,7 +29,7 @@ diff --git a/docs/reference/forest.html b/docs/reference/forest.html index 96e27a01..4dce5a27 100644 --- a/docs/reference/forest.html +++ b/docs/reference/forest.html @@ -29,7 +29,7 @@ diff --git a/docs/reference/forest.rma.html b/docs/reference/forest.rma.html index 9a88c078..1e9dc349 100644 --- a/docs/reference/forest.rma.html +++ b/docs/reference/forest.rma.html @@ -29,7 +29,7 @@ diff --git a/docs/reference/formatters.html b/docs/reference/formatters.html index c51dc33d..deb4ee9e 100644 --- a/docs/reference/formatters.html +++ b/docs/reference/formatters.html @@ -29,7 +29,7 @@ diff --git a/docs/reference/formula.rma.html b/docs/reference/formula.rma.html index 26be18fe..56da69ae 100644 --- a/docs/reference/formula.rma.html +++ b/docs/reference/formula.rma.html @@ -29,7 +29,7 @@ @@ -142,13 +142,13 @@transf.ztor() transf.logit() transf.ilogit() transf.arcsin() transf.iarcsin() transf.pft() transf.ipft() transf.ipft.hm() transf.isqrt() transf.irft() transf.iirft() transf.ahw() transf.iahw() transf.abt() transf.iabt() transf.r2toz() transf.ztor2() transf.ztor.int() transf.exp.int() transf.ilogit.int() transf.dtou1() transf.dtou2() transf.dtou3() transf.dtoovl() transf.dtocles() transf.dtobesd() transf.dtomd() transf.dtorpb() transf.dtorbis() transf.rpbtorbis() transf.rtorpb() transf.rtod() transf.rpbtod() transf.lnortord() transf.lnortorr() transf.lnortod.norm() transf.lnortod.logis() transf.dtolnor.norm() transf.dtolnor.logis() transf.lnortortet.pearson() transf.lnortortet.digby()
+ transf.rtoz() transf.ztor() transf.logit() transf.ilogit() transf.arcsin() transf.iarcsin() transf.pft() transf.ipft() transf.ipft.hm() transf.isqrt() transf.irft() transf.iirft() transf.ahw() transf.iahw() transf.abt() transf.iabt() transf.r2toz() transf.ztor2() transf.ztor.int() transf.exp.int() transf.ilogit.int() transf.dtou1() transf.dtou2() transf.dtou3() transf.dtoovl() transf.dtocles() transf.dtocliffd() transf.dtobesd() transf.dtomd() transf.dtorpb() transf.dtorbis() transf.rpbtorbis() transf.rtorpb() transf.rtod() transf.rpbtod() transf.lnortord() transf.lnortorr() transf.lnortod.norm() transf.lnortod.logis() transf.dtolnor.norm() transf.dtolnor.logis() transf.lnortortet.pearson() transf.lnortortet.digby()
Transformation Functions