Unusually high reliability - AAT Study

[degraces]

Hi there,

I'm posting in hopes of obtaining some guidance as my split-half reliability analysis is outputting unusually high numbers for an AAT (.77 - .81), especially compared to the findings of the Pronk et al study. For context, my design is a 2x2 within-subjects experiment. participants were exposed to a trauma cue and a neutral cue (Cue condition: IV #1), and responded to cannabis and neutral stimuli (stimuli condition: IV #2) on an AAT following each cue exposure. Participants' reaction times to the stimuli on the AAT (transformed into AAT scores) is our DV.

I'm wondering if anyone in the community notices any errors in my code that would inflate the split-half reliability estimates I'm getting.

Many thanks!

#make approach bias scores an object
approach <- function (data)
return (sum(data$APBIAS))

#stratify total dataset by stimulus type and cue condition
split_scores4 <- by_split(
data,
data$record_id,
approach,
replications = 1000,
replace = TRUE,
ncores = 1,
careful = FALSE,
stratification = data$STIM, data$CON,
)

#spearman brown coefficient
coefs <- split_coefs(split_scores4, spearman_brown)
hist(coefs)
mean(coefs)

[tpronk]

Hi @degraces,

Thanks for your question! I don't see any obvious issues with the code you shared, but I do have a recommendation: In Pronk et al. (2021) we saw a lot of variation in AAT reliability coefficients as a function of splitting method and in Pronk et al. (2022) we used a sub-sampling approach to demonstrate that the Monte Carlo splitting method could overestimate reliability relative to a permutated split.

Hence, I'd recommend trying a variety of splitting methods to see how much the resulting reliability coefficients differ. Also, we've got another thread in this repo where another researcher did a deep dive into AAT + Monte Carlo; maybe ask them too?

Best, Thomas

[degraces]

Thank you, Thomas! This was really helpful.

[tpronk]

You're welcome! I'll mark this question as answered, but feel free to continue the conversation if anything else pops up