As time goes by…  [presentation, pdf, 584 kB]

Wiedermann W., Gula B.
First author's affiliation: Department of Psychology, Klagenfurt, Austria

Observations obtained from reaction time (RT) experiments often violate the normality assumption underlying parametric significance tests. The present study compares several strategies that deal with non-normally distributed RT observations. In 2000 the Journal of Experimental Psychology: Human Perception and Performance published 385 experiments. In about 60% of these studies RTs were analyzed. The most common procedure was trimming (47%), ranging from 2 to 4 SDs. In 41% of the RT studies neither outlier analyses nor any distributional considerations were mentioned. This state is quite alarming because in case of non-normal data such analyses mostly fail to detect true differences. Other approaches such as transformation or nonparametric tests were rather uncommon. Based on our review, common criteria for trimming were chosen to compare the efficiency of this method with less common procedures. A Monte Carlo approach was used to simulate plausible RT observations based on twelve ex-Gaussian distributions (Miller, 1988). In order to investigate the two-sample problem, the t-test on raw RT scores was compared to (a) constant and adaptive trimming, (b) logarithmic and adaptive transformation, and (c) the non-parametric Wilcoxon-Mann-Whitney-test (WMW). Results reveal that all procedures are robust for the twelve distributions. However, the t-test on raw scores entails a great power loss, especially if distributions are extremely skewed (skewness: 2.09, kurtosis: 9.11). For these distributions the log-transformation, as well as the WMW-test were most powerful and an increasing amount of trimming enhances the power of the t-test for trimmed means. For less skewed distributions (skewness: 0.71, kurtosis: 4.54) choosing a transformation adaptively is slightly more powerful than the log-transformation and an increasing amount of trimming reduces the power of detecting true differences in trimmed means. Based on the characteristics of sample distributions, a detailed decision tree will be suggested to aid the choice of the most accurate procedure.