Abstract
Whether a published finding is robust is difficult to assess. Researchers often point at replication as a robustness check. However, conducting a replication on a new sample can cost a lot of time, effort, and money. In this talk, I propose a consecutive “four-step robustness check” that aims at the low-hanging fruit first. First, we check the internal consistency of statistical results (possibly using automated tools, such as “statcheck”). Second, we reanalyze the data using the original analytical strategy to see if the reported conclusions hold. Third, we check if the original result is robust to alternative analytical choices, for instance via a multi-verse analysis. Only then, in the fourth step, we perform a replication study on a new sample. This four-step approach allows detecting unreliable results, while wasting as little resources as possible. I will discuss potential advantages and limitations of this approach and focus on some practical steps researchers themselves can take in order to increase robustness of their own work in line with the proposed 4-step approach.
Speaker Bio
Michèle Nuijten is an Assistant Professor at the Meta-Research Center at Tilburg University, where she studies reproducibility and replicability in psychology. She obtained her PhD in Methodology and Statistics at Tilburg University in 2018. Her PhD thesis, titled “Research on Research: A Meta-Scientific Study of Problems and Solutions in Psychological Science”, was awarded the Tilburg University Dissertation Prize.
As part of her research, Michèle co-developed the free tool statcheck; a “spellchecker” for statistics. Statcheck has gained popularity as a pre-publication check: since its launch in 2016, the web app was visited tens of thousands times, and the journals Psychological Science and the Journal of Experimental Social Psychology have made statcheck a standard element in their peer review process.
Beside her research, Michèle is closely involved with the Society for the Improvement of Psychological Science (SIPS), having been a member of the executive committee, and past-chair of the program committee. She is also part of the Program Committee Replication Research of The Netherlands Organization for Scientific Research, advising them on distributing funding for replication research.