Papers by Etienne P LeBel
In recent years, there has been a growing concern regarding the replicability of findings in psyc... more In recent years, there has been a growing concern regarding the replicability of findings in psychology, including a mounting number of prominent findings that have failed to replicate via high-powered independent replication attempts. In the face of this replicability “crisis of confidence”, several initiatives have been implemented to increase the reliability of empirical findings. In the current article, I propose a new replication norm that aims to further boost the dependability of findings in psychology. Paralleling the extant social norm that researchers should peer review about three times as many articles that they themselves publish per year, the new replication norm states that researchers should aim to independently replicate important findings in their own research areas in proportion to the number of original studies they themselves publish per year (e.g., a 4:1 original-to-replication studies ratio). I argue this simple approach could significantly advance our science by increasing the reliability and cumulative nature of our empirical knowledge base, accelerating our theoretical understanding of psychological phenomena, instilling a focus on quality rather than quantity, and by facilitating our transformation toward a research culture where executing and reporting independent direct replications is viewed as an ordinary part of the research process. To help promote the new norm, I delineate (1) how each of the major constituencies of the research process (i.e., funders, journals, professional societies, departments, and individual researchers) can incentivize replications and promote the new norm and (2) any obstacles each constituency faces in supporting the new norm.
Reproducibility is a defining feature of science, but the extent to which it characterizes curren... more Reproducibility is a defining feature of science, but the extent to which it characterizes current research is unknown. We conducted replications of 100 experimental and correlational studies published in three psychology journals using high-powered designs and original
materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
Several researchers recently outlined unacknowledged costs of open science practices, arguing the... more Several researchers recently outlined unacknowledged costs of open science practices, arguing these costs may outweigh benefits and stifle discovery of novel findings. We scrutinize these researchers' (1) statistical concern that heightened stringency with respect to false-positives will increase false-negatives and (2) meta-scientific concern that larger samples and executing direct replications engender opportunity costs that will decrease the rate of making novel discoveries. We argue their statistical concern is unwarranted given open science proponents recommend such practices to reduce the inflated Type I error rate from .35 down to .05 and simultaneously call for high-powered research to reduce the inflated Type II error rate. Regarding their meta-concern, we demonstrate that incurring some costs is required to increase the rate (and frequency) of making true discoveries because distinguishing true from false hypotheses requires a low Type I error rate, high statistical power, and independent direct replications. We also examine pragmatic concerns raised regarding adopting open science practices for relationship science (pre-registration, open materials, open data, direct replications, sample size); while acknowledging these concerns, we argue they are overstated given available solutions. We conclude benefits of open science practices outweigh costs for both individual researchers and the collective field in the long run, but that short term costs may exist for researchers because of the currently dysfunctional academic incentive structure. Our analysis implies our field's incentive structure needs to change whereby better alignment exists between researcher's career interests and the field's cumulative progress. We delineate recent proposals aimed at such incentive structure re-alignment.
Finkel, Eastwick, and Reis (2016; FER2016) argued the post-2011 methodological reform movement ha... more Finkel, Eastwick, and Reis (2016; FER2016) argued the post-2011 methodological reform movement has focused narrowly on replicability, neglecting other essential goals of research. We agree multiple scientific goals are essential, but argue, however, a more fine-grained language, conceptualization, and approach to replication is needed to accomplish these goals. Replication is the general empirical mechanism for testing and falsifying theory. Sufficiently methodologically similar replications, also known as direct replications, test the basic existence of phenomena and ensure cumulative progress is possible a priori. In contrast, increasingly methodologically dissimilar replications, also known as conceptual replications, test the relevance of auxiliary hypotheses (e.g., manipulation and measurement issues, contextual factors) required to productively investigate validity and generalizability. Without prioritizing replicability, a field is not empirically falsifiable. We also disagree with FER2016’s position that “bigger samples are generally better, but … that very large samples could have the downside of commandeering resources that would have been better invested in other studies” (abstract). We identify problematic assumptions involved in FER2016’s modifications of our original research-economic model, and present an improved model that quantifies when (and whether) it is reasonable to worry that increasing statistical power will engender potential trade-offs. Sufficiently-powering studies (i.e., >80%) maximizes both research efficiency and confidence in the literature (research quality). Given we are in agreement with FER2016 on all key open science points, we are eager to start seeing the accelerated rate of cumulative knowledge development of social psychological phenomena such a sufficiently transparent, powered, and falsifiable approach will generate.
Perspectives on Psychological Science, 2012
SSRN Electronic Journal, 2000
The purpose of this paper is to extend to the field of relationship science, recent discussions a... more The purpose of this paper is to extend to the field of relationship science, recent discussions and suggested changes in open research practises. We demonstrate different ways that greater transparency of the research process in our field will accelerate scientific progress by increasing accuracy of reported research findings. Importantly, we make concrete recommendations for how relationship researchers can transition to greater disclosure of research practices in a manner that is sensitive to the unique design features of methodologies employed by relationship scientists. We discuss how to implement these recommendations for four different research designs regularly used in relationship research and practical limitations regarding implementing our recommendations and provide potential solutions to these problems.
Review of General Psychology, 2011
In this methodological commentary, we use Bem's (2011) recent article reporting experimental evid... more In this methodological commentary, we use Bem's (2011) recent article reporting experimental evidence for psi as a case study for discussing important deficiencies in modal research practice in empirical psychology. We focus on (a) overemphasis on conceptual rather than close replication, (b) insufficient attention to verifying the soundness of measurement and experimental procedures, and (c) flawed implementation of null hypothesis significance testing. We argue that these deficiencies contribute to weak method-relevant beliefs that, in conjunction with overly strong theory-relevant beliefs, lead to a systemic and pernicious bias in the interpretation of data that favors a researcher's theory. Ultimately, this interpretation bias increases the risk of drawing incorrect conclusions about human psychology. Our analysis points to concrete recommendations for improving research practice in empirical psychology. We recommend (a) a stronger emphasis on close replication, (b) routinely verifying the integrity of measurement instruments and experimental procedures, and (c) using stronger, more diagnostic forms of null hypothesis testing.
Perspectives on Psychological Science, 2013
There is currently an unprecedented level of doubt regarding the reliability of research findings... more There is currently an unprecedented level of doubt regarding the reliability of research findings in psychology. Many recommendations have been made to improve the current situation. In this article, we report results from PsychDisclosure.org, a novel open-science initiative that provides a platform for authors of recently published articles to disclose four methodological design specification details that are not required to be disclosed under current reporting standards but that are critical for accurate interpretation and evaluation of reported findings. Grassroots sentiment-as manifested in the positive and appreciative response to our initiative-indicates that psychologists want to see changes made at the systemic level regarding disclosure of such methodological details. Almost 50% of contacted researchers disclosed the requested design specifications for the four methodological categories (excluded subjects, nonreported conditions and measures, and sample size determination). Disclosed information provided by participating authors also revealed several instances of questionable editorial practices, which need to be thoroughly examined and redressed. On the basis of these results, we argue that the time is now for mandatory methods disclosure statements for all psychology journals, which would be an important step forward in improving the reliability of findings in psychology.
Psychonomic Bulletin & Review, 2014
Slepian, Masicampo, Toosi, and Ambady (Journal of Experimental Psychology: General, 141, 619-624,... more Slepian, Masicampo, Toosi, and Ambady (Journal of Experimental Psychology: General, 141, 619-624, 2012, Study 1) found that individuals recalling and writing about a big, meaningful secret judged a pictured hill as steeper than did those who recalled and wrote about a small, inconsequential secret (with estimates unrelated to physical effort unaffected). From an embodied cognition perspective, this result was interpreted as suggesting that important secrets weigh people down. Answering to mounting calls for the crucial need of independent direct replications of published findings to ensure the self-correcting nature of our science, we sought to corroborate Slepian et al.'s finding in two extremely high-powered, preregistered studies that were very faithful to all procedural and methodological details of the original study (i.e., same cover story, study title, manipulation, measures, item order, scale anchors, task instructions, sampling frame, population, and statistical analyses). In both samples, we were unsuccessful in replicating the target finding. Although Slepian et al. reported three other studies supporting the secret burdensomeness phenomenon, we advise that these three other findings need to be independently corroborated before the general phenomenon informs theory or health interventions.
Psychological Science, 2013
In his recent article, Vess (2012) was the first to integrate research on attachment anxiety with... more In his recent article, Vess (2012) was the first to integrate research on attachment anxiety with research linking physical temperature to perceptions of intimacy (i.e., intimacy is associated with physical warmth, whereas social isolation is associated with coldness; . In his first study, Vess found that individuals with high (but not low) levels of anxious attachment reported heightened preferences for warm foods when attachment concerns were activated (i.e., by reflecting on a romantic breakup). These findings suggest novel approaches for investigating how anxious individuals regulate feelings of intimacy, as well as for designing temperature-related interventions directed toward anxious individuals when distressed. Further, these findings show that attachment concerns may be reliably activated with fairly simple manipulations using online samples.
Perspectives on psychological science : a journal of the Association for Psychological Science, 2016
Finkel, Rusbult, Kumashiro, and Hannon (2002, Study 1) demonstrated a causal link between subject... more Finkel, Rusbult, Kumashiro, and Hannon (2002, Study 1) demonstrated a causal link between subjective commitment to a relationship and how people responded to hypothetical betrayals of that relationship. Participants primed to think about their commitment to their partner (high commitment) reacted to the betrayals with reduced exit and neglect responses relative to those primed to think about their independence from their partner (low commitment). The priming manipulation did not affect constructive voice and loyalty responses. Although other studies have demonstrated a correlation between subjective commitment and responses to betrayal, this study provides the only experimental evidence that inducing changes to subjective commitment can causally affect forgiveness responses. This Registered Replication Report (RRR) meta-analytically combines the results of 16 new direct replications of the original study, all of which followed a standardized, vetted, and preregistered protocol. The ...
Psychonomic bulletin & review, Jan 11, 2014
Correll (Journal of Personality and Social Psychology, 94, 48-59, 2008; Study 2) found that instr... more Correll (Journal of Personality and Social Psychology, 94, 48-59, 2008; Study 2) found that instructions to use or avoid race information decreased the emission of 1/f noise in a weapon identification task (WIT). These results suggested that 1/f noise in racial bias tasks reflected an effortful deliberative process, providing new insights regarding the mechanisms underlying implicit racial biases. Given the potential theoretical and applied importance of understanding the psychological processes underlying implicit racial biases - and in light of the growing demand for independent direct replications of findings to ensure the cumulative nature of our science - we attempted to replicate Correll's finding in two high-powered studies. Despite considerable effort to closely duplicate all procedural and methodological details of the original study (i.e., same cover story, experimental manipulation, implicit measure task, original stimuli, task instructions, sampling frame, population...
Payne, Hall, Cameron, and Bishara (2010, Experiment 1) advanced a multinomial processing tree (MP... more Payne, Hall, Cameron, and Bishara (2010, Experiment 1) advanced a multinomial processing tree (MPT) model formally specifying an affect misattribution process underlying responses in the affect misattribution procedure (AMP). The model was empirically supported in two studies, suggesting such an approach may provide deeper insights regarding the psychological mechanisms underlying the AMP and increased accuracy in estimating individuals' implicit attitudes. Given the potential theoretical importance and applied implications of the approach, we attempted to independently replicate Payne et al.'s Experiment 1 MPT model results in two high-powered studies. Despite considerable effort to closely duplicate all methodological details of the original study, overall we were unsuccessful in replicating the original findings. Our results suggest that the psychological processes underlying the AMP are likely more nuanced whereby different processes may be operating in different individuals. We suggest the use of more advanced modeling approaches -at the level of the individual -to understand the likely heterogeneity that exists in the processes underlying responses in the AMP. We urge researchers to exercise caution in using the AMP for theory-testing or applied purposes until further research clarifies such heterogeneity. Word count: 184 Replication Difficulties of MPT Model of AMP 3 Replication difficulties of Payne, Hall, Cameron, and Bishara's (2010) multinomial processing tree model of the affect misattribution procedure In recent years, psychologists have increasingly relied upon implicit measures of
Inspired by the history of the development of instruments in the physical sciences, and by past p... more Inspired by the history of the development of instruments in the physical sciences, and by past psychology giants, the following dissertation aimed to advance basic psychological science by investigating the metric calibration of psychological instruments. The overarching goal of the dissertation was to demonstrate that it is both useful and feasible to calibrate the metric of psychological instruments so as to render their metrics non-arbitrary.
Psychological Bulletin, May 1, 2009
2009) normative analysis of implicit measures provides an excellent clarification of several conc... more 2009) normative analysis of implicit measures provides an excellent clarification of several conceptual ambiguities surrounding the validation and use of implicit measures. The current comment discusses an important, yet unacknowledged, implication of J. De Houwer et al.'s analysis, namely, that investigations addressing the proposed implicitness criterion (i.e., does the relevant psychological attribute influence measurement outcomes in an automatic fashion?) will be susceptible to fundamental misinterpretations if they are conducted independently of the proposed what criterion (i.e., is the measurement outcome causally produced by the psychological attribute the measurement procedure was designed to assess?). As a solution, it is proposed that experimental validation studies should be combined with a correlational approach in order to determine whether a given manipulation influenced measurement scores via variations in the relevant psychological attribute or via secondary sources of systematic variance.
Social and Personality Psychology Compass, 2008
Over the last decade, a new class of indirect measurement procedures has become increasingly popu... more Over the last decade, a new class of indirect measurement procedures has become increasingly popular in many areas of psychology. However, these implicit measures have also sparked controversies about the nature of the constructs they assess. One controversy has been stimulated by the question of whether some implicit measures (or implicit measures in general) assess extra-personal rather than personal associations. We argue that, despite empirical and methodological advances stimulated by this debate, researchers have not sufficiently addressed the conceptual question of how to define extra-personal in contrast to personal associations. Based on a review of possible definitions, we argue that some definitions render the controversy obsolete, whereas others imply fundamentally different empirical and methodological questions. As an alternative to defining personal and extra-personal associations in an objective sense, we suggest an empirical approach that investigates the meta-cognitive inferences that make a given association subjectively personal or extra-personal for the individual.
Self and Identity, 2010
... Wiesner, & Schu¨tz, 2007; Shimizu & Pelham, 2004), anxiety (Spalding &amp... more ... Wiesner, & Schu¨tz, 2007; Shimizu & Pelham, 2004), anxiety (Spalding & Hardin, 1999), narcissism (Jordan, Spencer, Zanna, Hoshino-Browne, & Correll, 2003), social acceptance (Baccus, Baldwin, & Packer, 2004), unrealistic optimism (Bosson, Brown, Zeigler-Hill, & Swann ...
Perspectives on Psychological Science, 2007
Experimental paradigms designed to assess ''implicit'' representations are currently very popular... more Experimental paradigms designed to assess ''implicit'' representations are currently very popular in many areas of psychology. The present article addresses the validity of three widespread assumptions in research using these paradigms: that (a) implicit measures reflect unconscious or introspectively inaccessible representations; (b) the major difference between implicit measures and self-reports is that implicit measures are resistant or less susceptible to social desirability; and (c) implicit measures reflect highly stable, older representations that have their roots in long-term socialization experiences. Drawing on a review of the available evidence, we conclude that the validity of all three assumptions is equivocal and that theoretical interpretations should be adjusted accordingly. We discuss an alternative conceptualization that distinguishes between activation and validation processes.
Uploads
Papers by Etienne P LeBel
materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
materials when available. Replication effects were half the magnitude of original effects, representing a substantial decline. Ninety-seven percent of original studies had statistically significant results. Thirty-six percent of replications had statistically significant results; 47% of original effect sizes were in the 95% confidence interval of the replication effect size; 39% of effects were subjectively rated to have replicated the original result; and if no bias in original results is assumed, combining original and replication results left 68% with statistically significant effects. Correlational tests suggest that replication success was better predicted by the strength of original evidence than by characteristics of the original and replication teams.
improved understanding to address pressing medical and social problems. Scientific findings, however, can only justifiably inform theory or applied problems if they are at minimum internally and externally provisionally trustworthy. Internal trustworthiness is gauged by quantifying the analytic reproducibility and robustness of a study's results; external trustworthiness is gauged by quantifying the replicability and generalizability of published effects and phenomena. The following paper outlines a unified curation framework to quantify the reproducibility, robustness, replicability, and generalizability of scientific findings, categorically addressing all forms of researcher and publication bias. Five major challenges are addressed by the proposed framework: (1) a standardized workflow and principled metric to quantify the analytic reproducibility and robustness of reported results from primary, auxiliary, and secondary analyses; (2) a flexible workflow and replication taxonomy to categorize (i) sufficiently methodologically similar replications that can speak to replicability and (ii) eligible generalizations of an original effect that can speak to generalizability; (3) a principled meta- analytic approach to synthesizing replicability and generalizability evidence; (4) accounting for variations in study characteristics of replications and generalizations; and (5) a viable crowd- sourced web platform to allow the community of scientists to quantify the provisional trustworthiness of published findings in an incremental and ongoing basis. Ultimately, the framework will accelerate investigations into the validity of such trustworthy findings, and consequently accelerate our understanding of the world and development of applied solutions to societal problems.