Towards responsible research assessment: How to reward research quality.

Researchers would be more willing to prioritize research quality over quantity if the incentive structure of the academic system aligned with this goal. The winner of a 2023 Einstein Foundation Award for Promoting Quality in Research explains how they rose to this challenge.

Machine Learning and Risk Assessment: Random Forest Does Not Outperform Logistic Regression in the Prediction of Sexual Recidivism.

Although many studies supported the use of actuarial risk assessment instruments (ARAIs) because they outperformed unstructured judgments, it remains an ongoing challenge to seek potentials for improvement of their predictive performance. Machine learning (ML) algorithms, like random forests, are able to detect patterns in data useful for prediction purposes without explicitly programming them (e.g., by considering nonlinear effects between risk factors and the criterion). Therefore, the current study aims to compare conventional logistic regression analyses with the random forest algorithm on a sample of N = 511 adult male individuals convicted of sexual offenses. Data were collected at the Federal Evaluation Center for Violent and Sexual Offenders in Austria within a prospective-longitudinal research design and participants were followed-up for an average of M = 8.2 years. The Static-99, containing static risk factors, and the Stable-2007, containing stable dynamic risk factors, were included as predictors. The results demonstrated no superior predictive performance of the random forest compared with logistic regression; furthermore, methods of interpretable ML did not point to any robust nonlinear effects. Altogether, results supported the statistical use of logistic regression for the development and clinical application of ARAIs.

Replacing academic journals.

Replacing traditional journals with a more modern solution is not a new idea. Here, we propose ways to overcome the social dilemma underlying the decades of inaction. Any solution needs to not only resolve the current problems but also be capable of preventing takeover by corporations: it needs to replace traditional journals with a decentralized, resilient, evolvable network that is interconnected by open standards and open-source norms under the governance of the scholarly community. It needs to replace the monopolies connected to journals with a genuine, functioning and well-regulated market. In this new market, substitutable service providers compete and innovate according to the conditions of the scholarly community, avoiding sustained vendor lock-in. Therefore, a standards body needs to form under the governance of the scholarly community to allow the development of open scholarly infrastructures servicing the entire research workflow. We propose a redirection of money from legacy publishers to the new network by funding bodies broadening their minimal infrastructure requirements at recipient institutions to include modern infrastructure components replacing and complementing journal functionalities. Such updated eligibility criteria by funding agencies would help realign the financial incentives for recipient institutions with public and scholarly interest.

Replication of the natural selection of bad science.

This study reports an independent replication of the findings presented by Smaldino and McElreath (Smaldino, McElreath 2016 R. Soc. Open Sci. 3, 160384 (doi:10.1098/rsos.160384)). The replication was successful with one exception. We find that selection acting on scientist's propensity for replication frequency caused a brief period of exuberant replication not observed in the original paper due to a coding error. This difference does not, however, change the authors' original conclusions. We call for more replication studies for simulations as unique contributions to scientific quality assurance.

Big little lies: a compendium and simulation of p-hacking strategies.

In many research fields, the widespread use of questionable research practices has jeopardized the credibility of scientific results. One of the most prominent questionable research practices is p-hacking. Typically, p-hacking is defined as a compound of strategies targeted at rendering non-significant hypothesis testing results significant. However, a comprehensive overview of these p-hacking strategies is missing, and current meta-scientific research often ignores the heterogeneity of strategies. Here, we compile a list of 12 p-hacking strategies based on an extensive literature review, identify factors that control their level of severity, and demonstrate their impact on false-positive rates using simulation studies. We also use our simulation results to evaluate several approaches that have been proposed to mitigate the influence of questionable research practices. Our results show that investigating p-hacking at the level of strategies can provide a better understanding of the process of p-hacking, as well as a broader basis for developing effective countermeasures. By making our analyses available through a Shiny app and R package, we facilitate future meta-scientific research aimed at investigating the ramifications of p-hacking across multiple strategies, and we hope to start a broader discussion about different manifestations of p-hacking in practice.

A review of applications of the Bayes factor in psychological research.

The last 25 years have shown a steady increase in attention for the Bayes factor as a tool for hypothesis evaluation and model selection. The present review highlights the potential of the Bayes factor in psychological research. We discuss six types of applications: Bayesian evaluation of point null, interval, and informative hypotheses, Bayesian evidence synthesis, Bayesian variable selection and model averaging, and Bayesian evaluation of cognitive models. We elaborate what each application entails, give illustrative examples, and provide an overview of key references and software with links to other applications. The article is concluded with a discussion of the opportunities and pitfalls of Bayes factor applications and a sketch of corresponding future research lines. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The predictive power of insomnia symptoms on other aspects of mental health during the COVID-19 pandemic: a longitudinal study.

Symptoms of insomnia are an important risk factor for the development of mental disorders, especially during stressful life periods such as the coronavirus disease 2019 (COVID-19) pandemic. However, up to now, most studies have used cross-sectional data, and the prolonged impact of insomnia symptoms during the pandemic on later mental health remains unclear. Therefore, we investigated insomnia symptoms as a predictor of other aspects of mental health across 6 months, with altogether seven assessments (every 30 days, t0-t6), in a community sample (N = 166-267). Results showed no mean-level increase of insomnia symptoms and/or deterioration of mental health between baseline assessment (t0) and the 6- month follow-up (t6). As preregistered, higher insomnia symptoms (between persons) across all time points predicted reduced mental health at the 6-month follow-up. Interestingly, contrary to our hypothesis, higher insomnia symptoms at 1 month, within each person (i.e., compared to that person's symptoms at other time points), predicted improved rather than reduced aspects of mental health 1 month later. Hence, we replicated the predictive effect of averagely increased insomnia symptoms on impaired later mental health during the COVID-19 pandemic. However, we were surprised that increased insomnia symptoms at 1 month predicted aspects of improved mental health 1 month later. This unexpected effect might be specific for our study population and a consequence of our study design. Overall, increased insomnia symptoms may have served as a signal to engage in, and successfully implement, targeted countermeasures, which led to better short-term mental health in this healthy sample.

Demonstration of a 'leapfrog' randomized controlled trial as a method to accelerate the development and optimization of psychological interventions.

BACKGROUND: The scale of the global mental health burden indicates the inadequacy not only of current treatment options, but also the pace of the standard treatment development process. The 'leapfrog' trial design is a newly-developed simple Bayesian adaptive trial design with potential to accelerate treatment development. A first leapfrog trial was conducted to provide a demonstration and test feasibility, applying the method to a low-intensity internet-delivered intervention targeting anhedonia. METHODS: At the start of this online, single-blind leapfrog trial, participants self-reporting depression were randomized to an initial control arm comprising four weeks of weekly questionnaires, or one of two versions of a four-week cognitive training intervention, imagery cognitive bias modification (imagery CBM). Intervention arms were compared to control on an ongoing basis via sequential Bayesian analyses, based on a primary outcome of anhedonia at post-intervention. Results were used to eliminate and replace arms, or to promote them to become the control condition based on pre-specified Bayes factor and sample size thresholds. Two further intervention arms (variants of imagery CBM) were added into the trial as it progressed. RESULTS: N = 188 participants were randomized across the five trial arms. The leapfrog methodology was successfully implemented to identify a 'winning' version of the imagery CBM, i.e. the version most successful in reducing anhedonia, following sequential elimination of the other arms. CONCLUSIONS: The study demonstrates feasibility of the leapfrog design and provides a foundation for its adoption as a method to accelerate treatment development in mental health. Registration: clinicaltrials.gov, NCT04791137.

Proximal and distal honor fit and subjective well-being in the Mediterranean region.

OBJECTIVE: People's psychological tendencies are attuned to their sociocultural context and culture-specific ways of being, feeling, and thinking are believed to assist individuals in successfully navigating their environment. Supporting this idea, a stronger "fit" with one's cultural environment has often been linked to positive psychological outcomes. The current research expands the cultural, conceptual, and methodological space of cultural fit research by exploring the link between well-being and honor, a central driver of social behavior in the Mediterranean region. METHOD: Drawing on a multi-national sample from eight countries circum-Mediterranean (N = 2257), we examined the relationship between cultural fit in honor and well-being at the distal level (fit with one's perceived society) using response surface analysis (RSA) and at the proximal level (fit with one's university gender group) using profile analysis. RESULTS: We found positive links between fit and well-being in both distal (for some, but not all, honor facets) and proximal fit analyses (across all honor facets). Furthermore, most fit effects in the RSA were complemented with positive level effects of the predictors, with higher average honor levels predicting higher well-being. CONCLUSIONS: Our findings highlight the interplay between individual and environmental factors in honor as well as the important role honor plays in well-being in the Mediterranean region.

The true role that suppressor effects play in condition-based regression analysis: None. A reply to Fiedler (2021).

Condition-based regression analysis (CRA) is a statistical method for testing self-enhancement effects. That is, CRA indicates whether, in a set of empirical data, people with higher values on the directed discrepancy self-view S minus reality criterion R (i.e., S-R) tend to have higher values on some outcome variable (e.g., happiness). In a critical comment, Fiedler (2021) claims that CRA yields inaccurate conclusions in data with a suppressor effect. Here, we show that Fiedler's critique is unwarranted. All data that are simulated in his comment show a positive association between S-R and H, which is accurately detected by CRA. By construction, CRA indicates an association between S-R and H only when it is present in the data. In contrast to Fiedler's claim, it also yields valid conclusions when the outcome variable is related only to the self-view or when there is a suppressor effect. Our clarifications provide guidance for evaluating Fiedler's comment, clear up with the common heuristic that suppressor effects are always problematic, and assist readers in fully understanding CRA. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Motivational Interdependence in Couple Relationships.

This article presents an integrative conceptual model of motivational interdependence in couples, the MIC model. Based on theoretical tenets in motivation psychology, personality psychology, and research on interpersonal perception, the MIC model postulates that two partners' motive dispositions fundamentally interact in shaping their individual motivation and behavior. On a functional level, a partner's motivated behavior is conceptualized as an environmental cue that can contribute to an actor's motive expression and satisfaction. However, the partner's motivated behavior is considered to gain this motivational relevance only via the actor's subjective perception. Multilevel analyses of an extensive experience sampling study on partner-related communal motivation (N = up to 60,803 surveys from 508 individuals nested in 258 couples) supported the MIC model. Participants, particularly those with strong communal motive dispositions, behaved more communally at moments when they perceived their partners to behave more communally. In addition, participants experienced momentary boosts in satisfaction when they behaved more communally and, at the same time, perceived their partners' behavior as similarly communal. Broader implications of the MIC model for research on romantic relationships are discussed.

Efficiency in sequential testing: Comparing the sequential probability ratio test and the sequential Bayes factor test.

In a sequential hypothesis test, the analyst checks at multiple steps during data collection whether sufficient evidence has accrued to make a decision about the tested hypotheses. As soon as sufficient information has been obtained, data collection is terminated. Here, we compare two sequential hypothesis testing procedures that have recently been proposed for use in psychological research: Sequential Probability Ratio Test (SPRT; Psychological Methods, 25(2), 206-226, 2020) and the Sequential Bayes Factor Test (SBFT; Psychological Methods, 22(2), 322-339, 2017). We show that although the two methods have different philosophical roots, they share many similarities and can even be mathematically regarded as two instances of an overarching hypothesis testing framework. We demonstrate that the two methods use the same mechanisms for evidence monitoring and error control, and that differences in efficiency between the methods depend on the exact specification of the statistical models involved, as well as on the population truth. Our simulations indicate that when deciding on a sequential design within a unified sequential testing framework, researchers need to balance the needs of test efficiency, robustness against model misspecification, and appropriate uncertainty quantification. We provide guidance for navigating these design decisions based on individual preferences and simulation-based design analyses.

Cubic response surface analysis: Investigating asymmetric and level-dependent congruence effects with third-order polynomial models.

Congruence hypotheses play a major role in many areas of psychology. They refer to, for example, the consequences of person-environment fit, similarity, or self-other agreement. For example, are people psychologically better adjusted when their self-view is in line with their reputation? A valid statistical approach that can be applied to investigate congruence hypotheses of this kind is quadratic Response Surface Analysis (RSA) in which a second-order polynomial model is fit to the data and appropriately interpreted. However, quadratic RSA does not allow researchers to investigate more precise expectations about a congruence effect. Do the data support an asymmetric congruence effect, in the sense that congruence leads to the highest (or lowest) outcome, but incongruence in one direction (e.g., self-view exceeds reputation) affects the outcome differently than incongruence in the other direction (e.g., self-view falls behind reputation)? Is there a level-dependent congruence effect, such that the amount of congruence is more strongly related to the outcome variable for some levels of the predictors (e.g., high self-view and reputation) than for others (e.g., low self-view and reputation)? Such complex congruence hypotheses have frequently been suggested in the literature, but they could not be investigated because an appropriate statistical approach has yet to be developed. Here, we present analytical strategies, based on third-order polynomial models, that enable users to investigate asymmetric and level-dependent congruence effects, respectively. To facilitate the correct application of the suggested approaches, we provide respective step-by-step guidelines, corresponding R syntax, and illustrative analyses using simulated and real data. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Replicability, Robustness, and Reproducibility in Psychological Science.

Replication-an important, uncommon, and misunderstood practice-is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress.

Measuring motivational relationship processes in experience sampling: A reliability model for moments, days, and persons nested in couples.

The investigation of within-person process models, often done in experience sampling designs, requires a reliable assessment of within-person change. In this paper, we focus on dyadic intensive longitudinal designs where both partners of a couple are assessed multiple times each day across several days. We introduce a statistical model for variance decomposition based on generalizability theory (extending P. E. Shrout & S. P. Lane, 2012), which can estimate the relative proportion of variability on four hierarchical levels: moments within a day, days, persons, and couples. Based on these variance estimates, four reliability coefficients are derived: between-couples, between-persons, within-persons/between-days, and within-persons/between-moments. We apply the model to two dyadic intensive experience sampling studies (n1 = 130 persons, 5 surveys each day for 14 days, ≥ 7508 unique surveys; n2 = 508 persons, 5 surveys each day for 28 days, ≥ 47764 unique surveys). Five different scales in the domain of motivational processes and relationship quality were assessed with 2 to 5 items: State relationship satisfaction, communal motivation, and agentic motivation; the latter consists of two subscales, namely power and independence motivation. Largest variance components were on the level of persons, moments, couples, and days, where within-day variance was generally larger than between-day variance. Reliabilities ranged from .32 to .76 (couple level), .93 to .98 (person level), .61 to .88 (day level), and .28 to .72 (moment level). Scale intercorrelations reveal differential structures between and within persons, which has consequences for theory building and statistical modeling.

The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines.

For a given research question, there are usually a large variety of possible analysis strategies acceptable according to the scientific standards of the field, and there are concerns that this multiplicity of analysis strategies plays an important role in the non-replicability of research findings. Here, we define a general framework on common sources of uncertainty arising in computational analyses that lead to this multiplicity, and apply this framework within an overview of approaches proposed across disciplines to address the issue. Armed with this framework, and a set of recommendations derived therefrom, researchers will be able to recognize strategies applicable to their field and use them to generate findings more likely to be replicated in future studies, ultimately improving the credibility of the scientific process.

Measuring Implicit Motives with the Picture Story Exercise (PSE): Databases of Expert-Coded German Stories, Pictures, and Updated Picture Norms.

We present two openly accessible databases related to the assessment of implicit motives using Picture Story Exercises (PSEs): (a) A database of 183,415 German sentences, nested in 26,389 stories provided by 4,570 participants, which have been coded by experts using Winter's coding system for the implicit affiliation/intimacy, achievement, and power motives, and (b) a database of 54 classic and new pictures which have been used as PSE stimuli. Updated picture norms are provided which can be used to select appropriate pictures for PSE applications. Based on an analysis of the relations between raw motive scores, word count, and sentence count, we give recommendations on how to control motive scores for story length, and validate the recommendation with a meta-analysis on gender differences in the implicit affiliation motive that replicates existing findings. We discuss to what extent the guiding principles of the story length correction can be generalized to other content coding systems for narrative material. Several potential applications of the databases are discussed, including (un)supervised machine learning of text content, psychometrics, and better reproducibility of PSE research.

Predicting affective appraisals from facial expressions and physiology using machine learning.

The present study explored the interrelations between a broad set of appraisal ratings and five physiological signals, including facial EMG, electrodermal activity, and heart rate variability, that were assessed in 157 participants watching 10 emotionally charged videos. A total of 134 features were extracted from the physiological data, and a benchmark comparing different kinds of machine learning algorithms was conducted to test how well the appraisal dimensions can be predicted from these features. For 13 out of 21 appraisals, a robust positive R2 was attained, indicating that the dimensions are actually related to the considered physiological channels. The highest R2 (.407) was reached for the appraisal dimension intrinsic pleasantness. Moreover, the comparison of linear and nonlinear algorithms and the inspection of the links between the appraisals and single physiological features using accumulated local effects plots indicates that the relationship between physiology and appraisals is nonlinear. By constructing different importance measures for the assessed physiological channels, we showed that for the 13 predictable appraisals, the five channels explained different amounts of variance and that only a few blocks incrementally explained variance beyond the other physiological channels.