Did people really drink bleach to prevent COVID-19? A guide for protecting survey data against problematic respondents.

Survey respondents who are non-attentive, respond randomly, or misrepresent who they are can impact the outcomes of surveys. Prior findings reported by the CDC have suggested that people engaged in highly dangerous cleaning practices during the COVID-19 pandemic, including ingesting household cleaners such as bleach. In our attempts to replicate the CDC's results, we found that 100% of reported ingestion of household cleaners are made by problematic respondents. Once inattentive, acquiescent, and careless respondents are removed from the sample, we find no evidence that people ingested cleaning products to prevent a COVID-19 infection. These findings have important implications for public health and medical survey research, as well as for best practices for avoiding problematic respondents in all survey research conducted online.

Is it ethical to use Mechanical Turk for behavioral research? Relevant data from a representative survey of MTurk participants and wages.

To understand human behavior, social scientists need people and data. In the last decade, Amazon's Mechanical Turk (MTurk) emerged as a flexible, affordable, and reliable source of human participants and was widely adopted by academics. Yet despite MTurk's utility, some have questioned whether researchers should continue using the platform on ethical grounds. The brunt of their concern is that people on MTurk are financially insecure, subject to abuse, and earn inhumane wages. We investigated these issues with two representative probability surveys of the U.S. MTurk population (N = 4094). The surveys revealed: (1) the financial situation of people on MTurk mirrors the general population, (2) most participants do not find MTurk stressful or requesters abusive, and (3) MTurk offers flexibility and benefits that most people value above other options for work. People reported it is possible to earn more than $10 per hour and said they would not trade the flexibility of MTurk for less than $25 per hour. Altogether, our data are important for assessing whether MTurk is an ethical place for research.

Evaluating CloudResearch's Approved Group as a solution for problematic data quality on MTurk.

Maintaining data quality on Amazon Mechanical Turk (MTurk) has always been a concern for researchers. These concerns have grown recently due to the bot crisis of 2018 and observations that past safeguards of data quality (e.g., approval ratings of 95%) no longer work. To address data quality concerns, CloudResearch, a third-party website that interfaces with MTurk, has assessed ~165,000 MTurkers and categorized them into those that provide high- (~100,000, Approved) and low- (~65,000, Blocked) quality data. Here, we examined the predictive validity of CloudResearch's vetting. In a pre-registered study, participants (N = 900) from the Approved and Blocked groups, along with a Standard MTurk sample (95% HIT acceptance ratio, 100+ completed HITs), completed an array of data-quality measures. Across several indices, Approved participants (i) identified the content of images more accurately, (ii) answered more reading comprehension questions correctly, (iii) responded to reversed coded items more consistently, (iv) passed a greater number of attention checks, (v) self-reported less cheating and actually left the survey window less often on easily Googleable questions, (vi) replicated classic psychology experimental effects more reliably, and (vii) answered AI-stumping questions more accurately than Blocked participants, who performed at chance on multiple outcomes. Data quality of the Standard sample was generally in between the Approved and Blocked groups. We discuss how MTurk's Approval Rating system is no longer an effective data-quality control, and we discuss the advantages afforded by using the Approved group for scientific studies on MTurk.

Do you know the Wooly Bully? Testing era-based knowledge to verify participant age online.

People in online studies sometimes misrepresent themselves. Regardless of their motive for doing so, participant misrepresentation threatens the validity of research. Here, we propose and evaluate a way to verify the age of online respondents: a test of era-based knowledge. Across six studies (N = 1543), participants of various ages completed an age verification instrument. The instrument assessed familiarity with cultural phenomena (e.g., songs and TV shows) from decades past and present. We consistently found that our instrument discriminated between people of different ages. In Studies 1a and 1b, self-reported age correlated strongly with performance on the instrument (mean r = .8). In Study 2, the instrument reliably detected imposters who we knew were misrepresenting their age. For impostors, self-reported age did not correlate with performance on the instrument (r = .077). Finally, in Studies 3a, 3b, and 3c, the instrument remained robust with African Americans, people from low educational backgrounds, and recent immigrants to the United States. Thus, our instrument shows promise for verifying the age of online respondents, and, as we discuss, our approach of assessing "insider knowledge" holds great promise for verifying other identities within online studies.

Psychological Distress Among the First Quarantined Community in the United States: Initial Observations From the Early Days of the COVID-19 Crisis.

This study assesses distress and anxiety symptoms associated with quarantine due to COVID-19 exposure among the first quarantined community in the United States and identifies potential areas of intervention. All participants were directly or peripherally related to "patient 1,"-the first confirmed community-acquired case of COVID-19 in the New York Area. As such, this is a historically significant sample whose experiences highlight a transitional moment from a pre-pandemic to a pandemic period in the United States. In March 2020, an anonymous survey was distributed to 1,250 members of a NYC area community that was under community-wide quarantine orders due to the COVID-19 outbreak. Distress was measured using the Subjective Units of Distress Scale (SUDS) and symptoms of anxiety were measured using the Beck Anxiety Inventory (BAI). A variety of psychosocial predictors relevant to the current crisis were explored. Three hundred and three individuals responded within forty-eight hours of survey distribution. Mean levels of distress in the sample were heightened and sustained, with 69% reporting moderate to severe distress on the SUDS and 53% of the sample reported mild, moderate, or severe anxiety symptoms on the BAI. The greatest percentage of variance of distress and anxiety symptoms was accounted for by modifiable factors amenable to behavioral and psychological interventions.

A Look at the First Quarantined Community in the USA: Response of Religious Communal Organizations and Implications for Public Health During the COVID-19 Pandemic.

The current study examined anxiety and distress among members of the first community to be quarantined in the USA due to the COVID-19 pandemic. In addition to being historically significant, the current sample was unusual in that those quarantined were all members of a Modern Orthodox Jewish community and were connected via religious institutions at which exposure may have occurred. We sought to explore the community and religious factors unique to this sample, as they relate to the psychological and public health impact of quarantine. Community organizations were trusted more than any other source of COVID-19-related information, including federal, state and other government agencies, including the CDC, WHO and media news sources. This was supported qualitatively with open-ended responses in which participants described the range of supports organized by community organizations. These included tangible needs (i.e., food delivery), social support, virtual religious services, and dissemination of COVID-19-related information. The overall levels of distress and anxiety were elevated and directly associated with what was reported to be largely inadequate and inconsistent health-related information received from local departments of health. In addition, the majority of participants felt that perception of or concern about future stigma related to a COVID-19 diagnosis or association of COVID-19 with the Jewish community was high and also significantly predicted distress and anxiety. The current study demonstrates the ways in which religious institutions can play a vital role in promoting the well-being of their constituents. During this unprecedented pandemic, public health authorities have an opportunity to form partnerships with religious institutions in the common interests of promoting health, relaying accurate information and supporting the psychosocial needs of community members, as well as protecting communities against stigma and discrimination.

The persistence of pay inequality: The gender pay gap in an anonymous online labor market.

Studies of the gender pay gap are seldom able to simultaneously account for the range of alternative putative mechanisms underlying it. Using CloudResearch, an online microtask platform connecting employers to workers who perform research-related tasks, we examine whether gender pay discrepancies are still evident in a labor market characterized by anonymity, relatively homogeneous work, and flexibility. For 22,271 Mechanical Turk workers who participated in nearly 5 million tasks, we analyze hourly earnings by gender, controlling for key covariates which have been shown previously to lead to differential pay for men and women. On average, women's hourly earnings were 10.5% lower than men's. Several factors contributed to the gender pay gap, including the tendency for women to select tasks that have a lower advertised hourly pay. This study provides evidence that gender pay gaps can arise despite the absence of overt discrimination, labor segregation, and inflexible work arrangements, even after experience, education, and other human capital factors are controlled for. Findings highlight the need to examine other possible causes of the gender pay gap. Potential strategies for reducing the pay gap on online labor markets are also discussed.

Tapped out or barely tapped? Recommendations for how to harness the vast and largely unused potential of the Mechanical Turk participant pool.

Mechanical Turk (MTurk) is a common source of research participants within the academic community. Despite MTurk's utility and benefits over traditional subject pools some researchers have questioned whether it is sustainable. Specifically, some have asked whether MTurk workers are too familiar with manipulations and measures common in the social sciences, the result of many researchers relying on the same small participant pool. Here, we show that concerns about non-naivete on MTurk are due less to the MTurk platform itself and more to the way researchers use the platform. Specifically, we find that there are at least 250,000 MTurk workers worldwide and that a large majority of US workers are new to the platform each year and therefore relatively inexperienced as research participants. We describe how inexperienced workers are excluded from studies, in part, because of the worker reputation qualifications researchers commonly use. Then, we propose and evaluate an alternative approach to sampling on MTurk that allows researchers to access inexperienced participants without sacrificing data quality. We recommend that in some cases researchers should limit the number of highly experienced workers allowed in their study by excluding these workers or by stratifying sample recruitment based on worker experience levels. We discuss the trade-offs of different sampling practices on MTurk and describe how the above sampling strategies can help researchers harness the vast and largely untapped potential of the Mechanical Turk participant pool.

Online panels in social science research: Expanding sampling methods beyond Mechanical Turk.

Amazon Mechanical Turk (MTurk) is widely used by behavioral scientists to recruit research participants. MTurk offers advantages over traditional student subject pools, but it also has important limitations. In particular, the MTurk population is small and potentially overused, and some groups of interest to behavioral scientists are underrepresented and difficult to recruit. Here we examined whether online research panels can avoid these limitations. Specifically, we compared sample composition, data quality (measured by effect sizes, internal reliability, and attention checks), and the non-naivete of participants recruited from MTurk and Prime Panels-an aggregate of online research panels. Prime Panels participants were more diverse in age, family composition, religiosity, education, and political attitudes. Prime Panels participants also reported less exposure to classic protocols and produced larger effect sizes, but only after screening out several participants who failed a screening task. We conclude that online research panels offer a unique opportunity for research, yet one with some important trade-offs.

Accumulative fusion and the issue of age: Reconciling the model with the data.

We discuss a disconnect between the predictions of Whitehouse's model regarding the accumulative nature of fusion and real-world data regarding the age at which people generally engage in self-sacrifice. We argue that incorporating the link between age and identity development into Whitehouse's theoretical framework is central to understanding when and why people engage in self-sacrifice on behalf of the group.

The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk.

In this study, we examined data quality among Amazon Mechanical Turk (MTurk) workers based in India, and the effect of monetary compensation on their data quality. Recent studies have shown that work quality is independent of compensation rates, and that compensation primarily affects the quantity but not the quality of work. However, the results of these studies were generally based on compensation rates below the minimum wage, and far below a level that was likely to play a practical role in the lives of workers. In this study, compensation rates were set around the minimum wage in India. To examine data quality, we developed the squared discrepancy procedure, which is a task-based quality assurance approach for survey tasks whose goal is to identify inattentive participants. We showed that data quality is directly affected by compensation rates for India-based participants. We also found that data were of a lesser quality among India-based than among US participants, even when optimal payment strategies were utilized. We additionally showed that the motivation of MTurk users has shifted, and that monetary compensation is now reported to be the primary reason for working on MTurk, among both US- and India-based workers. Overall, MTurk is a constantly evolving marketplace where multiple factors can contribute to data quality. High-quality survey data can be acquired on MTurk among India-based participants when an appropriate pay rate is provided and task-specific quality assurance procedures are utilized.