Factors Associated with Interviewers' Evaluations of Respondents' Performance in Telephone Interviews: Behavior, Response Quality Indicators, and Characteristics of Respondents and Interviewers.

Interviewers' postinterview evaluations of respondents' performance (IEPs) are paradata, used to describe the quality of the data obtained from respondents. IEPs are driven by a combination of factors, including respondents' and interviewers' sociodemographic characteristics and what actually transpires during the interview. However, relatively few studies examine how IEPs are associated with features of the response process, including facets of the interviewer-respondent interaction and patterns of responding that index data quality. We examine whether features of the response process-various respondents' behaviors and response quality indicators-are associated with IEPs in a survey with a diverse set of respondents focused on barriers and facilitators to participating in medical research. We also examine whether there are differences in IEPs across respondents' and interviewers' sociodemographic characteristics. Our results show that both respondents' behaviors and response quality indicators predict IEPs, indicating that IEPs reflect what transpires in the interview. In addition, interviewers appear to approach the task of evaluating respondents with differing frameworks, as evidenced by the variation in IEPs attributable to interviewers and associations between IEPs and interviewers' gender. Further, IEPs were associated with respondents' education and ethnoracial identity, net of respondents' behaviors, response quality indicators, and sociodemographic characteristics of respondents and interviewers. Future research should continue to build on studies that examine the correlates of IEPs to better inform whether, when, and how to use IEPs as paradata about the quality of the data obtained.

The measurement of gender expression in survey research.

Previous research on the survey measurement of sexual orientation, gender identity, and gender expression (SOGIE) often focuses on the measurement of identity, with comparably little research focused on gender expression as a key feature of how gender is lived and experienced. This study examines the reliability and validity of survey questions about gender expression in a 2-by-5-by-2 factorial experiment that varies the question order, type of response scale, and the order of gender presentation in the response scale. The results indicate that the effect of which (side of the) scale is presented first on gender expression varies by gender for each of the unipolar items and one of the bipolar items (behavior). In addition, the unipolar items also show distinctions among the gender minority population in ratings of gender expression as well as more nuance with respect to concurrent validity in predicting health outcomes among cisgender respondents. The results of this study have implications for researchers who are interested in accounting for gender holistically in survey and health disparities research.

Exploring the Relationship between Medical Research Literacy and Respondents' Expressed Likelihood to Participate in a Clinical Trial.

Medical research literacy (MRL) is a facet of health literacy that measures a person's understanding of informed consent and other aspects of participation in medical research. While existing research on MRL is limited, there are reasons to believe MRL may be associated with a willingness to participate in medical research. We use data from a racially balanced sample of survey respondents (n = 410): (1) to analyze how MRL scores vary by respondents' socio-demographic characteristics; (2) to examine how MRL relates to respondents' expressed likelihood to participate in a clinical trial; and (3) to provide considerations on the measurement of MRL. The results indicate no differences in MRL scores by race or gender; younger (p < 0.05) and more educated (p < 0.001) individuals have significantly higher MRL scores. Further, higher MRL scores are associated with significantly lower levels of expressed likelihood to participate in a clinical trial. Additionally, the MRL scale included both true and false statements, and analyses demonstrate significant differences in how these relate to outcomes. Altogether, the results signal that further research is needed to understand MRL and how it relates to socio-demographic characteristics associated with research participation and can be measured effectively.

Towards a reconsideration of the use of agree-disagree questions in measuring subjective evaluations.

Agree-disagree (AD) or Likert questions (e.g., "I am extremely satisfied: strongly agree … strongly disagree") are among the most frequently used response formats to measure attitudes and opinions in the social and medical sciences. This review and research synthesis focuses on the measurement properties and potential limitations of AD questions. The research leads us to advocate for an alternative questioning strategy in which items are written to directly ask about their underlying response dimensions using response categories tailored to match the response dimension, which we refer to as item-specific (IS) (e.g., "How satisfied are you: not at all … extremely"). In this review we: 1) synthesize past research comparing data quality for AD and IS questions; 2) present conceptual models of and review research supporting respondents' cognitive processing of AD and IS questions; and 3) provide an overview of question characteristics that frequently differ between AD and IS questions and may affect respondents' cognitive processing and data quality. Although experimental studies directly comparing AD and IS questions yield some mixed results, more studies find IS questions are associated with desirable data quality outcomes (e.g., validity and reliability) and AD questions are associated with undesirable outcomes (e.g., acquiescence, response effects, etc.). Based on available research, models of cognitive processing, and a review of question characteristics, we recommended IS questions over AD questions for most purposes. For researchers considering the use of previously administered AD questions and instruments, issues surrounding the challenges of translating questions from AD to IS response formats are discussed.

The Survey Measurement of Sexual Orientation: Configurations of Sexual Identity and Attraction and Associations with Mental Health.

Purpose: This study aimed to examine how configurations of sexual identity and attraction are associated with mental health outcomes. Methods: Data came from the 2015, 2016, and 2017 waves of the National Survey on Drug Use and Health, one of the few nationally representative surveys to ask about sexual attraction. Sexual identity and attraction were combined into groups that are coincident (heterosexual-opposite gender attraction, gay/lesbian-same gender attraction, or bisexual-any multiple gender attraction) or branched (heterosexual-any same gender attraction, gay/lesbian-any opposite gender attraction, bisexual-only same or opposite gender attraction). The association between these configurations and various measures of mental health and well-being-severe psychological distress, major depressive episode, suicidal ideation, and suicide plan or attempt-was examined. Results: Heterosexual coincidence-being heterosexual and only attracted to the opposite gender-was associated with lower mental health risks than all other configurations of sexual identity and attraction. In addition, bisexual with coincident attraction was often associated with worse mental health outcomes than other configurations of identity and attraction, whereas bisexual with branched attraction did not necessarily follow this pattern. Finally, heterosexual with branched attraction was associated with worse mental health outcomes than heterosexual with coincident attraction, but better mental health outcomes than some of the other sexual identity and attraction configurations. Conclusion: Including one question on sexual attraction and its intersection with sexual identity adds nuance to our understanding of disparities in mental health and well-being among previously identified sexual minority and majority groups.

Perceptions of Cultural Competency Among Premedical Undergraduate Students.

BACKGROUND: Cultural competence is a difficult skill to teach, as it has several operational definitions as well as limited and unstandardized training procedures. Currently, there is no formal cultural competency training at the undergraduate level for students who seek to become a medical doctor. The purpose of this study is to explore perceptions of cultural competence among premedical undergraduates by assessing how they define and understand cultural competency and their knowledge (and sources thereof) of sociocultural realities in health and medicine. METHODS: Structured in-depth interviews took place in 2016 and 2017 at a medium-sized private college in the Midwestern United States. Twenty premedical students were interviewed. The interviews were transcribed and thematically coded following an inductive, iterative, and systematic process. RESULTS: Most students can provide a definition of cultural competence that includes at least one component of how it is conceptualized by the Association of American Medical Colleges. However, students focus largely on defining cultural competence as individual attitudes and interaction rather than systemic or structural realities that produce inequalities in health care. When explicitly asked, students varied in the level of detail provided in explaining the social determinants of health (such as race or ethnicity, sex, gender, and socioeconomic status) and varied in the accuracy of their definitions of traditional health practices. Each student noted the importance of training on cultural competence and many placed patients' health at the center of their reason for doing so rather than focusing on their own training as a motivation. Students discussed various aspects of sociocultural differences and the need for physicians to understand patients' outlooks on health care and be able to communicate to patients the purpose of suggested medical treatment, as well as the inherent tension in balancing patients as individuals and members of sociocultural groups. Premedical undergraduate students see their own cultural competence as an informal skill that is gained through social interactions across various areas of life, such as work, family, friends, and school. CONCLUSION: This study traces the sources of sociocultural information that premedical students will bring to their medical training as well as places where cultural competence can be further explored, practiced, and formally integrated in premedical education.

Gender Differences in Program Factors Important to Applicants When Evaluating Orthopaedic Surgery Residency Programs.

BACKGROUND: Despite specialty-driven efforts to improve diversity in the field, few women apply to orthopaedic residency, and women are unevenly distributed among programs. There is little evidence-based information on factors that may attract female applicants. OBJECTIVE: This study aims to identify factors important to applicants when evaluating orthopaedic residency programs and to identify gender-specific differences. METHODS: All applicants to a single orthopaedic surgery residency program in the 2017 Match were asked to fill out an anonymous survey. Respondents rated the importance of 35 factors when evaluating orthopaedic residency programs. The percentage of highly rated factors was calculated. Statistical analysis was performed for each factor to assess differences by gender. RESULTS: Of 1013 applicants who applied to orthopaedic surgery residency in 2017, 815 (80%) applied to our program, and 218 (27%) completed the survey. The most important factors when evaluating a residency program for both genders were (1) perceptions of current residents; (2) interactions with members of the program; (3) program reputation and fellowship placement; (4) geographic location; and (5) impressions after rotation at a program. Female applicants rated the presence of female and minority residents and faculty and program reputation for gender and racial/ethnic diversity higher than male applicants. CONCLUSIONS: When choosing an orthopaedic surgery residency program, women more often reported the presence of female residents and faculty, program reputation for gender diversity, reputation for racial/ethnic diversity, presence of minority residents and faculty, and their personal interactions with members of the program as important factors.

Different domains of physical activity: The role of leisure, housework/care work, and paid work in socioeconomic differences in reported physical activity.

Inequality in socioeconomic status (SES)-education, income, and occupation-may further exacerbate the health gap between the "haves" and "have nots" by shaping health behaviors such as physical activity. For example, those in higher socioeconomic positions are consistently found to engage in more physical activity according to public health reports that focus on leisure activity. However, previous research investigating the role of SES in shaping engagement in housework, childcare, and paid work suggests different opportunities for physical activity. This discrepancy in how researchers ask questions about physical activity and the pathways people take to healthy activity raises the question: Do socioeconomic differences in physical activity look different when we look at other domains of physical activity beyond leisure? And, does how we measure SES matter? We draw on data from the American Time Use Survey (ATUS) to assess the roles of education, income, and occupation in the amount of time individuals spend in different types of physical activity. Results demonstrate that socioeconomic differences in physical activity change depending on the activity domain and, therefore, when all domains of physical activity are accounted for compared to leisure-only. Further, the measurement of SES matters: key indicators of SES (education, income, and occupation) have varying associations with levels and types of physical activity. Findings from this research have important implications for the assessment of physical activity across SES, ultimately impacting survey research and public health.

Interviewers' Ratings of Respondents' Health: Predictors and Association With Mortality.

OBJECTIVES: Recent research indicates that survey interviewers' ratings of respondents' health (IRH) may provide supplementary health information about respondents in surveys of older adults. Although IRH is a potentially promising measure of health to include in surveys, our understanding of the factors contributing to IRH remains incomplete. METHODS: We use data from the 2011 face-to-face wave of the Wisconsin Longitudinal Study, a longitudinal study of older adults from the Wisconsin high school class of 1957 and their selected siblings. We first examine whether a range of factors predict IRH: respondents' characteristics that interviewers learn about and observe as respondents answer survey questions, interviewers' evaluations of some of what they observe, and interviewers' characteristics. We then examine the role of IRH, respondents' self-rated health (SRH), and associated factors in predicting mortality over a 3-year follow-up. RESULTS: As in prior studies, we find that IRH is associated with respondents' characteristics. In addition, this study is the first to document how IRH is associated with both interviewers' evaluations of respondents and interviewers' characteristics. Furthermore, we find that the association between IRH and the strong criterion of mortality remains after controlling for respondents' characteristics and interviewers' evaluations of respondents. DISCUSSION: We propose that researchers incorporate IRH in surveys of older adults as a cost-effective, easily implemented, and supplementary measure of health.

Measuring Trust in Medical Researchers: Adding Insights from Cognitive Interviews to Examine Agree-Disagree and Construct-Specific Survey Questions.

While scales measuring subjective constructs historically rely on agree-disagree (AD) questions, recent research demonstrates that construct-specific (CS) questions clarify underlying response dimensions that AD questions leave implicit and CS questions often yield higher measures of data quality. Given acknowledged issues with AD questions and certain established advantages of CS items, the evidence for the superiority of CS questions is more mixed than one might expect. We build on previous investigations by using cognitive interviewing to deepen understanding of AD and CS response processing and potential sources of measurement error. We randomized 64 participants to receive an AD or CS version of a scale measuring trust in medical researchers. We examine several indicators of data quality and cognitive response processing including: reliability, concurrent validity, recency, response latencies, and indicators of response processing difficulties (e.g., uncodable answers). Overall, results indicate reliability is higher for the AD scale, neither scale is more valid, and the CS scale is more susceptible to recency effects for certain questions. Results for response latencies and behavioral indicators provide evidence that the CS questions promote deeper processing. Qualitative analysis reveals five sources of difficulties with response processing that shed light on under-examined reasons why AD and CS questions can produce different results, with CS not always yielding higher measures of data quality than AD.

Which activities count? Using experimental data to understand conceptualizations of physical activity.

US health surveys consistently report that men and those with higher socioeconomic status (SES) engage in more physical activity than women and lower SES counterparts, using questions that ask about physical activity during leisure time. However, social characteristics such as gender and SES shape understandings of and access to leisure-based physical activity as well as other domains where healthy activity is available - namely house work, care work, and paid work. Thus, the physical activity of US adults may look different when what counts as physical activity expands beyond leisure activity. The current study uses Amazon's Mechanical Turk platform to conduct a 2-by-2-by-2 factorial experiment that crosses three types of physical activities: leisure, house or care work, and paid work. We find that physical activity questions that prime respondents - that is, ask respondents - to consider house/care work or paid work lead to increased minutes reported of physical activity compared to not priming for physical activity, while asking about leisure is no different from having no physical activity primed. The effect on reported physical activity of priming with house/care work is stronger for women than men, demonstrating support for gendered specialization of time spent in the house and care work domain. The effects on reported physical activity of priming with house/care work and paid work are stronger for those with less education compared to more education, consistent with socioeconomic divisions in access to physical activity in house/care work and employment. This study highlights the contingence of our understanding of the physical activity of US adults on both its measurement in surveys and the social forces which shape understanding of and access to physical activity.

GREETING AND RESPONSE: PREDICTING PARTICIPATION FROM THE CALL OPENING.

Although researchers have used phone surveys for decades, the lack of an accurate picture of the call opening reduces our ability to train interviewers to succeed. Sample members decide about participation quickly. We predict participation using the earliest moments of the call; to do this, we analyze matched pairs of acceptances and declinations from the Wisconsin Longitudinal Study using a case-control design and conditional logistic regression. We focus on components of the first speaking turns: acoustic-prosodic components and interviewer's actions. The sample member's "hello" is external to the causal processes within the call and may carry information about the propensity to respond. As predicted by Pillet-Shore (2012), we find that when the pitch span of the sample member's "hello" is greater the odds of participation are higher, but in contradiction to her prediction, the (less reliably measured) pitch pattern of the greeting does not predict participation. The structure of actions in the interviewer's first turn has a large impact. The large majority of calls in our analysis begin with either an "efficient" or "canonical" turn. In an efficient first turn, the interviewer delays identifying themselves (and thereby suggesting the purpose of the call) until they are sure they are speaking to the sample member, with the resulting efficiency that they introduce themselves only once. In a canonical turn, the interviewer introduces themselves and asks to speak to the sample member, but risks having to introduce themselves twice if the answerer is not the sample member. The odds of participation are substantially and significantly lower for an efficient turn compared to a canonical turn. It appears that how interviewers handle identification in their first turn has consequences for participation; an analysis of actions could facilitate experiments to design first interviewer turns for different target populations, study designs, and calling technologies.

How participants report their health status: cognitive interviews of self-rated health across race/ethnicity, gender, age, and educational attainment.

BACKGROUND: Self-rated health (SRH) is widely used to measure subjective health. Yet it is unclear what underlies health ratings, with implications for understanding the validity of SRH overall and across sociodemographic characteristics. We analyze participants' explanations of how they formulated their SRH answer in addition to which health factors they considered and examine group differences in these processes. METHODS: Cognitive interviews were conducted with 64 participants in a convenience quota sample crossing dimensions of race/ethnicity (white, Latino, black, American Indian), gender, age, and education. Participants rated their health then described their thoughts when answering SRH. We coded participants' answers in an inductive, iterative, and systematic process from interview transcripts, developing analytic categories (i.e., themes) and subdimensions within. We examined whether the presence of each dimension of an analytic category varied across sociodemographic groups. RESULTS: Our qualitative analysis led to the identification and classification of various subdimensions of the following analytic categories: types of health factors mentioned, valence of health factors, temporality of health factors, conditional health statements, and descriptions and definitions of health. We found differences across groups in some types of health factors mentioned-corresponding, conflicting, or novel with respect to prior research. Furthermore, we also documented various processes through which respondents integrate seemingly disparate health factors to formulate an answer through valence and conditional health statements. Finally, we found some evidence of sociodemographic group differences with respect to types of health factors mentioned, valence of health factors, and conditional health statements, highlighting avenues for future research. CONCLUSION: This study provides a description of how participants rate their general health status and highlights potential differences in these processes across sociodemographic groups, helping to provide a more comprehensive understanding of how SRH functions as a measure of health.

Interviewing Practices, Conversational Practices, and Rapport: Responsiveness and Engagement in the Standardized Survey Interview.

"Rapport" has been used to refer to a range of positive psychological features of an interaction -- including a situated sense of connection or affiliation between interactional partners, comfort, willingness to disclose or share sensitive information, motivation to please, or empathy. Rapport could potentially benefit survey participation and response quality by increasing respondents' motivation to participate, disclose, or provide accurate information. Rapport could also harm data quality if motivation to ingratiate or affiliate caused respondents to suppress undesirable information. Some previous research suggests that motives elicited when rapport is high conflict with the goals of standardized interviewing. We examine rapport as an interactional phenomenon, attending to both the content and structure of talk. Using questions about end-of-life planning in the 2003-2005 wave of the Wisconsin Longitudinal Study, we observe that rapport consists of behaviors that can be characterized as dimensions of responsiveness by interviewers and engagement by respondents. We identify and describe types of responsiveness and engagement in selected question-answer sequences and then devise a coding scheme to examine their analytic potential with respect to the criterion of future study participation. Our analysis suggests that responsive and engaged behaviors vary with respect to the goals of standardization-some conflict with these goals, while others complement them.

Research in and Prospects for the Measurement of Health Using Self-Rated Health.

Self-rated health (SRH)-for example, "in general would you say your health is excellent, very good, good, fair, or poor?"-is the most widely used measure of health across a range of survey research studies. This paper synthesizes extant research and provides a framework for future research on the measurement of health using SRH, focusing on four interrelated topics: the factors that influence respondents' health ratings, the survey measurement features of SRH, how SRH answers are analyzed, and the stated purpose of SRH as a proxy for more objective health or as a perception of health. Extant research on the health, psychological, and social factors influencing respondents' SRH answers is reviewed, as is research concerned with the survey measurement features of SRH that influence how respondents rate their health. The synthesis proposes a framework for future research that focuses on further explicating the factors that underlie respondents' SRH answers and improving features of SRH measurement and analysis in ways that are consistent with the various goals of the researchers who both collect and analyze the data.

The effect of response option order on self-rated health: a replication study.

PURPOSE: Following calls for replication of research studies, this study documents the results of two studies that experimentally examine the impact of response option order on self-rated health (SRH). METHODS: Two studies from an online panel survey examined how the order of response options (positive to negative versus negative to positive) influences the distribution of SRH answers. RESULTS: The results of both studies indicate that the distribution of SRH varies across the experimental treatments, and mean SRH is lower (worse) when the response options start with "poor" rather than "excellent." In addition, there are differences across the two studies in the distribution of SRH and mean SRH when the response options begin with "excellent," but not when the response options begin with "poor." CONCLUSION: The similarities in the general findings across the two studies strengthen the claim that SRH will be lower (worse) when the response options are ordered beginning with "poor" rather than "excellent" in online self-administered questionnaires, with implications for the validity of SRH. The slight differences in the administration of the seemingly identical studies further strengthen the claim and also serve as a reminder of the inherent variability of a single permutation of any given study.

Racial/ethnic disparities in midlife depressive symptoms: The role of cumulative disadvantage across the life course.

This study examines the role of cumulative disadvantage mechanisms across the life course in the production of racial and ethnic disparities in depressive symptoms at midlife, including the early life exposure to health risk factors, the persistent exposure to health risk factors, and varying mental health returns to health risk factors across racial and ethnic groups. Using data from the over-40 health module of the National Longitudinal Study of Youth (NLSY) 1979 cohort, this study uses regression decomposition techniques to attend to differences in the composition of health risk factors across racial and ethnic groups, differences by race and ethnicity in the association between depressive symptoms and health risk factors, and how these differences combine within racial and ethnic groups to produce group-specific levels of--and disparities in--depressive symptoms at midlife. While the results vary depending on the groups being compared across race/ethnicity and gender, the study documents how racial and ethnic mental health disparities at midlife stem from life course processes of cumulative disadvantage through both unequal distribution and unequal associations across racial and ethnic groups.

The effects of response option order and question order on self-rated health.

OBJECTIVES: This study aims to assess the impact of response option order and question order on the distribution of responses to the self-rated health (SRH) question and the relationship between SRH and other health-related measures. METHODS: In an online panel survey, we implement a 2-by-2 between-subjects factorial experiment, manipulating the following levels of each factor: (1) order of response options ("excellent" to "poor" versus "poor" to "excellent") and (2) order of SRH item (either preceding or following the administration of domain-specific health items). We use Chi-square difference tests, polychoric correlations, and differences in means and proportions to evaluate the effect of the experimental treatments on SRH responses and the relationship between SRH and other health measures. RESULTS: Mean SRH is higher (better health) and proportion in "fair" or "poor" health lower when response options are ordered from "excellent" to "poor" and SRH is presented first compared to other experimental treatments. Presenting SRH after domain-specific health items increases its correlation with these items, particularly when response options are ordered "excellent" to "poor." Among participants with the highest level of current health risks, SRH is worse when it is presented last versus first. CONCLUSION: While more research on the presentation of SRH is needed across a range of surveys, we suggest that ordering response options from "poor" to "excellent" might reduce positive clustering. Given the question order effects found here, we suggest presenting SRH before domain-specific health items in order to increase inter-survey comparability, as domain-specific health items will vary across surveys.

Comparing self and maternal reports of adolescents' general health status: do self and proxy reports differ in their relationships with covariates?

PURPOSE: Given that mothers often-but do not always-report children's health status in surveys, it is essential to gain an understanding of whether the relationship between children's general health status and relevant covariates depends on who reports children's general health status. METHODS: Using data from the first wave of the National Longitudinal Study of Youth 1997 cohort (N = 6,466), a nationally representative sample of adolescents in the United States ages 12 to 17 in 1997, the study first examined the concordance between self and maternal reports of adolescents' general health status. Then, self and maternal reports of adolescents' general health status were each regressed on health-relevant covariates, and tests of differences in coefficients across the models were estimated. RESULTS: Self and maternal reports of adolescents' general health status are moderately concordant. Furthermore, the associations of adolescents' general health status with adolescent BMI and the adolescent being female significantly differ across reporters, such that the negative relationships are even more negative with self compared to maternal reports of adolescents' general health status. The associations of adolescents' general health status with the measures of adolescents' health limitations, maternal self-rated health, and certain sociodemographic covariates differ across reporters, such that each has a greater relationship with maternal compared to self-reports of adolescents' general health status. CONCLUSION: The results are important for interpreting research on the causes and consequences of child and adolescent health, as results across studies may not be comparable if the reporter is not the same.

The interplay between child and maternal health: reciprocal relationships and cumulative disadvantage during childhood and adolescence.

While many studies use parental socioeconomic status and health to predict children's health, this study examines the interplay over time between child and maternal health across childhood and adolescence. Using data from women in the National Longitudinal Study of Youth 1979 cohort and their children (N = 2,225), autoregressive cross-lagged models demonstrate a reciprocal relationship between child activity limitations and maternal health limitations in direct effects of child activity limitations on maternal health limitations two years later and vice versa-net of a range of health-relevant time-varying and time-invariant covariates. Furthermore, there are indirect effects of child activity limitations on subsequent maternal health limitations and indirect effects of maternal health limitations on subsequent child activity limitations via intervening health statuses. This study examines how the interplay between child and maternal health unfolds over time and describes how these interdependent statuses jointly experience health disadvantages.