Estimating Web Survey Mode and Panel Effects in a Nationwide Survey of Alcohol Use.

Random-digit dialing (RDD) telephone surveys are challenged by declining response rates and increasing costs. Many surveys that were traditionally conducted via telephone are seeking cost-effective alternatives, such as address-based sampling (ABS) with self-administered web or mail questionnaires. At a fraction of the cost of both telephone and ABS surveys, opt-in web panels are an attractive alternative. The 2019-2020 National Alcohol Survey (NAS) employed three methods: (1) an RDD telephone survey (traditional NAS method); (2) an ABS push-to-web survey; and (3) an opt-in web panel. The study reported here evaluated differences in the three data-collection methods, which we will refer to as "mode effects," on alcohol consumption and health topics. To evaluate mode effects, multivariate regression models were developed predicting these characteristics, and the presence of a mode effect on each outcome was determined by the significance of the three-level effect (RDD-telephone, ABS-web, opt-in web panel) in each model. Those results were then used to adjust for mode effects and produce a "telephone-equivalent" estimate for the ABS and panel data sources. The study found that ABS-web and RDD were similar for most estimates but exhibited differences for sensitive questions including getting drunk and experiencing depression. The opt-in web panel exhibited more differences between it and the other two survey modes. One notable example is the reporting of drinking alcohol at least 3-4 times per week, which was 21 percent for RDD-phone, 24 percent for ABS-web, and 34 percent for opt-in web panel. The regression model adjusts for mode effects, improving comparability with past surveys conducted by telephone; however, the models result in higher variance of the estimates. This method of adjusting for mode effects has broad applications to mode and sample transitions throughout the survey research industry.

Assessing the feasibility and sample quality of a national random-digit dialing cellular phone survey of young adults.

The majority of adults aged 18-34 years have only cellular phones, making random-digit dialing of landline telephones an obsolete methodology for surveillance of this population. However, 95% of this group has cellular phones. This article reports on the 2011 National Young Adult Health Survey (NYAHS), a pilot study conducted in the 50 US states and Washington, DC, that used random-digit dialing of cellular phones and benchmarked this methodology against that of the 2011 Behavioral Risk Factor Surveillance System (BRFSS). Comparisons of the demographic distributions of subjects in the NYAHS and BRFSS (aged 18-34 years) with US Census data revealed adequate reach for all demographic subgroups. After adjustment for design factors, the mean absolute deviations across demographic groups were 3 percentage points for the NYAHS and 2.8 percentage points for the BRFSS, nationally, and were comparable for each census region. Two-sided z tests comparing cigarette smoking prevalence revealed no significant differences between NYAHS and BRFSS participants overall or by subgroups. The design effects of the sampling weight were 2.09 for the NYAHS and 3.26 for the BRFSS. Response rates for the NYAHS and BRFSS cellular phone sampling frames were comparable. Our assessment of the NYAHS methodology found that random-digit dialing of cellular phones is a feasible methodology for surveillance of young adults.

Estimates of cigarette smoking from the NJ adult tobacco survey: real or spurious?

OBJECTIVE: To explore, post hoc, whether a large decline in smoking estimates between the 2005 and 2006 New Jersey Adult Tobacco Surveys is real or spurious given various methodological and environmental changes between the 2 time periods of data collection. METHODS: Using multiple data sources, we explored survey timing, poststratification approach, midinterview terminations, wireless substitution, and question order. RESULTS: Changes in question order were likely responsible for the majority of the unexpected decline in smoking prevalence; to a lesser degree, wireless substitution and midinterview terminations also likely contributed to an artificially exaggerated decline. CONCLUSION: Methodological changes can artificially affect trends in prevalence estimates.