QUAID: a questionnaire evaluation aid for survey methodologists.

QUAID (question-understanding aid) is a software tool that assists survey methodologists, social scientists, and designers of questionnaires in improving the wording, syntax, and semantics of questions. The tool identifies potential problems that respondents might have in comprehending the meaning of questions on questionnaires. These problems can be scrutinized by researchers when they revise questions to improve question comprehension and, thereby, enhance the reliability and validity of answers. QUAID was designed to identify nine classes of problems, but only five of these problems are addressed in this article: unfamiliar technical term, vague or imprecise relative term, vague or ambiguous noun phrase, complex syntax, and working memory overload. We compared the output of QUAID with ratings of language experts who evaluated a corpus of questions on the five classes of problems. The corpus consisted of 505 questions on 11 surveys developed by the U.S. Census Bureau. Analyses of hit rates, false alarm rates, d' scores, recall scores, and precision scores revealed that QUAID was able to identify these five problems with questions, although improvements in QUAID's performance are anticipated in future research and development.

Verification of statements about story worlds that deviate from normal conceptions of time: what is true about Einstein's Dreams?

College students read chapters from a novel written by Alan Lightman (Einstein's Dreams) and later provided verification judgments on the truth/falsity of test statements. Each chapter described a different fictional village that incorporated assumptions about time that deviate from our normal TIME schema, e.g., citizens knowing exactly when the world will end, time flowing backward instead of forward. These novel assumptions about time provided interesting insights about life and reality. In two experiments, we examined whether readers could accurately incorporate these novel assumptions about time in the fictional story worlds, as manifested in the verification judgments for statements after story comprehension. The test statements included verbatim typical, verbatim atypical, inference typical, and inference atypical information from the perspective of mundane reality that meshes with a normal TIME schema. Verification ratings were collected on a 6-point scale in Experiment 1, whereas Experiment 2 used a signal-response technique in which binary true/false decisions were extracted at-.5, 1.5, 3.5, 5.5, and 10.0 s. The college students were measured on literary expertise, reading skill, working memory span, and reading time. Readers with comparatively high literary expertise showed truth discrimination scores that were compatible with a schema copy plus tag model, which assumes that readers are good at detecting and remembering atypical verbatim information; this model predicts better (and faster) truth discrimination for verbatim atypical statements than for verbatim typical statements. In contrast, fast readers with comparatively low literary expertise were compatible with a filtering model; this model predicts that readers gloss over (or suppress) atypical verbatim information and show advantages for verbatim typical information. All groups of readers had trouble inferentially propagating the novel assumptions about time in a fictional story world, but the slower readers were more accurate in their verification of the atypical inferences. A construction-integration model could explain the interactions among literary expertise, reading time, and the typicality of test statements.