Design and validation of instruments to measure knowledge.

BACKGROUND: Measuring health care providers' learning after they have participated in educational interventions that use experimental designs requires valid, reliable, and practical instruments. METHODS: A literature review was conducted. In addition, experience gained from designing and validating instruments for measuring the effect of an educational intervention informed this process. RESULTS: The eight main steps for designing, validating, and testing the reliability of instruments for measuring learning outcomes are presented. The key considerations and rationale for this process are discussed. Methods for critiquing and adapting existent instruments and creating new ones are offered. CONCLUSIONS: This study may help other investigators in developing valid, reliable, and practical instruments for measuring the outcomes of educational activities.

Instruments to measure cancer management knowledge of rural health care providers.

BACKGROUND: Instruments to measure cancer management knowledge of rural physicians, nurses, and pharmacists were needed to evaluate the effect of an educational intervention. Since such instruments did not exist, the authors designed and validated a new instrument for each discipline. METHODS: The design and validation process for these instruments are described. RESULTS: These three instruments were shown to be practical and to have high content and construct validity. Content validation demonstrated that all items were rated as essential or useful by 90% or more of the respondents. Construct validation show highly significant differences in mean scores among several levels of learners and practitioners as expected. CONCLUSIONS: These instruments may be useful to other investigators for measuring cancer management knowledge of rural physicians, nurses, and pharmacists.

Trimellitic anhydride-induced allergic response in the guinea pig lung involves antibody-dependent and -independent complement system activation.

Trimellitic anhydride (TMA) is one of many low molecular weight compounds known to cause occupational asthma. In our previous studies the TMA-induced allergic response in guinea pigs was attenuated by depletion of complement. Specifically, the leakage of red blood cells and infiltration of inflammatory cells into the lung after TMA challenge was significantly reduced. Thus, we hypothesize that in the presence of specific antibody, TMA activates the complement system and complement activation products play a role in mediating inflammatory cell infiltration into the lung and lung hemorrhage. Guinea pigs were sensitized by intradermal injection of TMA in corn oil. An increase in the complement activation product C3a was detected in bronchoalveolar lavage, but not in plasma, of both sensitized and nonsensitized guinea pigs after intratracheal challenge with TMA conjugated to GPSA (TMA-GPSA). In vitro experiments demonstrated that TMA-GPSA caused complement activation by antibody-dependent as well as antibody-independent pathways. In sensitized animals, TMA-GPSA challenge caused significant increases in eosinophils, neutrophils, and macrophages in lung, along with increases in red blood cells and protein in the airspace. The infiltration of eosinophils was unique in that the magnitude of the GPSA/TMA-GPSA effect was significantly different between nonsensitized and sensitized animals. C3a concentrations in BAL correlated with all measures of cell infiltration in sensitized animals, but not in nonsensitized animals. These data indicate that complement activation in the absence of antibody is not sufficient for the complete allergic response to occur. Both sensitization and the complement system are required for TMA-induced eosinophilia.

Lake Superior Rural Cancer Care Project, part I: an interventional trial.

PURPOSE: To date, effective cancer care and control intervention studies have been carried out largely in urban and suburban populations. This study was conducted to test innovative interventions, using experimental designs, to improve the care and outcomes of patients with cancer in rural settings. DESCRIPTION OF STUDY: The Lake Superior Rural Cancer Care Project (LSRCCP) tested an innovative, multimodal, multidisciplinary intervention that involved rural healthcare providers and their healthcare system. An experimental design was used, with the rural community as the unit of randomization. Outcomes were measured at three levels: rural providers' knowledge of cancer management, providers' practice performance, and patient outcomes. This 5-year study was conducted in rural areas of northern Minnesota, Wisconsin, and the western part of the Upper Peninsula of Michigan. RESULTS: Baseline data from the study are provided, and details of the design and methods are presented. The study outcomes are reported in part in "Lake Superior Rural Cancer Care Project Part II" in this issue and will be reported further in future issues. CLINICAL IMPLICATIONS: This article describes the hypotheses, design, and methods of the LSRCCP. The design and methods as well as the results of this study may be useful to cancer researchers and clinicians in rural areas across the United States.

Lake Superior Rural Cancer Care Project, part II: provider knowledge.

PURPOSE: The purpose of this article is to report the main learning outcomes of the Lake Superior Rural Cancer Care Project. DESCRIPTION OF STUDY: The authors designed and tested a multimodal intervention directed at rural providers and their healthcare systems in a large rural area in the north central United States. An experimental design was used to randomize rural providers at the group level. The intervention consisted of providing increased education for rural providers with a number of approaches, including the use of clinical opinion leaders. The main outcome of the intervention was knowledge scoring on discipline-specific cancer management tests. RESULTS: Knowledge scores for providers in the experimental group significantly increased from pretest to post-test: 66 to 79 for physicians (and physician assistants) (P=.02); 58 to 71 for nurses (P=.01); and 54 to 64 for pharmacists (P=.01). At post-test, participating providers in the experimental group performed significantly better on the knowledge tests (P <.01) than those in the control groups. CLINICAL IMPLICATIONS: This study may be the first to test educational interventions to improve rural providers' knowledge about cancer practice using an experimental design. The intervention may possibly change provider practice behaviors and, thus, patient outcomes, data that will be reported in a future issue. Finally, this educational intervention may prove useful for providers in other rural areas.

Accuracy of alternative approaches to capture-recapture estimates of disease frequency: internal validity analysis of data from five sources.

The authors used "internal validity analysis" to evaluate the performance of various capture-recapture methods. Data from studies with five overlapping, incomplete lists generated subgroups whose known sizes were compared with estimates derived from various four-source capture-recapture analyses. In 15 data sets unanalyzed previously (five subgroups of each of three new studies), the authors observed a trend toward mean underestimation of the known population size by 16-25%. (Coverage of the 90% confidence intervals associated with the method found to be optimal was acceptable (13/15), despite the downward bias.) The authors conjectured that (with the obvious exception of geographically disparate lists) most data sets used by epidemiologists tend to have a net positive dependence; that is, cases captured by one source are more likely to be captured by some other available source than are cases selected randomly from the population, and this trend results in a bias toward underestimation. Attempts to ensure that the underlying assumptions of the methods are met, such as minimizing (or adjusting adequately) for the possibility of loss due to death or migration, as was undertaken in one exceptional study, appear likely to improve the behavior of these methods.

Public health nurses' responses to domestic violence: a report from the Enhanced Domestic Abuse Intervention Project.

Public health nurses (PHNs) can play an important role in the detection of domestic violence. This study examines whether the introduction of a domestic violence assessment protocol by public health nurses in a maternal and child health visiting program increases the identification and referral rates of women experiencing domestic violence. Data collected from case files during the baseline year prior to the initiation of the protocol were compared to case file information after the protocol had been implemented. When the protocol was used, there was a higher rate of identification, although the difference was not statistically significant. Significantly more women, however, were provided with information about domestic violence resources after the protocol was in place, and significantly more women were referred to services in the second year after the protocol had been implemented. This study provides support for the use of a domestic violence protocol to improve the public health nursing response to domestic violence.

Recommendations for presentation and evaluation of capture-recapture estimates in epidemiology.

We propose 15 recommendations for approaches to capture-recapture analysis in epidemiology. We apply them to a report of such an analysis of a measles epidemic [McGilchrist et al., J Clinical Epidemiol 1996, 49: 293-296] and to comments thereon by R. C. Cormack [J Clinical Epidemiol 1999; 52: 909-914]. The latter challenged the utility of the data on the measles outbreak for any reliable capture-recapture estimates. We suggest that, adopting the perspective of W. Edwards Deming, one can only make judgments as to the reliability of capture-recapture data, methods, and derived estimates in the light of (i.e., conditional upon) their eventual intended use. Capture-recapture approaches "unreliable" from one perspective may be "reliable," and/or more appropriately, "useful" from another. We consider the utility of ancillary and ad hoc information that may be available or worth seeking to supplement a capture-recapture analysis. We use information within the study of McGilchrist et al. to illustrate how, with such ancillary information, one may overcome the main thrust of the objections of Cormack in situations in which one observes apparently anomalous or hard to understand data structures. Making certain simple assumptions we regard as plausible, we estimate the number of affected in the measles epidemic as between about 700-1300. We derive this from data on 502 cases in a Register, an ad hoc sample of 91 cases in one age group in the general population, and the report of 41 cases in both of these. Our result is only 15-30% the total implied by the estimates McGilchrist et al. derived with more complex methods and many assumptions in addition to our own. We discuss various approaches to evaluating "reliability" of our estimate conditional upon intended uses by policy makers.

The obese gene is expressed in lean littermates of the genetically obese mouse (C57BL/6J ob/ob).

Some individuals of the mixed group of "lean" littermates (+/ob and +/+) of (C57BL/6J ob/ob) often suggest phenotypic characteristics of ob/ob animals. Therefore, it was of interest to determine whether expression of the ob gene had physiological significance in +/ob animals. Body weight (BW), fasting blood glucose (FBG), and body core temperature (Tr) were monitored between 62 and 364 days of age in +/+ and +/ob mice. Among females but not males, +/ob mice were heavier (P = 0.003) and FBG levels were greater (P = 0.04) than in +/+ animals. Comparison of Tr indicated differences suggesting falling Tr in +/ob but rising Tr in +/+ mice with age in males but not females. Multivariate analysis of variance yielded genotype effects for both males (P = 0.002) and females (P = 0.02). BW, FBG, and Tr alone were sufficient at the 75% level for genotypic characterization and separation of +/? animals as +/ob or +/+; clearly, expression of the ob gene in heterozygotes of the +/ob animal may make the mixed +/? group inappropriate as lean controls.

An exact test for all-way interaction in a 2M contingency table: application to interval capture-recapture estimation of population size.

An exact conditional test for an M-way log-linear interaction in a fully observed 2M contingency table is formulated. From this is derived a procedure for interval estimation of the total count N in a 2M contingency table, one of whose entries is unobserved. This procedure has an immediate application to interval estimation of the size of a closed population from incomplete, overlapping lists of records, as in capture-recapture analysis of epidemiological data. Data on the prevalence of spina bifida in live births in upstate New York in 1969-1974 illustrate this application.

Marginal versus conditional versus 'structural source' models: a rationale for an alternative to log-linear methods for capture-recapture estimates.

Log-linear models for capture-recapture type data are widely used for estimating sizes of populations. Log-linear methods model conditional interactions between the sources. Often, however, the marginal associations are more appropriate and easier for the practitioner to conceptualize. Analyses here of previously published data on cases of spina bifida in upstate New York are used to show how the assumption that sources are conditionally independent can give biased estimates if in fact the sources are marginally independent. A plausible model for the structural sources of interactions between the sources of information about spina bifida cases is developed which implies marginal independence of two of the sources rather than conditional independence. Estimates of the population total based on marginal independence are derived and give larger estimates of the population total than those derived based upon conditional dependence. When investigators can in fact model the likely underlying relationships of the sources in the population, we suggest considering modelling the potential interdependencies of the sources, which we term 'structural source modeling'.

Validity of methods for model selection, weighting for model uncertainty, and small sample adjustment in capture-recapture estimation.

In log-linear capture-recapture approaches to population size, the method of model selection may have a major effect upon the estimate. In addition, the estimate may also be very sensitive if certain cells are null or very sparse, even with the use of multiple sources. The authors evaluated 1) various approaches to the issue of model uncertainty and 2) a small sample correction for three or more sources recently proposed by Hook and Regal. The authors compared the estimates derived using 1) three different information criteria that included Akaike's Information Criterion (AIC) and two alternative formulations of the Bayesian Information Criterion (BIC), one proposed by Draper ("two pi") and one by Schwarz ("not two pi"); 2) two related methods of weighting estimates associated with models; 3) the independent model; and 4) the saturated model, with the known totals in 20 different populations studied by five separate groups of investigators. For each method, we also compared the estimate derived with or without the proposed small sample correction. At least in these data sets, the use of AIC appeared on balance to be preferable. The BIC formulation suggested by Draper appeared slightly preferable to that suggested by Schwarz. Adjustment for model uncertainty appears to improve results slightly. The proposed small sample correction appeared to diminish relative log bias but only when sparse cells were present. Otherwise, its use tended to increase relative log bias. Use of the saturated model (with or without the small sample correction) appears to be optimal if the associated interval is not uselessly large, and if one can plausibly exclude an all-source interaction. All other approaches led to an estimate that was too low by about one standard deviation.

Internal validity analysis: a method for adjusting capture-recapture estimates of prevalence.

The authors propose a method for adjusting results of log-linear multi-source capture-recapture estimates of total population. The method compares the totals in some subpopulations of known size with estimates derived from various capture-recapture approaches to these subpopulations. The authors term such an approach an "internal validity analysis". Trends in the ratios of the estimates to the known true values of these subpopulations provide a plausible indicator of the bias of some types of estimates of the total population especially when underlying assumptions of the methods used have not been met in analysis of the total population. The authors apply this method to published data on an open population of injection drug users that had been previously analyzed with a standard capture-recapture analysis as if it were a closed population. Internal validity analysis suggests that the size of this population is about 15% greater than that previously estimated.

Relation of the crude relative risk of a disorder to relative risks in strata of a "susceptible" variable.

If tau d is the relative risk of a disorder in the entire population, and tau d,i is the relative risk of the disorder in i = 1, ... m strata, then one may show readily that tau d = sigma ci tau d,i, where ci is the product of two terms tau i, the risk ratio of being in the ith stratum, and pi,unexp d, the proportion of those with the disorder and unexposed who are in the ith stratum. This formulation is of primary interest in epidemiology when relative risks are available on one or only some strata of a variable that itself may be affected by exposure (what one may define as a "susceptible" covariate) such as mortality or hospitalization. Although relative risks within strata of such a variable may be of some intrinsic clinical interest, only the risk ratio unstratified on such a variate may be pertinent to a causal effect (unlike the case for nonsusceptible variables such as sex, age, etc). In some instances, as for birth defects, one may have data from a few strata or only one (for example, livebirths) of a susceptible covariate (for example, conceptus viability). But one may still be able to draw useful inferences about tau d, the risk ratio in the entire population, because if tau d,i > or = 1/ci (or k/ci), one may conclude that tau d is, at least, greater than 1.0 (or k). Similarly, a study of a disorder limited to hospitalized cases and controls may enable investigators to infer, using the same criterion, a positive association in the entire population despite the presence of hospitalization bias of the type described by Berkson.

Effect of variation in probability of ascertainment by sources ("variable catchability") upon "capture-recapture" estimates of prevalence.

Capture-recapture methods in epidemiology analyze data from overlapping lists of cases from various sources of ascertainment to generate estimates of missing cases and the total affected. Applications of these methods usually recognize the possibility of, and attempt to adjust for, nonindependent ascertainment by the various sources used. However, separate from the issue of dependencies between sources is the complexity of within source variation in probability of ascertainment of cases, e.g., variation in ascertainment by population subgroups, such as socioeconomic classes, races, or other subdivisions. The authors present a general approach to this issue for the two-source case that takes account of not only biases that arise from such "variable catchability" within sources but also the separate complexity of dependencies between sources. A general formula, (K - delta)/(K + delta), is derived that allows simultaneous calculation of the effects of variable catchability, delta, and source dependencies, delta, upon the accuracy of the two-source estimate. The effect of variable catchability upon accuracy and applications to data by race on the neurodegenerative disorder, Huntington's disease, are presented. In the latter analysis, multiple different two-source estimates of prevalence were made, considering each source versus all others pooled. Most of the likely bias was found to be due to source dependencies; variable catchability contributed relatively little bias. Multiple poolings of all but one source may prove a generally efficient method for overcoming the problem of likely variable catchability, at least when there are data from many distinct sources.

Representative and misrepresentative associations of birth defects in livebirths. Conditions under which relative risks greater than unity in livebirths necessarily imply relative risks greater than unity in all conceptuses.

One may express the relative risk of defect in all conceptuses, r(def), as a function of the relative risk of defect in livebirths, r(def,lb), and in embryonic and fetal deaths, r(def,efd), as r(def) = C(lb)r(def,lb) + C(efd)r(def,efd), where C(lb) and C(efd) are coefficients defined in terms of conceptus and defect viability and lethality. If the relative risk of birth defect in livebirths, r(def,lb), is greater than unity, but the relative risk of defect in all conceptuses, r(def), is equal to or less than unity (or the reverse pattern holds), then the relative risk of defects in livebirths may be said to be "distorting" or "misrepresentative" because it does not reflect the nature of the association in all conceptuses. The authors define and present an explicit expression for a boundary upon the relative risk of defect in livebirths. If the relative risk of defect in livebirths is (validly) greater than this boundary value, then the relative risk in all conceptuses must be greater than unity and the observed relative risk of defect in births is "representative" and not distorting. The authors show that the boundary value is equal to 1/C(lb), where C(lb) is a simple function of the lethality of all unexposed conceptuses, the lethality of unexposed conceptuses with defect, and the relative risk of any embryonic and fetal death. Tables of the boundary relative risk for various values of these variables are presented. Over a very wide range of reference variables, a (valid) relative risk of defect in livebirths of 3.5 or greater implies a positive association with defect in all (recognized) conceptuses in the population studied.

The value of capture-recapture methods even for apparent exhaustive surveys. The need for adjustment for source of ascertainment intersection in attempted complete prevalence studies.

Almost all reported prevalence studies of which we are aware make exhaustive attempts to find diagnosed individuals and report all affected individuals, but make no attempt to estimate or adjust for missing cases. Yet very simple methods introduced in the planning stage of a prevalence study may enable investigators, or at least those subsequently reading their reports, to derive such adjusted estimates. If investigators keep track of the nature of the ascertainment of cases by source and collect and report data that allow calculation of the number of cases by source intersection, then they, or at least others, may derive estimates of missing cases and of the total population affected, by using readily available analogues of capture-recapture methods developed for wildlife populations censuses. Unfortunately, such methods are often inappropriately disparaged or ignored by epidemiologists. The derived estimates are sensitive to assumptions about dependence or independence ("interaction") of various sources, assumptions that sometimes are unprovable, and these estimates have some uncertainty because of statistical fluctuation. Moreover, most investigators who attempt exhaustive prevalence studies apparently believe that they have ascertained all cases and that there is no need to attempt to adjust for, let alone provide data pertinent to, the number of missing cases or to use a statistical method that will at best imply a certain imprecision to their result. Yet a survey that reports prevalence data without adjustment for, or data on, source intersection in essence makes an estimate of missing cases--zero--while providing no quantitative grounds for that claim. The results of all such surveys should be regarded with skepticism because, at best (if the case reports are accurate), they provide only a lower boundary of prevalence. We illustrate the grounds for these views by analyzing data from an apparently exhaustive prevalence study that used at least 14 distinct sources for ascertainment, including advertising, to find cases. Available limited data on source intersection provided in the report enable the plausible inference that the study missed about 25-40% of cases. We urge that no attempted complete prevalence studies be presented without data on ascertainment by source intersection.