Identifying Young Adults at High Risk for Weight Gain Using Machine Learning.

INTRODUCTION: Weight gain among young adults continues to increase. Identifying adults at high risk for weight gain and intervening before they gain weight could have a major public health impact. Our objective was to develop and test electronic health record-based machine learning models to predict weight gain in young adults with overweight/class 1 obesity. METHODS: Seven machine learning models were assessed, including three regression models, random forest, single-layer neural network, gradient-boosted decision trees, and support vector machine (SVM) models. Four categories of predictors were included: 1) demographics; 2) obesity-related health conditions; 3) laboratory data and vital signs; and 4) neighborhood-level variables. The cohort was split 60:40 for model training and validation. Area under the receiver operating characteristic curves (AUC) were calculated to determine model accuracy at predicting high-risk individuals, defined by ≥ 10% total body weight gain within 2 y. Variable importance was measured via generalized analysis of variance procedures. RESULTS: Of the 24,183 patients (mean [SD] age, 32.0 [6.3] y; 55.1% females) in the study, 14.2% gained ≥10% total body weight. Area under the receiver operating characteristic curves varied from 0.557 (SVM) to 0.675 (gradient-boosted decision trees). Age, sex, and baseline body mass index were the most important predictors among the models except SVM and neural network. CONCLUSIONS: Our machine learning models performed similarly and had modest accuracy for identifying young adults at risk of weight gain. Future models may need to incorporate behavioral and/or genetic information to enhance model accuracy.

Association between neighborhood food environments and bariatric surgery outcomes.

BACKGROUND: Individual characteristics associated with weight loss after bariatric surgery are well established, but the neighborhood characteristics that influence outcomes are unknown. OBJECTIVES: The objective of this study was to determine if neighborhood characteristics, including social determinants and lifestyle characteristics, were associated with weight loss after bariatric surgery. SETTING: Single university healthcare system, United States. METHODS: In this retrospective cohort study, all patients who underwent primary bariatric surgery from 2008 to 2017 and had at least 1 year of follow-up data were included. Patient-level demographics and neighborhood-level social determinants (area deprivation index, urbanicity, and walkability) and lifestyle factors (organic food use, fresh fruit/vegetable consumption, diet to maintain weight, soda consumption, and exercise) were analyzed. Median regression with percent total body weight (%TBW) loss as the outcome was applied to examine factors associated with weight loss after surgery. RESULTS: Of the 647 patients who met inclusion criteria, the average follow-up period was 3.1 years, and the mean %TBW loss at the follow-up was 22%. In adjusted median regression analyses, Roux-en-Y gastric bypass was associated with greater %TBW loss (11.22%, 95% confidence interval [8.96, 13.48]) compared to sleeve, while longer follow-up time (-2.42% TBW loss per year, 95% confidence interval [-4.63, -0.20]) and a preoperative diagnosis of diabetes (-1.00% TBW loss, 95% confidence interval [-1.55, -0.44]) were associated with less. None of the 8 neighborhood level characteristics was associated with weight loss. CONCLUSIONS: Patient characteristics rather than neighborhood-level social determinants and lifestyle factors were associated with weight loss after bariatric surgery in our cohort of bariatric surgery patients. Patients from socioeconomically deprived neighborhoods can achieve excellent weight loss after bariatric surgery.

Obesity and "obesity-related" cancers: are there body mass index cut-points?

BACKGROUND: Despite compelling links between excess body weight and cancer, body mass index (BMI) cut-points, or thresholds above which cancer incidence increased, have not been identified. The objective of this study was to determine if BMI cut-points exist for 14 obesity-related cancers. SUBJECTS/METHODS: In this retrospective cohort study, patients 18-75 years old were included if they had ≥2 clinical encounters with BMI measurements in the electronic health record (EHR) at a single academic medical center from 2008 to 2018. Patients who were pregnant, had a history of cancer, or had undergone bariatric surgery were excluded. Adjusted logistic regression was performed to identify cancers that were associated with increasing BMI. For those cancers, BMI cut-points were calculated using adjusted quantile regression for cancer incidence at 80% sensitivity. Logistic and quantile regression models were adjusted for age, sex, race/ethnicity, and smoking status. RESULTS: A total of 7079 cancer patients (mean age 58.5 years, mean BMI 30.5 kg/m2) and 270,441 non-cancer patients (mean age 43.8 years, mean BMI 28.8 kg/m2) were included in the study. In adjusted logistic regression analyses, statistically significant associations were identified between increasing BMI and the incidence of kidney, thyroid, and uterine cancer. BMI cut-points were identified for kidney (26.3 kg/m2) and uterine (26.9 kg/m2) cancer. CONCLUSIONS: BMI cut-points that accurately predicted development kidney and uterine cancer occurred in the overweight category. Analysis of multi-institutional EHR data may help determine if these relationships are generalizable to other health care settings. If they are, incorporation of BMI into the screening algorithms for these cancers may be warranted.

Simultaneous spatial smoothing and outlier detection using penalized regression, with application to childhood obesity surveillance from electronic health records.

Electronic health records (EHRs) have become a platform for data-driven granular-level surveillance in recent years. In this paper, we make use of EHRs for early prevention of childhood obesity. The proposed method simultaneously provides smooth disease mapping and outlier information for obesity prevalence that are useful for raising public awareness and facilitating targeted intervention. More precisely, we consider a penalized multilevel generalized linear model. We decompose regional contribution into smooth and sparse signals, which are automatically identified by a combination of fusion and sparse penalties imposed on the likelihood function. In addition, we weigh the proposed likelihood to account for the missingness and potential nonrepresentativeness arising from the EHR data. We develop a novel alternating minimization algorithm, which is computationally efficient, easy to implement, and guarantees convergence. Simulation studies demonstrate superior performance of the proposed method. Finally, we apply our method to the University of Wisconsin Population Health Information Exchange database.

Association Between Medicaid Status, Social Determinants of Health, and Bariatric Surgery Outcomes.

OBJECTIVE: To compare outcomes after bariatric surgery between Medicaid and non-Medicaid patients and assess whether differences in social determinants of health were associated with postoperative weight loss. BACKGROUND: The literature remains mixed on weight loss outcomes and healthcare utilization for Medicaid patients after bariatric surgery. It is unclear if social determinants of health geocoded at the neighborhood level are associated with outcomes. METHODS: Patients who underwent laparoscopic sleeve gastrectomy (SG) or Roux-en-Y gastric bypass (RYGB) from 2008 to 2017 and had ≥1 year of follow-up within a large health system were included. Baseline characteristics, 90-day and 1-year outcomes, and weight loss were compared between Medicaid and non-Medicaid patients. Area deprivation index (ADI), urbanicity, and walkability were analyzed at the neighborhood level. Median regression with percent total body weight (TBW) loss as the outcome was used to assess predictors of weight loss after surgery. RESULTS: Six hundred forty-seven patients met study criteria (191 Medicaid and 456 non-Medicaid). Medicaid patients had a higher 90-day readmission rate compared to non-Medicaid patients (19.9% vs 12.3%, P < 0.016). Weight loss was similar between Medicaid and non-Medicaid patients (23.1% vs 21.9% TBW loss, respectively; P = 0.266) at a median follow-up of 3.1 years. In adjusted analyses, Medicaid status, ADI, urbanicity, and walkability were not associated with weight loss outcomes. CONCLUSIONS: Medicaid status and social determinants of health at the neighborhood level were not associated with weight loss outcomes after bariatric surgery. These findings suggest that if Medicaid patients are appropriately selected for bariatric surgery, they can achieve equivalent outcomes as non-Medicaid patients.

Weight Loss for Patients With Obesity: An Analysis of Long-Term Electronic Health Record Data.

BACKGROUND: Numerous studies have reported that losing as little as 5% of one's total body weight (TBW) can improve health, but no studies have used electronic health record data to examine long-term changes in weight, particularly for adults with severe obesity [body mass index (BMI) ≥35 kg/m]. OBJECTIVE: To measure long-term weight changes and examine their predictors for adults in a large academic health care system. RESEARCH DESIGN: Observational study. SUBJECTS: We included 59,816 patients aged 18-70 years who had at least 2 BMI measurements 5 years apart. Patients who were underweight, pregnant, diagnosed with cancer, or had undergone bariatric surgery were excluded. MEASURES: Over a 5-year period: (1) ≥5% TBW loss; (2) weight loss into a nonobese BMI category (BMI <30 kg/m); and (3) predictors of %TBW change via quantile regression. RESULTS: Of those with class 2 or 3 obesity, 24.2% and 27.8%, respectively, lost at least 5% TBW. Only 3.2% and 0.2% of patients with class 2 and 3 obesity, respectively, lost enough weight to attain a BMI <30 kg/m. In quantile regression, the median weight change for the population was a net gain of 2.5% TBW. CONCLUSIONS: Although adults with severe obesity were more likely to lose at least 5% TBW compared with overweight patients and patients with class 1 obesity, sufficient weight loss to attain a nonobese weight class was very uncommon. The pattern of ongoing weight gain found in our study population requires solutions at societal and health systems levels.

Circannual growth in Wisconsin children and adolescents: Identifying optimal periods of obesity prevention.

BACKGROUND: Recent studies suggest kids tend to gain the most weight in summer, but schools are chastised for supporting obesogenic environments. Conclusions on circannual weight gain are hampered by infrequent body mass index (BMI) measurements, and guidance is limited on the optimal timeframe for paediatric weight interventions. OBJECTIVES: This study characterized circannual trends in BMI in Wisconsin children and adolescents and identified sociodemographic differences in excess weight gain. METHODS: An observational study was used to pool data from 2010 to 2015 to examine circannual BMI z-score trends for Marshfield Clinic patients age 3 to 17 years. Daily 0.20, 0.50, and 0.80 quantiles of BMI z-score were estimated, stratified by gender, race, and age. RESULTS: BMI z-scores increased July to September, followed by a decrease in October to December, and another increase to decrease cycle beginning in February. For adolescents, the summer increase in BMI was greater among those in the upper BMI z-score quantile relative to those in the lower quantile (+0.15 units vs +0.04 units). This pattern was opposite in children. CONCLUSIONS: BMI increased most rapidly in late summer. This growth persisted through autumn in adolescents who were larger, suggesting weight management support may be beneficial for kids who are overweight at the start of the school year.

Patient Engagement and Attitudes Toward Using the Electronic Medical Record for Medical Research: The 2015 Greater Plains Collaborative Health and Medical Research Family Survey.

BACKGROUND: Electronic health records (EHRs) are ubiquitous. Yet little is known about the use of EHRs for prospective research purposes, and even less is known about patient perspectives regarding the use of their EHR for research. OBJECTIVE: This paper reports results from the initial obesity project from the Greater Plains Collaborative that is part of the Patient-Centered Outcomes Research Institute's National Patient-Centered Clinical Research Network (PCORNet). The purpose of the project was to (1) assess the ability to recruit samples of adults of child-rearing age using the EHR; (2) prospectively assess the willingness of adults of child-rearing age to participate in research, and their willingness (if parents) to have their children participate in medical research; and (3) to assess their views regarding the use of their EHRs for research. METHODS: The EHRs of 10 Midwestern academic medical centers were used to select patients. Patients completed a survey that was designed to assess patient willingness to participate in research and their thoughts about the use of their EHR data for research. The survey included questions regarding interest in medical research, as well as basic demographic and health information. A variety of contact methods were used. RESULTS: A cohort of 54,269 patients was created, and 3139 (5.78%) patients responded. Completers were more likely to be female (53.84%) and white (85.84%). These and other factors differed significantly by site. Respondents were overwhelmingly positive (83.9%) about using EHRs for research. CONCLUSIONS: EHRs are an important resource for engaging patients in research, and our respondents concurred. The primary limitation of this work was a very low response rate, which varied by the method of contact, geographic location, and respondent characteristics. The primary strength of this work was the ability to ascertain the clinically observed characteristics of nonrespondents and respondents to determine factors that may contribute to participation, and to allow for the derivation of reliable study estimates for weighting responses and oversampling of difficult-to-reach subpopulations. These data suggest that EHRs are a promising new and effective tool for patient-engaged health research.

Electronic Health Record Data Versus the National Health and Nutrition Examination Survey (NHANES): A Comparison of Overweight and Obesity Rates.

BACKGROUND: Estimating population-level obesity rates is important for informing policy and targeting treatment. The current gold standard for obesity measurement in the United States-the National Health and Nutrition Examination Survey (NHANES)-samples <0.1% of the population and does not target state-level or health system-level measurement. OBJECTIVE: To assess the feasibility of using body mass index (BMI) data from the electronic health record (EHR) to assess rates of overweight and obesity and compare these rates to national NHANES estimates. RESEARCH DESIGN: Using outpatient data from 42 clinics, we studied 388,762 patients in a large health system with at least 1 primary care visit in 2011-2012. MEASURES: We compared crude and adjusted overweight and obesity rates by age category and ethnicity (white, black, Hispanic, Other) between EHR and NHANES participants. Adjusted overweight (BMI≥25) and obesity rates were calculated by a 2-step process. Step 1 accounted for missing BMI data using inverse probability weighting, whereas step 2 included a poststratification correction to adjust the EHR population to a nationally representative sample. RESULTS: Adjusted rates of obesity (BMI≥30) for EHR patients were 37.3% [95% confidence interval (95% CI), 37.1-37.5] compared with 35.1% (95% CI, 32.3-38.1) for NHANES patients. Among the 16 different obesity class, ethnicity, and sex strata that were compared between EHR and NHANES patients, 14 (87.5%) contained similar obesity estimates (ie, overlapping 95% CIs). CONCLUSIONS: EHRs may be an ideal tool for identifying and targeting patients with obesity for implementation of public health and/or individual level interventions.

Using Electronic Health Records to Examine Disease Risk in Small Populations: Obesity Among American Indian Children, Wisconsin, 2007-2012.

INTRODUCTION: Tribe-based or reservation-based data consistently show disproportionately high obesity rates among American Indian children, but little is known about the approximately 75% of American Indian children living off-reservation. We examined obesity among American Indian children seeking care off-reservation by using a database of de-identified electronic health records linked to community-level census variables. METHODS: Data from electronic health records from American Indian children and a reference sample of non-Hispanic white children collected from 2007 through 2012 were abstracted to determine obesity prevalence. Related community-level and individual-level risk factors (eg, economic hardship, demographics) were examined using logistic regression. RESULTS: The obesity rate for American Indian children (n = 1,482) was double the rate among non-Hispanic white children (n = 81,042) (20.0% vs 10.6%, P < .001). American Indian children were less likely to have had a well-child visit (55.9% vs 67.1%, P < .001) during which body mass index (BMI) was measured, which may partially explain why BMI was more likely to be missing from American Indian records (18.3% vs 14.6%, P < .001). Logistic regression demonstrated significantly increased obesity risk among American Indian children (odds ratio, 1.8; 95% confidence interval, 1.6-2.1) independent of age, sex, economic hardship, insurance status, and geographic designation. CONCLUSION: An electronic health record data set demonstrated high obesity rates for nonreservation-based American Indian children, rates that had not been previously assessed. This low-cost method may be used for assessing health risk for other understudied populations and to plan and evaluate targeted interventions.

Roles of Clinician, Patient, and Community Characteristics in the Management of Pediatric Upper Respiratory Tract Infections.

PURPOSE: Prior studies have evaluated factors predictive of inappropriate antibiotic prescription for upper respiratory tract infections (URIs). Community factors, however, have not been examined. The aim of this study was to evaluate the roles of patient, clinician, and community factors in predicting appropriate management of URIs in children. METHODS: We used a novel database exchange, linking electronic health record data with community statistics, to identify all patients aged 3 months to 18 years in whom URI was diagnosed in the period from 2007 to 2012. We followed the Healthcare Effectiveness Data and Information Set (HEDIS) quality measurement titled "Appropriate treatment for children with upper respiratory infection" to determine the rate of appropriate management of URIs. We then stratified data across individual and community characteristics and used multiple logistic regression modeling to identify variables that independently predicted antibiotic prescription. RESULTS: Of 20,581 patients, the overall rate for appropriate management for URI was 93.5%. Family medicine clinicians (AOR = 1.5; 95% CI 1.31, 1.71; reference = pediatric clinicians), urgent care clinicians (AOR = 2.23; 95% CI 1.93, 2.57; reference = pediatric clinicians), patients aged 12 to 18 years (AOR = 1.44; 95% CI 1.25, 1.67; reference = age 3 months to 4 years), and patients of white race/ ethnicity (AOR = 1.83; 95% CI 1.41, 2.37; reference = black non-Hispanic) were independently predictive of antibiotic prescription. No community factors were independently predictive of antibiotic prescription. CONCLUSIONS: Results correlate with prior studies in which non-pediatric clinicians and white race/ethnicity were predictive of antibiotic prescription, while association with older patient age has not been previously reported. Findings illustrate the promise of linking electronic health records with community data to evaluate health care disparities.

Electronic health records and community health surveillance of childhood obesity.

BACKGROUND: Childhood obesity remains a public health concern, and tracking local progress may require local surveillance systems. Electronic health record data may provide a cost-effective solution. PURPOSE: To demonstrate the feasibility of estimating childhood obesity rates using de-identified electronic health records for the purpose of public health surveillance and health promotion. METHODS: Data were extracted from the Public Health Information Exchange (PHINEX) database. PHINEX contains de-identified electronic health records from patients primarily in south central Wisconsin. Data on children and adolescents (aged 2-19 years, 2011-2012, n=93,130) were transformed in a two-step procedure that adjusted for missing data and weighted for a national population distribution. Weighted and adjusted obesity rates were compared to the 2011-2012 National Health and Nutrition Examination Survey (NHANES). Data were analyzed in 2014. RESULTS: The weighted and adjusted obesity rate was 16.1% (95% CI=15.8, 16.4). Non-Hispanic white children and adolescents (11.8%, 95% CI=11.5, 12.1) had lower obesity rates compared to non-Hispanic black (22.0%, 95% CI=20.7, 23.2) and Hispanic (23.8%, 95% CI=22.4, 25.1) patients. Overall, electronic health record-derived point estimates were comparable to NHANES, revealing disparities from preschool onward. CONCLUSIONS: Electronic health records that are weighted and adjusted to account for intrinsic bias may create an opportunity for comparing regional disparities with precision. In PHINEX patients, childhood obesity disparities were measurable from a young age, highlighting the need for early intervention for at-risk children. The electronic health record is a cost-effective, promising tool for local obesity prevention efforts.

Sparse modeling of spatial environmental variables associated with asthma.

Geographically distributed environmental factors influence the burden of diseases such as asthma. Our objective was to identify sparse environmental variables associated with asthma diagnosis gathered from a large electronic health record (EHR) dataset while controlling for spatial variation. An EHR dataset from the University of Wisconsin's Family Medicine, Internal Medicine and Pediatrics Departments was obtained for 199,220 patients aged 5-50years over a three-year period. Each patient's home address was geocoded to one of 3456 geographic census block groups. Over one thousand block group variables were obtained from a commercial database. We developed a Sparse Spatial Environmental Analysis (SASEA). Using this method, the environmental variables were first dimensionally reduced with sparse principal component analysis. Logistic thin plate regression spline modeling was then used to identify block group variables associated with asthma from sparse principal components. The addresses of patients from the EHR dataset were distributed throughout the majority of Wisconsin's geography. Logistic thin plate regression spline modeling captured spatial variation of asthma. Four sparse principal components identified via model selection consisted of food at home, dog ownership, household size, and disposable income variables. In rural areas, dog ownership and renter occupied housing units from significant sparse principal components were associated with asthma. Our main contribution is the incorporation of sparsity in spatial modeling. SASEA sequentially added sparse principal components to Logistic thin plate regression spline modeling. This method allowed association of geographically distributed environmental factors with asthma using EHR and environmental datasets. SASEA can be applied to other diseases with environmental risk factors.

Estimating Wisconsin asthma prevalence using clinical electronic health records and public health data.

OBJECTIVES: We compared a statewide telephone health survey with electronic health record (EHR) data from a large Wisconsin health system to estimate asthma prevalence in Wisconsin. METHODS: We developed frequency tables and logistic regression models using Wisconsin Behavioral Risk Factor Surveillance System and University of Wisconsin primary care clinic data. We compared adjusted odds ratios (AORs) from each model. RESULTS: Between 2007 and 2009, the EHR database contained 376,000 patients (30,000 with asthma), and 23,000 (1850 with asthma) responded to the Behavioral Risk Factor Surveillance System telephone survey. AORs for asthma were similar in magnitude and direction for the majority of covariates, including gender, age, and race/ethnicity, between survey and EHR models. The EHR data had greater statistical power to detect associations than did survey data, especially in pediatric and ethnic populations, because of larger sample sizes. CONCLUSIONS: EHRs can be used to estimate asthma prevalence in Wisconsin adults and children. EHR data may improve public health chronic disease surveillance using high-quality data at the local level to better identify areas of disparity and risk factors and guide education and health care interventions.

The theory and application of UW ehealth-PHINEX, a clinical electronic health record-public health information exchange.

BACKGROUND: Electronic health records (EHRs) hold the promise of improving clinical quality and population health while reducing health care costs. However, it is not clear how these goals can be achieved in practice. METHODS: Clinician-led teams developed EHR data extracts to support chronic disease use cases. EHRs were linked with community-level data to describe disease prevalence and health care quality at the patient, health care system, and community risk factor levels. Software was developed and statistical modeling included multivariate, mixed-model, longitudinal, data mining, and geographic information system (GIS)/spatial regression approaches. RESULTS: A HIPAA-compliant limited data set was created on 192,201 patients seen in University of Wisconsin Family Medicine clinics throughout Wisconsin in 2007-2009. It was linked to a commercially available database of approximately 6000 variables describing community-level risk factors at the census block group. Areas of increased asthma and diabetes prevalence have been mapped, identified, and compared to economic hardship. CONCLUSIONS: A comprehensive framework has been developed for clinical-public health data exchange to develop new evidence and apply it to clinical practice and health policy. EHR data at the neighborhood level can be used for future population studies and may enhance understanding of community-level patterns of illness and care.

Medical informatics in population health: building Wisconsin's strategic framework for health information technology.

Medicine is increasingly practiced through the application of information sciences. Medical informatics deals with optimal information use within bioinformatics, imaging, clinical, and population health domains. Population health informatics plays an important role in that it critically informs practice in each of the other domains. Proper functioning of health care systems requires an advanced health information network that supports clinical care, personal health management, population health, and research. But this infrastructure does not yet exist in the United States. A number of federal initiatives are underway to address this problem, including the development of a framework for a national health information network and funding for implementation. This network will be facilitated by federal leadership, but public and private partnerships, and state, regional, and local implementation and policy development will play a critical role. In this article, we describe several Wisconsin initiatives that are keys to developing a strategic framework and building the state's electronic health information infrastructure.

Relationship of serum levels of individual PCB, dioxin, and furan congeners and DDE with Great Lakes sport-caught fish consumption.

Great Lakes (GL) sport fish consumption is a potential route of exposure for environmentally persistent organochlorine contaminants, which may have human health effects. In this report, relationships are explored among individual congeners in a large cohort of frequent and infrequent GL sport fish consumers. Blood samples were obtained in 1993-1995 and analyzed for 1,1-bis(4-chlorophenyl)-2,2-dichloroethene (DDE) and for 62 noncoplanar polychlorinated biphenyls (PCBs), 4 coplanar PCBs, 8 polychlorinated dibenzo-p-dioxin (dioxin), and10 dibenzofuran (furan) congeners. All GL fish eaters and referents had detectable levels of DDE, total noncoplanar PCBs, total coplanar PCBs, total dioxins, and total furans. Noncoplanar PCBs were higher in GL sport fish consumers than in a referent population from the same geographic area, were associated with GL sport-caught fish (GLSCF) consumption, and varied significantly by Great Lake. DDE, lower chlorinated dioxin and furan toxic equivalents (TEQs), and coplanar PCB TEQs were positively associated with noncoplanar PCBs but were not associated with GL sport fish consumption independent of PCB level. Highly chlorinated dioxin and furan congener TEQs were not significantly associated with noncoplanar PCBs or GL sport fish consumption, suggesting that participants were acquiring some of these TEQs from a source other than GLSCF. In epidemiologic studies, it may be important to include populations with higher organochlorine exposures as well as background exposures and to consider the effects of individual congeners or mixtures of congeners on health outcomes.

Using Web-based reporting systems to evaluate health promotion: the experience of two statewide HIV prevention programs.

While Web-based reporting systems are becoming more common in public health for disease surveillance and for tracking interventions such as immunizations, their use for program evaluation is relatively new. This article describes two Web-based reporting systems developed to enable local agencies that conduct health promotion activities to enter process and outcome data for their own use, as well as for analysis by researchers and funders. The systems support the three major uses of evaluation: accountability, program improvement, and generating knowledge for the field. Annually, these programs obtain evaluation information on thousands of clients and individual and group sessions. Developing and introducing the Web-based systems was time-consuming and required significant State Health Department and local agency resources. Involvement of end users in the development process was critical to creating responsive systems that were accepted by staff in local agencies. Staff members from grantee agencies responded well to systems, as evidenced by high rates of user compliance (over 90%) and positive reactions (over 80%) on anonymous surveys. Concerns about resistance from contractors to use of the system, based on fears about breaches in client confidentiality or concerns about the difficulty in using the technology, were not borne out.