Uptake of a Dashboard Designed to Give Realtime Feedback to a Sentinel Network About Key Data Required for Influenza Vaccine Effectiveness Studies.

Dashboards are technologies that bringing together a range of data sources for observational or analytical purposes. We have created a customised dashboard that includes all the key data elements required for monitoring flu vaccine effectiveness (FVE). This delivers a unique dashboard for each primary care provider (general practice) providing data to the Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC), one of the oldest European surveillance systems. These FVE studies use a test negative case control (TNCC) design. TNCC requires knowledge of practice denominator; vaccine exposure, and results of influenza virology swabs carried out to identify in an influenza-like-illness (ILI), a clinical diagnosis, really is influenza. The dashboard displays the denominator uploaded each week into the surveillance system, compared with the nationally known practice size (providing face-validity for the denominator); it identifies those exposed to the vaccine (by age group and risk category) and virology specimens taken and missed opportunities for surveillance (again by category). All sentinel practices can access in near real time (4 working days in areas) their rates of vaccine exposure and swabs conducted. Initial feedback is positive; 80% (32/40) practices responded positively.

Using the computer in the clinical consultation; setting the stage, reviewing, recording, and taking actions: multi-channel video study.

BACKGROUND AND OBJECTIVE: Electronic patient record (EPR) systems are widely used. This study explores the context and use of systems to provide insights into improving their use in clinical practice. METHODS: We used video to observe 163 consultations by 16 clinicians using four EPR brands. We made a visual study of the consultation room and coded interactions between clinician, patient, and computer. Few patients (6.9%, n=12) declined to participate. RESULTS: Patients looked at the computer twice as much (47.6 s vs 20.6 s, p<0.001) when it was within their gaze. A quarter of consultations were interrupted (27.6%, n=45); and in half the clinician left the room (12.3%, n=20). The core consultation takes about 87% of the total session time; 5% of time is spent pre-consultation, reading the record and calling the patient in; and 8% of time is spent post-consultation, largely entering notes. Consultations with more than one person and where prescribing took place were longer (R(2) adj=22.5%, p<0.001). The core consultation can be divided into 61% of direct clinician-patient interaction, of which 15% is examination, 25% computer use with no patient involvement, and 14% simultaneous clinician-computer-patient interplay. The proportions of computer use are similar between consultations (mean=40.6%, SD=13.7%). There was more data coding in problem-orientated EPR systems, though clinicians often used vague codes. CONCLUSIONS: The EPR system is used for a consistent proportion of the consultation and should be designed to facilitate multi-tasking. Clinicians who want to promote screen sharing should change their consulting room layout.

The pattern of silent time in the clinical consultation: an observational multichannel video study.

OBJECTIVE: To describe silent time in the clinical consultation: who initiates and terminates it and at what stage most silence occurs. METHODS: We conducted an analysis of 127 multichannel video recordings of consultations by 12 GPs; filmed using the ALFA (Aggregation of Log Files for Analysis) open-source toolkit. The start and end of silence was manually coded using an observational data capture tool. We report who initiates and terminates silence, describe the proportion of the consultation, what happens within it and the distribution of silent periods by quartile of the consultation. RESULTS: We found the median proportion of silence was 12.3% and interquartile range 14.3%. Silent periods (52.4%) were both initiated and terminated by the doctor. The majority of silent time (78.1%) is spent on computer-based activities and physical examination. Silent periods which do not involve physical examination mainly occur in the second half of the consultation and represent 70.6% of the total duration and 64.8% of the episodes of silence. CONCLUSIONS: The computer is a third party in the GP consultation and often requires silent time during doctor-computer interaction. Doctors' control and patients allow silence for the doctor to complete tasks often involving the computer and also for time out from the consultation. There is a clear pattern of when doctors need most to have silence and consultation models should be developed that reflect this need.

Disparities in testing for renal function in UK primary care: cross-sectional study.

BACKGROUND: In the UK, explicit quality standards for chronic disease management, including for diabetes and chronic kidney disease (CKD), are set out National Service Frameworks and pay-for-performance indicators. These conditions are common with a prevalence of 4% and 5.4%, respectively. CKD is largely asymptomatic, detected following renal function testing and important because associated with increased mortality and morbidity, especially in people with diabetes and proteinuria. OBJECTIVES: To investigate who has their renal function tested and any association with age, sex, ethnicity and diabetes. METHOD: A cross-sectional survey in a primary care research network in south-west London (n = 220 721). The following data were extracted from routine data: age, gender, ethnicity, latest serum creatinine, diagnosis of diabetes and recording of proteinuria. We used logistic regression to explore any association in testing for CKD. RESULTS: People (82.1%) with diabetes had renal function and proteinuria tested; the proportion was much smaller (<0.5%) in those without. Women were more likely to have a creatinine test than men (28% versus 24%, P < 0.05), but this association was modified by age, ethnicity and presence of diabetes. People >75 years and with diabetes were most likely to have been tested. Black [adjusted odds ratio (AOR) 2.1, 95% confidence interval (CI) 2.0-2.2] and south Asian (AOR 1.65, 95% CI 1.56-1.75) patients were more likely to be tested than whites. Those where ethnicity was not stated were the only group not tested more than whites. CONCLUSIONS: Quality improvement initiatives and equity audits, which include CKD should take account of disparities in renal function testing.

A system for solution-orientated reporting of errors associated with the extraction of routinely collected clinical data for research and quality improvement.

BACKGROUND: We have used routinely collected clinical data in epidemiological and quality improvement research for over 10 years. We extract, pseudonymise and link data from heterogeneous distributed databases; inevitably encountering errors and problems. OBJECTIVE: To develop a solution-orientated system of error reporting which enables appropriate corrective action. METHOD: Review of the 94 errors, which occurred in 2008/9. Previously we had described failures in terms of the data missing from our response files; however this provided little information about causation. We therefore developed a taxonomy based on the IT component limiting data extraction. RESULTS: Our final taxonomy categorised errors as: (A) Data extraction Method and Process; (B) Translation Layer and Proxy Specification; (C) Shape and Complexity of the Original Schema; (D) Communication and System (mainly Software-based) Faults; (E) Hardware and Infrastructure; (F) Generic/Uncategorised and/or Human Errors. We found 79 distinct errors among the 94 reported; and the categories were generally predictive of the time needed to develop fixes. CONCLUSIONS: A systematic approach to errors and linking them to problem solving has improved project efficiency and enabled us to better predict any associated delays.

Addressing modifiable risk factors for coronary heart disease in primary care: an evidence-base lost in translation.

BACKGROUND: Risk factors for cardiovascular disease can be modified in primary care. Electronic patient record (EPR) systems include embedded cardiovascular risk factor calculators and should facilitate this process. OBJECTIVE: To observe how the evidence base for assessing and managing cardiovascular risk is implemented in practice. METHOD: We compared the different risk calculators promoted for calculating cardiovascular risk in primary care using four test cases. We looked to see how these calculators were implemented in primary care EPR systems. We explored through a workshop which risk factors GPs thought were important and felt motivated to address as part of clinical care. RESULTS: The risk calculators reviewed use different sets of risk factors and the levels of risk calculated for the same test patient profiles vary by up to 100%. The risk factor calculators embedded within UK computer systems all include the Framingham equation though there is variation in interface, default values and patient groups included. GPs showed consensus around the importance of managing smoking, blood pressure, obesity (body mass index), diabetes and cholesterol but also stressed the importance of providing personalized care and exercising professional judgement. CONCLUSIONS: There appears to be an evidence-base lost in translation. Different guidelines calculate risk differently, and even when the same guideline is used, variation in implementation leads to further variation. Education and development of improved risk calculators should enable the most appropriate calculator to be used for an individual patient; accreditation of implementation could be achieved through the use of a standard set of test cases.

Measuring the impact of the computer on the consultation: an open source application to combine multiple observational outputs.

A diverse range of tools and techniques can be used to observe the clinical consultation and the use of information technology. These technologies range from transcripts; to video observation with one or more cameras; to voice and pattern recognition applications. Currently, these have to be observed separately and there is limited capacity to combine them. Consequently, when multiple methods are used to analyse the consultation a significant proportion of time is spent linking events in one log file (e.g. mouse movements and keyboard use when prescribing alerts appear) with what was happening in the consultation at that time. The objective of this study was to develop an application capable of combining and comparing activity log-files and with facilities to view simultaneously all data relating to any time point or activity. Interviews, observations and design prototypes were used to develop a specification. Class diagram of the application design was used to make further development decisions. The application development used object-orientated design principles. We used open source tools; Java as the programming language and JDeveloper as the development environment. The final output is log file aggregation (LFA) tool which forms part of the wider aggregation of log files for analysis (ALFA) open source toolkit ( www.biomedicalinformatics.info/alfa/ ). Testing was done using sample log files and reviewed the application's utility for analysis of the consultation activities. Separation of the presentation and functionality in the design stage enabled us to develop a modular and extensible application. The application is capable of converting and aggregating several log files of different formats and displays them in different presentation layouts. We used the Java Media Framework to aggregate video channels. Java extensible mark-up language (XML) package facilitated the conversion of aggregated output into XML format. Analysts can now move easily between observation tools and find all the data related to an activity. The LFA application makes new analysis tasks feasible and established tasks much more efficient. Researchers can now store multiple log file data as a single file isolate and investigate different doctor-computer-patient interaction.

Using an open source observational tool to measure the influence of the doctor's consulting style and the computer system on the outcomes of the clinical consultation.

Computerization of general practice is an international phenomenon. Many of the Electronic Patient Record (EPR) systems have developed organically with considerable variation in their interface and functionality. Consequently they have differing impact on the clinical consultation. There is a dearth of tools available to study their impact on the consultation. The objective is to use ALFA to film and analyze a simulated clinical consultation. We used the ALFA (Activity Log File Aggregation) open source toolkit, to make video based observation and analysis of the computer mediated consultation. ALFA enables precise comparison of core elements of EPR systems. It allows multiple video channels including screen capture, data about computer use, and verbal interactions to be synchronized, timed and navigated through for analysis. The toolkit is free and can be downloaded under an open source license from www.biomedicalinformatics.info/alfa/. Its outputs, which include Unified Modelling Language (UML), provide the evidence-base for assessing the impact of the computer on the consultation the designing of EPR systems. ALFA has been used to compare different brands of primary care computer systems; nurse case-load selection and consultation in psychiatry.

The ALFA (Activity Log Files Aggregation) toolkit: a method for precise observation of the consultation.

BACKGROUND: There is a lack of tools to evaluate and compare Electronic patient record (EPR) systems to inform a rational choice or development agenda. OBJECTIVE: To develop a tool kit to measure the impact of different EPR system features on the consultation. METHODS: We first developed a specification to overcome the limitations of existing methods. We divided this into work packages: (1) developing a method to display multichannel video of the consultation; (2) code and measure activities, including computer use and verbal interactions; (3) automate the capture of nonverbal interactions; (4) aggregate multiple observations into a single navigable output; and (5) produce an output interpretable by software developers. We piloted this method by filming live consultations (n = 22) by 4 general practitioners (GPs) using different EPR systems. We compared the time taken and variations during coded data entry, prescribing, and blood pressure (BP) recording. We used nonparametric tests to make statistical comparisons. We contrasted methods of BP recording using Unified Modeling Language (UML) sequence diagrams. RESULTS: We found that 4 channels of video were optimal. We identified an existing application for manual coding of video output. We developed in-house tools for capturing use of keyboard and mouse and to time stamp speech. The transcript is then typed within this time stamp. Although we managed to capture body language using pattern recognition software, we were unable to use this data quantitatively. We loaded these observational outputs into our aggregation tool, which allows simultaneous navigation and viewing of multiple files. This also creates a single exportable file in XML format, which we used to develop UML sequence diagrams. In our pilot, the GP using the EMIS LV (Egton Medical Information Systems Limited, Leeds, UK) system took the longest time to code data (mean 11.5 s, 95% CI 8.7-14.2). Nonparametric comparison of EMIS LV with the other systems showed a significant difference, with EMIS PCS (Egton Medical Information Systems Limited, Leeds, UK) (P = .007), iSoft Synergy (iSOFT, Banbury, UK) (P = .014), and INPS Vision (INPS, London, UK) (P = .006) facilitating faster coding. In contrast, prescribing was fastest with EMIS LV (mean 23.7 s, 95% CI 20.5-26.8), but nonparametric comparison showed no statistically significant difference. UML sequence diagrams showed that the simplest BP recording interface was not the easiest to use, as users spent longer navigating or looking up previous blood pressures separately. Complex interfaces with free-text boxes left clinicians unsure of what to add. CONCLUSIONS: The ALFA method allows the precise observation of the clinical consultation. It enables rigorous comparison of core elements of EPR systems. Pilot data suggests its capacity to demonstrate differences between systems. Its outputs could provide the evidence base for making more objective choices between systems.

Measuring the impact of different brands of computer systems on the clinical consultation: a pilot study.

BACKGROUND: UK general practitioners largely conduct computer-mediated consultations. Although historically there were many small general practice (GP) computer suppliers there are now around five widely used electronic patient record (EPR) systems. A new method has been developed for assessing the impact of the computer on doctor-patient interaction through detailed observation of the consultation and computer use. OBJECTIVE: To pilot the latest version of a method to measure the difference in coding and prescribing times on two different brands of general practice EPR system. METHOD: We compared two GP EPR systems by observing use in real life consultations. Three video cameras recorded the consultation and screen capture software recorded computer activity. We piloted semi-automated user action recording (UAR) software to record mouse and keyboard use, to overcome limitations in manual measurement. Six trained raters analysed the videos using data capture software to measure the doctor-patient-computer interactions; we used interclass correlation coefficients (ICC) to measure reliability. RESULTS: Raters demonstrated high inter-rater reliability for verbal interactions and prescribing (ICC 0.74 to 0.99), but for measures of computer use they were not reliable. We used UAR to capture computer use and found it more reliable. Coded data entry time varied between the systems: 6.8 compared with 11.5 seconds (P = 0.006). However, the EPR with the shortest coding time had a longer prescribing time: 27.5 compared with 23.7 seconds (P = 0.64). CONCLUSION: This methodological development improves the reliability of our method for measuring the impact of different computer systems on the GP consultation. UAR added more objectivity to the observation of doctor-computer interactions. If larger studies were to reproduce the differences between computer systems demonstrated in this pilot it might be possible to make objective comparisons between systems.

Assessing the impact of recording quality target data on the GP consultation using multi-channel video.

BACKGROUND: In the UK routinely collected computerized clinical data is used to assess progress towards financially incentivised quality targets for chronic disease management including hypertension. OBJECTIVE: To develop a method for assessing the impact of recording quality target data in the clinical consultation. METHODS: Raters were trained how to rate a multi-channel video of a simulated clinical consultation for interaction between actors, computer use, non-verbal communication. RESULTS: 25% of consultation time is computer use and a median of 4 to 5 items were coded per consultation mainly items related to the hypertension quality target. Intraclass correlation coefficient showed good inter-rater reliability (>0.9; p<0.001). CONCLUSION: We have successfully piloted a novel technique for observing the influence of the computer on the consultation. Despite increasing computer use to record quality target data the overwhelming proportion of the consultation remains doctor patient interaction.

The feasibility of using UML to compare the impact of different brands of computer system on the clinical consultation.

BACKGROUND: UK general practice is universally computerised, with computers used in the consulting room at the point of care. Practices use a range of different brands of computer system, which have developed organically to meet the needs of general practitioners and health service managers. Unified Modelling Language (UML) is a standard modelling and specification notation widely used in software engineering. OBJECTIVE: To examine the feasibility of UML notation to compare the impact of different brands of general practice computer system on the clinical consultation. METHOD: Multi-channel video recordings of simulated consultation sessions were recorded on three different clinical computer systems in common use (EMIS, iSOFT Synergy and IPS Vision). User action recorder software recorded time logs of keyboard and mouse use, and pattern recognition software captured non-verbal communication. The outputs of these were used to create UML class and sequence diagrams for each consultation. We compared 'definition of the presenting problem' and 'prescribing', as these tasks were present in all the consultations analysed. RESULTS: Class diagrams identified the entities involved in the clinical consultation. Sequence diagrams identified common elements of the consultation (such as prescribing) and enabled comparisons to be made between the different brands of computer system. The clinician and computer system interaction varied greatly between the different brands. CONCLUSIONS: UML sequence diagrams are useful in identifying common tasks in the clinical consultation, and for contrasting the impact of the different brands of computer system on the clinical consultation. Further research is needed to see if patterns demonstrated in this pilot study are consistently displayed.

A study of cardiovascular risk in overweight and obese people in England.

OBJECTIVES: To report current levels of obesity and associated cardiac risk using routinely collected primary care computer data. METHODS: 67 practices took part in an educational intervention to improve computer data quality and care in cardiovascular disease. Data were extracted from 435,102 general practice computer records. 64.3% (229,108/362,861) of people age 15 y and older had a body mass index (BMI) recording or a valid height and weight record that enabled BMI to be derived. Data about cardiovascular disease and risk factors were also extracted. The prevalence of disease and the control of risk factors in the overweight and obese population were compared with those of normal body weight. RESULTS: 56.8% of men and 69.3% of women aged over 15 y had a BMI record. 22% of men and 32.3% of women aged 15 to 24 y were overweight or obese; rising each decade to a peak of 65.6% of men and 57.5% of women aged 55 to 64 y. Thereafter, the proportion who were overweight or obese declined. The prevalence of ischaemic heart disease, diabetes mellitus and hypertension rose with increasing levels of obesity; their prevalence in those who are moderately obese was between two and three times that of the general population. Systolic and diastolic blood pressure, blood glucose even in non-diabetics, cholesterol and triglycerides were all elevated in the overweight and obese population. CONCLUSION: Based on the recorded data over half of men and nearly half of women are overweight or obese. They have increased cardiovascular risk, which is not adequately controlled by current practice.

Routinely-collected general practice data are complex, but with systematic processing can be used for quality improvement and research.

BACKGROUND: UK general practice is computerised, and quality targets based on computer data provide a further incentive to improve data quality. A National Programme for Information Technology is standardising the technical infrastructure and removing some of the barriers to data aggregation. Routinely collected data is an underused resource, yet little has been written about the wide range of factors that need to be taken into account if we are to infer meaning from general practice data. OBJECTIVE: To report the complexity of general practice computer data and factors that need to be taken into account in its processing and interpretation. METHOD: We run clinically focused programmes that provide clinically relevant feedback to clinicians, and overview statistics to localities and researchers. However, to take account of the complexity of these data we have carefully devised a system of process stages and process controls to maintain referential integrity, and improve data quality and error reduction. These are integrated into our design and processing stages. Our systems document the query, reference code set and create unique patient ID. The design stage is followed by appraisal of: data entry issues, how concepts might be represented in clinical systems, coding ambiguities, using surrogates where needed, validation and pilot-ing. The subsequent processing of data includes extraction, migration and integration of data from different sources, cleaning, processing and analysis. RESULTS: Results are presented to illustrate issues with the population denominator, data entry problems, identification of people with unmet needs, and how routine data can be used for real-world testing of pharmaceuticals. CONCLUSIONS: Routinely collected primary care data could contribute more to the process of health improvement; however, those working with these data need to understand fully the complexity of the context within which data entry takes place.