NSW Health COVID-19 Emergency Response Priority Research program: a case study of rapid translation of research into health decision making.

OBJECTIVES: The NSW Health COVID-19 Research Program was established in April 2020 to contribute to minimising the health, social and economic impacts of the coronavirus disease 2019 (COVID-19) pandemic in New South Wales (NSW). This paper describes the establishment and implemention of one element of the Program, the Emergency Response Priority Research (Emergency Response) workstream, which is focused on the rapid creation of evidence to support urgent operational work for the public health management of COVID-19 in NSW. METHODS: Narrative description. RESULTS: As at June 2021, nine Emergency Response projects had been funded. Mechanisms used to expedite projects included: embedding academic researchers in NSW Health to work directly with routinely collected NSW Health data; adapting existing research projects to include a COVID-19 component; leveraging established research partnerships to conduct rapid pilots; and directly commissioning urgent projects with experienced and trusted local researchers. LESSONS LEARNT: Evidence from Emergency Response projects has contributed directly to informing the NSW public health response. For example, findings from a study of COVID-19 transmission in schools and childcare settings in the early stages of the pandemic informed decisions around the resumption of on-campus education in 2020 and helped shape policy around higher risk activities to help reduce transmission in education settings. Similarly, findings from a project to validate methods for identifying SARS-CoV-2 virus fragments in wastewater were subsequently incorporated into the NSW Sewage Surveillance Program, which continues to provide NSW Health with information to support targeted messaging and testing. The approach to establishing and implementing the Emergency Response workstream highlights the importance of continuing to ensure a well-trained public health research community and actively supporting a collaborative research sector.

Can an electronic monitoring system capture implementation of health promotion programs? A focussed ethnographic exploration of the story behind program monitoring data.

BACKGROUND: There is a pressing need for policy makers to demonstrate progress made on investments in prevention, but few examples of monitoring systems capable of tracking population-level prevention policies and programs and their implementation. In New South Wales, Australia, the scale up of childhood obesity prevention programs to over 6000 childcare centres and primary schools is monitored via an electronic monitoring system, "PHIMS". METHODS: Via a focussed ethnography with all 14 health promotion implementation teams in the state, we set out to explore what aspects of program implementation are captured via PHIMS, what aspects are not, and the implications for future IT implementation monitoring systems as a result. RESULTS: Practitioners perform a range of activities in the context of delivering obesity prevention programs, but only specific activities are captured via PHIMS. PHIMS thereby defines and standardises certain activities, while non-captured activities can be considered as "extra" work by practitioners. The achievement of implementation targets is influenced by multi-level contextual factors, with only some of the factors accounted for in PHIMS. This evidences incongruencies between work done, recorded and, therefore, recognised. CONCLUSIONS: While monitoring systems cannot and should not capture every aspect of implementation, better accounting for aspects of context and "extra" work involved in program implementation could help illuminate why implementation succeeds or fails. Failure to do so may result in policy makers drawing false conclusions about what is required to achieve implementation targets. Practitioners, as experts of context, are well placed to assist policy makers to develop accurate and meaningful implementation targets and approaches to monitoring.

Key Performance Indicators for program scale-up and divergent practice styles: a study from NSW, Australia.

Implementing programs at scale has become a vital part of the government response to the continuing childhood obesity epidemic. We are studying the largest ever scale-up of school and child care obesity prevention programs in Australia. Health promotion teams support primary schools and early childhood services in their area to achieve a number of specified, evidence-based practices aimed at organizational changes to improve healthy eating and physical activity. Key performance indicators (KPIs) were devised to track program uptake across different areas-measuring both the proportion of schools and early childhood services reached and the proportion of practices achieved in each setting (i.e. the proportion of sites implementing programs as planned). Using a 'tight-loose-tight' model, all local health districts receive funding and are held accountable to reaching KPI implementation targets. However, local teams have independent discretion over how to best use funds to reach targets. Based on 12 months of ethnographic fieldwork and interviews across all districts, this study examines variations in the decision making and strategizing processes of the health promotion teams. We identified three distinct styles of practice: KPI-driven practice (strategic, focussed on targets); relationship-driven practice (focussed on long-term goals); and equity-driven practice (directing resources to sites most in need). In adapting to KPIs, teams make trade-offs and choices. Some teams struggled to balance a moral imperative to attend to equity issues, with a practical need to meet implementation targets. We discuss how models of program scale-up and tracking could possibly evolve to recognize this complexity.

Capturing implementation knowledge: applying focused ethnography to study how implementers generate and manage knowledge in the scale-up of obesity prevention programs.

BACKGROUND: Bespoke electronic information management systems are being used for large-scale implementation delivery of population health programs. They record sites reached, coordinate activity, and track target achievement. However, many systems have been abandoned or failed to integrate into practice. We investigated the unusual endurance of an electronic information management system that has supported the successful statewide implementation of two evidence-based childhood obesity prevention programs for over 5 years. Upwards of 80% of implementation targets are being achieved. METHODS: We undertook co-designed partnership research with policymakers, practitioners, and IT designers. Our working hypothesis was that the science of getting evidence-based programs into practice rests on an in-depth understanding of the role programs play in the ongoing system of local relationships and multiple accountabilities. We conducted a 12-month multisite ethnography of 14 implementation teams, including their use of an electronic information management system, the Population Health Information Management System (PHIMS). RESULTS: All teams used PHIMS, but also drew on additional informal tools and technologies to manage, curate, and store critical information for implementation. We identified six functions these tools performed: (1) relationship management, (2) monitoring progress towards target achievement, (3) guiding and troubleshooting PHIMS use, (4) supporting teamwork, (5) evaluation, and (6) recording extra work at sites not related to program implementation. Informal tools enabled practitioners to create locally derived implementation knowledge and provided a conduit between knowledge generation and entry into PHIMS. CONCLUSIONS: Implementation involves knowing and formalizing what to do, as well as how to do it. Our ethnography revealed the importance of hitherto uncharted knowledge about how practitioners develop implementation knowledge about how to do implementation locally, within the context of scaling up. Harnessing this knowledge for local use required adaptive and flexible systems which were enabled by informal tools and technologies. The use of informal tools also complemented and supported PHIMS use suggesting that both informal and standardized systems are required to support coordinated, large-scale implementation. While the content of the supplementary knowledge required to deliver the program was specific to context, functions like managing relationships with sites and helping others in the team may be applicable elsewhere.

How are evidence generation partnerships between researchers and policy-makers enacted in practice? A qualitative interview study.

BACKGROUND: Evidence generation partnerships between researchers and policy-makers are a potential method for producing more relevant research with greater potential to impact on policy and practice. Little is known about how such partnerships are enacted in practice, however, or how to increase their effectiveness. We aimed to determine why researchers and policy-makers choose to work together, how they work together, which partnership models are most common, and what the key (1) relationship-based and (2) practical components of successful research partnerships are. METHODS: Semi-structured qualitative interviews were conducted with 18 key informants largely based in New South Wales, Australia, who were (1) researchers experienced in working in partnership with policy in health or health-related areas or (2) policy and programme developers and health system decision-makers experienced in working in partnership with researchers. Data was analysed thematically by two researchers. RESULTS: Researcher-initiated and policy agency-initiated evidence generation partnerships were common. While policy-initiated partnerships were thought to be the most likely to result in impact, researcher-initiated projects were considered important in advancing the science and were favoured by researchers due to greater perceived opportunities to achieve key academic career metrics. Participants acknowledged that levels of collaboration varied widely in research/policy partnerships from minimal to co-production. Co-production was considered a worthy goal by all, conferring a range of benefits, but one that was difficult to achieve in practice. Some participants asserted that the increased time and resources required for effective co-production meant it was best suited to evaluation and implementation projects where the tacit, experiential knowledge of policy-makers provided critical nuance to underpin study design, implementation and analysis. Partnerships that were mutually considered to have produced the desired outcomes were seen to be underpinned by a range of both relationship-based (such as shared aims and goals and trust) and practical factors (such as sound governance and processes). CONCLUSIONS: Our findings highlight the important role of policy-makers in New South Wales in ensuring the relevance of research. There is still much to understand about how to initiate and sustain successful research/policy partnerships, particularly at the highly collaborative end.

Codesign of the Population Health Information Management System to measure reach and practice change of childhood obesity programs.

INTRODUCTION: Childhood obesity prevalence is an issue of international public health concern and governments have a significant role to play in its reduction. The Healthy Children Initiative (HCI) has been delivered in New South Wales (NSW), Australia, since 2011 to support implementation of childhood obesity prevention programs at scale. Consequently, a system to support local implementation and data collection, analysis and reporting at local and state levels was necessary. The Population Health Information Management System (PHIMS) was developed to meet this need. Design and development: A collaborative and iterative process was applied to the design and development of the system. The process comprised identifying technical requirements, building system infrastructure, delivering training, deploying the system and implementing quality measures. Use of PHIMS: Implementation of PHIMS resulted in rapid data retrieval and reporting against agreed performance measures for the HCI. The system has 150 users who account for the monitoring and reporting of more than 6000 HCI intervention sites (early childhood services and primary schools). LESSONS LEARNT: Developing and implementing PHIMS presented a number of complexities including: applying an information technology (IT) development methodology to a traditional health promotion setting; data access and confidentiality issues; and managing system development and deployment to intended timelines and budget. PHIMS was successfully codesigned as a flexible, scalable and sustainable IT solution that supports state-wide HCI program implementation, monitoring and reporting.

The Prevention Research Support Program: supporting innovation in research, translation and capability building.

OBJECTIVES: The Prevention Research Support Program (PRSP) is a New South Wales (NSW) Ministry of Health funding scheme. The scheme aims to build capability, and strengthen prevention and early intervention research that is important to the NSW public health system (NSW Health) and that leads to improved health and reduced health inequities for the people of NSW. This paper describes how PRSP funding has supported recipients to produce high-quality, policy-relevant research, and increase the impact of research on policy and practice. Type of program: The PRSP is a competitive funding program that supports NSW research organisations that conduct prevention and early intervention research that aligns with NSW Health priorities. The objectives of the PRSP are to: increase high-quality and internationally recognised prevention research in NSW; support the generation of research evidence that addresses NSW Health prevention priorities, including cross-government priorities; encourage the adoption of research evidence in relevant policies, programs and services in NSW; and build the prevention research capability of NSW Health staff and the NSW Health system. METHODS: Funding recipients provide information about their research, translation and capability building achievements in their funding applications and submit annual progress reports. Data from these sources were aggregated to illustrate trends in indicators of research excellence over time. Prior to the most recent call for applications, the program was reviewed. The review included consultations with funding recipients, policy and practice partners, and key funding stakeholders. Stakeholders' perceptions of the benefits and challenges associated with the PRSP were drawn from the consultation data. RESULTS: PRSP funding recipients demonstrate considerable increases over time on several indicators of research excellence, including peer-reviewed journal publications, grant income, and research students supervised. Recipients use a range of strategies to ensure dialogue with health system partners, and report research impacts at the local, state, national and international levels. PRSP funding also supports the development of research capability. LESSONS LEARNT: The PRSP is a unique scheme that is highly valued by both funding recipients and health system stakeholders. The continuity of funding provided under the scheme enables recipients to adopt a strategic approach to their research and develop innovative strategies to support its conduct and use.

Dynamics behind the scale up of evidence-based obesity prevention: protocol for a multi-site case study of an electronic implementation monitoring system in health promotion practice.

BACKGROUND: The effectiveness of many interventions to promote health and prevent disease has been well established. The imperative has therefore shifted from amassing evidence about efficacy to scale-up to maximise population-level health gains. Electronic implementation monitoring, or 'e-monitoring', systems have been designed to assist and track the delivery of preventive policies and programs. However, there is little evidence on whether e-monitoring systems improve the dissemination, adoption, and ongoing delivery of evidence-based preventive programs. Also, given considerable difficulties with e-monitoring systems in the clinical sector, scholars have called for a more sophisticated re-examination of e-monitoring's role in enhancing implementation. METHODS: In the state of New South Wales (NSW), Australia, the Population Health Information Management System (PHIMS) was created to support the dissemination of obesity prevention programs to 6000 childcare centres and elementary schools across all 15 local health districts. We have established a three-way university-policymaker-practice research partnership to investigate the impact of PHIMS on practice, how PHIMS is used, and how achievement of key performance indicators of program adoption may be associated with local contextual factors. Our methods encompass ethnographic observation, key informant interviews and participatory workshops for data interpretation at a state and local level. We use an on-line social network analysis of the collaborative relationships across local health district health promotion teams to explore the relationship between PHIMS use and the organisational structure of practice. DISCUSSION: Insights will be sensitised by institutional theory, practice theory and complex adaptive system thinking, among other theories which make sense of socio-technical action. Our working hypothesis is that the science of getting evidence-based programs into practice rests on an in-depth understanding of the role they play in the on-going system of local relationships and multiple accountabilities. Data will be synthesised to produce a typology to characterise local context, PHIMS use and key performance indicator achievement (of program implementation) across the 15 local health districts. Results could be used to continuously align e-monitoring technologies within quality improvement processes to ensure that such technologies enhance practice and innovation. A partnership approach to knowledge production increases the likelihood that findings will be put into practice.

A long-term, strategic approach to evidence generation and knowledge translation in NSW, Australia.

There is a growing body of literature about the barriers to, and enablers of, the use of research evidence in policy and practice. Research funders are in a unique position to influence activities aimed at promoting research use. During the past decade, NSW (New South Wales) Health has systematically built and invested in an integrated population and health services research portfolio made up of different types of investment and policy focuses. Each of these investments has an explicit focus on translation. Ensuring a long-term, sustained, strategic approach to priority-driven research generation, true coproduction of evidence and effective knowledge translation will continue to deliver results for NSW. The NSW Population Health Research Strategy 2017-2021, currently in development, will have a strong emphasis on fostering environments and actions that promote the use of research in policy and practice.

A new model of collaborative research: experiences from one of Australia's NHMRC Partnership Centres for Better Health.

There is often a disconnection between the creation of evidence and its use in policy and practice. Cross-sectoral, multidisciplinary partnership research, founded on shared governance and coproduction, is considered to be one of the most effective means of overcoming this research-policy-practice disconnect. Similar to a number of funding bodies internationally, Australia's National Health and Medical Research Council has introduced Partnership Centres for Better Health: a scheme explicitly designed to encourage coproduced partnership research. In this paper, we describe our experiences of The Australian Prevention Partnership Centre, established in June 2013 to explore the systems, strategies and structures that inform decisions about how to prevent lifestyle-related chronic disease. We present our view on how the Partnership Centre model is working in practice. We comment on the unique features of the Partnership Centre funding model, how these features enable ways of working that are different from both investigator-initiated and commissioned research, and how these ways of working can result in unique outcomes that would otherwise not have been possible. Although not without challenges, the Partnership Centre approach addresses a major gap in the Australian research environment, whereby large-scale, research-policy-practice partnerships are established with sufficient time, resources and flexibility to deliver highly innovative, timely and accessible research that is of use to policy and practice.

Socioeconomic, remoteness and sex differences in life expectancy in New South Wales, Australia, 2001-2012: a population-based study.

OBJECTIVES: Despite being one of the healthiest countries in the world, Australia displays substantial mortality differentials by socioeconomic disadvantage, remoteness and sex. In this study, we examined how these mortality differentials translated to differences in life expectancy between 2001 and 2012. DESIGN AND SETTING: Population-based study using mortality and estimated residential population data from Australia's largest state, New South Wales (NSW), between 2001 and 2012. Age-group-specific death rates by socioeconomic disadvantage quintile, remoteness (major cities vs regional and remote areas), sex and year were estimated via Poisson regression, and inputted into life table calculations to estimate life expectancy. RESULTS: Life expectancy decreased with increasing socioeconomic disadvantage in males and females. The disparity between the most and least socioeconomically deprived quintiles was 3.77 years in males and 2.39 years in females in 2012. Differences in life expectancy by socioeconomic disadvantage were mostly stable over time. Gender gaps in life expectancy ranged from 3.50 to 4.93 years (in 2012), increased with increasing socioeconomic disadvantage and decreased by ∼1 year for all quintiles between 2001 and 2012. Overall, life expectancy varied little by remoteness, but was 1.8 years higher in major cities compared to regional/remote areas in the most socioeconomically deprived regions in 2012. CONCLUSIONS: Socioeconomic disadvantage and sex were strongly associated with life expectancy. The disparity in life expectancy across the socioeconomic spectrum was larger in males and was stable over time. In contrast, gender gaps reduced for all quintiles between 2001 and 2012, and a remoteness effect was evident in 2012, but only for those living in the most deprived areas.

Diagnosis-based emergency department alcohol harm surveillance: What can it tell us about acute alcohol harms at the population level?

INTRODUCTION AND AIMS: Acute harm from heavy drinking episodes is an increasing focus of public health policy, but capturing timely data on acute harms in the population is challenging. This study aimed to evaluate the precision of readily available administrative emergency department (ED) data in public health surveillance of acute alcohol harms. DESIGN AND METHODS: We selected a random sample of 1000 ED presentations assigned an ED diagnosis code for alcohol harms (the 'alcohol syndrome') in the New South Wales, Australia, automatic syndromic surveillance system. The sample was selected from 68 public hospitals during 2014. Nursing triage free-text fields were independently reviewed to confirm alcohol consumption and classify each presentation into either an 'acute' or 'chronic' harm. Positive predictive value (PPV) for acute harm was calculated, and predictors of acute harm presentations were estimated using logistic regression. RESULTS: The PPV of the alcohol syndrome for acute alcohol harm was 53.5%. Independent predictors of acute harm were ambulance arrival [adjusted odds ratio (aOR) = 3.4, 95% confidence interval (CI) 2.4-4.7], younger age (12-24 vs. 25-39 years: aOR = 3.4, 95% CI 2.2-5.3), not being admitted (aOR 2.2, 95% CI 1.5-3.2) and arriving between 10 pm and 5.59 am (aOR 2.1, 95% CI 1.5-2.8). PPV among 12 to 24-year-olds was 82%. DISCUSSION AND CONCLUSIONS: The alcohol syndrome provides moderate precision as an indicator of acute alcohol harms presenting to the ED. Precision for monitoring acute harm in the population is improved by filtering the syndrome by the strongest independent predictors of acute alcohol harm presentations. [Whitlam G, Dinh M, Rodgers C, Muscatello DJ, McGuire R, Ryan T, Thackway S. Diagnosis-based emergency department alcohol harm surveillance: What can it tell us about acute alcohol harms at the population level? Drug Alcohol Rev 2016;35:693-701].

Building research and evaluation capacity in population health: the NSW Health approach.

Issue addressed An organisational culture that values and uses research and evaluation (R&E) evidence to inform policy and practice is fundamental to improving health outcomes. The 2016 NSW Government Program Evaluation Guidelines recommend investment in training and development to improve evaluation capacity. The purpose of this paper is to outline the approaches taken by the NSW Ministry of Health to develop R&E capacity and assess these against existing models of practice. Method The Ministry of Health's Centre for Epidemiology and Evidence (CEE) takes an evidence-based approach to building R&E capacity in population health. Strategies are informed by: the NSW Population Health Research Strategy, R&E communities of practice across the Ministry and health Pillar agencies and a review of the published evidence on evaluation capacity building (ECB). An internal survey is conducted biennially to monitor research activity within the Ministry's Population and Public Health Division. One representative from each of the six centres that make up the Division coordinates completion of the survey by relevant staff members for their centre. Results The review identified several ECB success factors including: implementing a tailored multifaceted approach; an organisational commitment to R&E; and offering experiential training and ongoing technical support to the workforce. The survey of research activity found that the Division funded a mix of research assets, research funding schemes, research centres and commissioned R&E projects. CEE provides technical advice and support services for staff involved in R&E and in 2015, 22 program evaluations were supported. R&E capacity building also includes a series of guides to assist policy makers, practitioners and researchers to commission, undertake and use policy-relevant R&E. Staff training includes workshops on critical appraisal, program logic and evaluation methods. From January 2013 to June 2014 divisional staff published 84 peer-reviewed papers and one book chapter. Conclusion A strategic approach to R&E capacity building compares favourably with organisational dimensions of ECB and has facilitated the generation of high quality population health R&E in NSW. So what? An evidence-based multistrategy approach to population health R&E can result in substantial contributions to the population-health evidence base.

Automatic classification of diseases from free-text death certificates for real-time surveillance.

BACKGROUND: Death certificates provide an invaluable source for mortality statistics which can be used for surveillance and early warnings of increases in disease activity and to support the development and monitoring of prevention or response strategies. However, their value can be realised only if accurate, quantitative data can be extracted from death certificates, an aim hampered by both the volume and variable nature of certificates written in natural language. This study aims to develop a set of machine learning and rule-based methods to automatically classify death certificates according to four high impact diseases of interest: diabetes, influenza, pneumonia and HIV. METHODS: Two classification methods are presented: i) a machine learning approach, where detailed features (terms, term n-grams and SNOMED CT concepts) are extracted from death certificates and used to train a set of supervised machine learning models (Support Vector Machines); and ii) a set of keyword-matching rules. These methods were used to identify the presence of diabetes, influenza, pneumonia and HIV in a death certificate. An empirical evaluation was conducted using 340,142 death certificates, divided between training and test sets, covering deaths from 2000-2007 in New South Wales, Australia. Precision and recall (positive predictive value and sensitivity) were used as evaluation measures, with F-measure providing a single, overall measure of effectiveness. A detailed error analysis was performed on classification errors. RESULTS: Classification of diabetes, influenza, pneumonia and HIV was highly accurate (F-measure 0.96). More fine-grained ICD-10 classification effectiveness was more variable but still high (F-measure 0.80). The error analysis revealed that word variations as well as certain word combinations adversely affected classification. In addition, anomalies in the ground truth likely led to an underestimation of the effectiveness. CONCLUSIONS: The high accuracy and low cost of the classification methods allow for an effective means for automatic and real-time surveillance of diabetes, influenza, pneumonia and HIV deaths. In addition, the methods are generally applicable to other diseases of interest and to other sources of medical free-text besides death certificates.

Inaccurate ascertainment of morbidity and mortality due to influenza in administrative databases: a population-based record linkage study.

BACKGROUND: Historically, counting influenza recorded in administrative health outcome databases has been considered insufficient to estimate influenza attributable morbidity and mortality in populations. We used database record linkage to evaluate whether modern databases have similar limitations. METHODS: Person-level records were linked across databases of laboratory notified influenza, emergency department (ED) presentations, hospital admissions and death registrations, from the population (∼6.9 million) of New South Wales (NSW), Australia, 2005 to 2008. RESULTS: There were 2568 virologically diagnosed influenza infections notified. Among those, 25% of 40 who died, 49% of 1451 with a hospital admission and 7% of 1742 with an ED presentation had influenza recorded on the respective database record. Compared with persons aged ≥65 years and residents of regional and remote areas, respectively, children and residents of major cities were more likely to have influenza coded on their admission record. Compared with older persons and admitted patients, respectively, working age persons and non-admitted persons were more likely to have influenza coded on their ED record. On both ED and admission records, persons with influenza type A infection were more likely than those with type B infection to have influenza coded. Among death registrations, hospital admissions and ED presentations with influenza recorded as a cause of illness, 15%, 28% and 1.4%, respectively, also had laboratory notified influenza. Time trends in counts of influenza recorded on the ED, admission and death databases reflected the trend in counts of virologically diagnosed influenza. CONCLUSIONS: A minority of the death, hospital admission and ED records for persons with a virologically diagnosed influenza infection identified influenza as a cause of illness. Few database records with influenza recorded as a cause had laboratory confirmation. The databases have limited value for estimating incidence of influenza outcomes, but can be used for monitoring variation in incidence over time.