Telling the story of complex change: an Impact Framework for the real world.

BACKGROUND: In the National Health Service (NHS) in England, traditional approaches to evidencing impact and value have an important role to play but are unlikely to demonstrate the full value of national quality improvement programmes and large-scale change initiatives in health and care. This type of work almost always takes place in complex and complicated settings, in that it involves multiple players, numerous interventions and a host of other confounding factors. Improvement work is usually emergent, with cause and effect only understood in hindsight; challenges around contribution and attribution can lead the key players to question how they can be certain that the described or observed changes are due to their intervention and would not have happened without them. In this complex environment, there is a risk of oversimplifying the observed impact and focusing instead on those things that are easier to measure, missing that which is important but more difficult to evidence. METHODS: Between 2016 and 2019, an action-orientated approach, drawing on realist and development evaluation approaches, was taken to designing and testing the Impact Framework. First, we undertook a pragmatic review of tools and approaches used by others to capture and demonstrate their impact both within and outside the health and care environment. Following the identification and review of these tools and approaches, and in consultation with national improvement teams in England about their evaluation challenges and aspirations, we developed a set of underpinning principles to inform the design and build of the framework. The principles were informed and finessed following conversations with improvement teams and programme leads in NHS England with respect to the challenges that they were facing and their aspirations in terms of demonstrating their impact and learning as they worked. RESULTS: The 'Impact Framework' described in this article offers a practical approach to capturing the impact of improvement work at any scale, taking account of unintended outcomes, considering attribution and contribution, and using a narrative approach to uncover the difference made by improvement initiatives in rich detail. In this article, we describe how the Impact Framework has been used with one of NHS England's national programmes, Time for Care, which was delivered between 2016 and 2020. CONCLUSIONS: The Impact Framework continues to be used, developed and further tested by national improvement programmes being delivered by NHS England and NHS Improvement and is updated regularly. The framework has been developed to be accessible to frontline teams and is supported by a set of resources to help improvement teams and individuals to use by themselves (https://www.england.nhs.uk/sustainableimprovement/impact-framework/).

Capability building for large-scale transformational change: learning from an evaluation of a national programme.

BACKGROUND: The Long Term Plan presents an ambitious vision for England's National Health Service which will require a sustained programme of transformational change. The Virtual Academy of Large-Scale Change (VALSC) was developed to build capability in health and care system teams involved in transformation or redesign programmes. METHODS: To evaluate the VALSC, quantitative and qualitative data were collected and reviewed against the Kirkpatrick model. Quantitative data were collected via end-of-session surveys to assess individual knowledge before and after participating in capability-building interventions. Qualitative data were also collected and included post-intervention surveys and interviews. Interviews were transcribed and analysed using an inductive approach to identify themes that were subsequently assessed against the Kirkpatrick model. RESULTS: Results suggest that the VALSC programme has helped build capability for large-scale change in terms of learning, behaviour change and impact. Participants' ipsative self-assessment of knowledge demonstrated a significant change (p<0.001) and qualitative data suggested three broad themes in which the VALSC made an impact. First, participants were empowered with transformation and change skills which they applied to local health and care challenges. Second, VALSC helped strengthen connections within and between transformational change teams. Third, VALSC helped transformational change teams to engage more effectively with their stakeholders. CONCLUSIONS: The VALSC developed knowledge, skills, behavioural change and application impact that built capability in individuals and teams. Therefore, continuing to develop capability-building offers that empower and build agency in front-line staff working on service transformation and equip them with approaches, methods and tools to increase their chances of success, is recommended.

Childhood disability and socio-economic circumstances in low and middle income countries: systematic review.

BACKGROUND: The majority of children with disability live in low and middle income (LAMI) countries. Although a number of important reviews of childhood disability in LAMI countries have been published, these have not, to our knowledge, addressed the association between childhood disability and the home socio-economic circumstances (SEC). The objective of this study is to establish the current state of knowledge on the SECs of children with disability and their households in LAMI countries through a systematic review and quality assessment of existing research. METHODS: Electronic databases (MEDLINE; EMBASE; PUBMED; Web of Knowledge; PsycInfo; ASSIA; Virtual Health Library; POPLINE; Google scholar) were searched using terms specific to childhood disability and SECs in LAMI countries. Publications from organisations including the World Bank, UNICEF, International Monetary Fund were searched for. Primary studies and reviews from 1990 onwards were included. Studies were assessed for inclusion, categorisation and quality by 2 researchers. RESULTS: 24 primary studies and 13 reviews were identified. Evidence from the available literature on the association between childhood disability and SECs was inconsistent and inconclusive. Potential mechanisms by which poverty and low household SEC may be both a cause and consequence of disability are outlined in the reviews and the qualitative studies. The association of poor SECs with learning disability and behaviour problems was the most consistent finding and these studies had low/medium risk of bias. Where overall disability was the outcome of interest, findings were divergent and many studies had a high/medium risk of bias. Qualitative studies were methodologically weak. CONCLUSIONS: This review indicates that, despite socially and biologically plausible mechanisms underlying the association of low household SEC with childhood disability in LAMI countries, the empirical evidence from quantitative studies is inconsistent and contradictory. There is evidence for a bidirectional association of low household SEC and disability and longitudinal data is needed to clarify the nature of this association.