Evaluating Explanations From AI Algorithms for Clinical Decision-Making: A Social Science-Based Approach.

Explainable Artificial Intelligence (XAI) techniques generate explanations for predictions from AI models. These explanations can be evaluated for (i) faithfulness to the prediction, i.e., its correctness about the reasons for prediction, and (ii) usefulness to the user. While there are metrics to evaluate faithfulness, to our knowledge, there are no automated metrics to evaluate the usefulness of explanations in the clinical context. Our objective is to develop a new metric to evaluate usefulness of AI explanations to clinicians. Usefulness evaluation needs to consider both (a) how humans generally process explanations and (b) clinicians' specific requirements from explanations presented by clinical decision support systems (CDSS). Our new scoring method can evaluate the usefulness of explanations generated by any XAI method that provides importance values for the input features of the prediction model. Our method draws on theories from social science to gauge usefulness, and uses literature-derived biomedical knowledge graphs to quantify support for the explanations from clinical literature. We evaluate our method in a case study on predicting onset of sepsis in intensive care units. Our analysis shows that the scores obtained using our method corroborate with independent evidence from clinical literature and have the required qualities expected from such a metric. Thus, our method can be used to evaluate and select useful explanations from a diverse set of XAI techniques in clinical contexts, making it a fundamental tool for future research in the design of AI-driven CDSS.

Optimally generate policy-based evidence before scaling.

Social scientists have increasingly turned to the experimental method to understand human behaviour. One critical issue that makes solving social problems difficult is scaling up the idea from a small group to a larger group in more diverse situations. The urgency of scaling policies impacts us every day, whether it is protecting the health and safety of a community or enhancing the opportunities of future generations. Yet, a common result is that, when we scale up ideas, most experience a 'voltage drop'-that is, on scaling, the cost-benefit profile depreciates considerably. Here I argue that, to reduce voltage drops, we must optimally generate policy-based evidence. Optimality requires answering two crucial questions: what information should be generated and in what sequence. The economics underlying the science of scaling provides insights into these questions, which are in some cases at odds with conventional approaches. For example, there are important situations in which I advocate flipping the traditional social science research model to an approach that, from the beginning, produces the type of policy-based evidence that the science of scaling demands. To do so, I propose augmenting efficacy trials by including relevant tests of scale in the original discovery process, which forces the scientist to naturally start with a recognition of the big picture: what information do I need to have scaling confidence?

More journal articles and fewer books: Publication practices in the social sciences in the 2010's.

The number of scholarly journal articles published each year is growing, but little is known about the relationship between journal article growth and other forms of scholarly dissemination (e.g., books and monographs). Journal articles are the de facto currency of evaluation and prestige in STEM fields, but social scientists routinely publish books as well as articles, representing a unique opportunity to study increased article publications in disciplines with other dissemination options. We studied the publishing activity of social science faculty members in 12 disciplines at 290 Ph.D. granting institutions in the United States between 2011 and 2019, asking: 1) have publication practices changed such that more or fewer books and articles are written now than in the recent past?; 2) has the percentage of scholars actively participating in a particular publishing type changed over time?; and 3) do different age cohorts evince different publication strategies? In all disciplines, journal articles per person increased between 3% and 64% between 2011 and 2019, while books per person decreased by at least 31% and as much as 54%. All age cohorts show increased article authorship over the study period, and early career scholars author more articles per person than the other cohorts in eight disciplines. The article-dominated literatures of the social sciences are becoming increasingly similar to those of STEM disciplines.

Human social sensing is an untapped resource for computational social science.

The ability to 'sense' the social environment and thereby to understand the thoughts and actions of others allows humans to fit into their social worlds, communicate and cooperate, and learn from others' experiences. Here we argue that, through the lens of computational social science, this ability can be used to advance research into human sociality. When strategically selected to represent a specific population of interest, human social sensors can help to describe and predict societal trends. In addition, their reports of how they experience their social worlds can help to build models of social dynamics that are constrained by the empirical reality of human social systems.

Meaningful measures of human society in the twenty-first century.

Science rarely proceeds beyond what scientists can observe and measure, and sometimes what can be observed proceeds far ahead of scientific understanding. The twenty-first century offers such a moment in the study of human societies. A vastly larger share of behaviours is observed today than would have been imaginable at the close of the twentieth century. Our interpersonal communication, our movements and many of our everyday actions, are all potentially accessible for scientific research; sometimes through purposive instrumentation for scientific objectives (for example, satellite imagery), but far more often these objectives are, literally, an afterthought (for example, Twitter data streams). Here we evaluate the potential of this massive instrumentation-the creation of techniques for the structured representation and quantification-of human behaviour through the lens of scientific measurement and its principles. In particular, we focus on the question of how we extract scientific meaning from data that often were not created for such purposes. These data present conceptual, computational and ethical challenges that require a rejuvenation of our scientific theories to keep up with the rapidly changing social realities and our capacities to capture them. We require, in other words, new approaches to manage, use and analyse data.

Integrating explanation and prediction in computational social science.

Computational social science is more than just large repositories of digital data and the computational methods needed to construct and analyse them. It also represents a convergence of different fields with different ways of thinking about and doing science. The goal of this Perspective is to provide some clarity around how these approaches differ from one another and to propose how they might be productively integrated. Towards this end we make two contributions. The first is a schema for thinking about research activities along two dimensions-the extent to which work is explanatory, focusing on identifying and estimating causal effects, and the degree of consideration given to testing predictions of outcomes-and how these two priorities can complement, rather than compete with, one another. Our second contribution is to advocate that computational social scientists devote more attention to combining prediction and explanation, which we call integrative modelling, and to outline some practical suggestions for realizing this goal.

Measuring algorithmically infused societies.

It has been the historic responsibility of the social sciences to investigate human societies. Fulfilling this responsibility requires social theories, measurement models and social data. Most existing theories and measurement models in the social sciences were not developed with the deep societal reach of algorithms in mind. The emergence of 'algorithmically infused societies'-societies whose very fabric is co-shaped by algorithmic and human behaviour-raises three key challenges: the insufficient quality of measurements, the complex consequences of (mis)measurements, and the limits of existing social theories. Here we argue that tackling these challenges requires new social theories that account for the impact of algorithmic systems on social realities. To develop such theories, we need new methodologies for integrating data and measurements into theory construction. Given the scale at which measurements can be applied, we believe measurement models should be trustworthy, auditable and just. To achieve this, the development of measurements should be transparent and participatory, and include mechanisms to ensure measurement quality and identify possible harms. We argue that computational social scientists should rethink what aspects of algorithmically infused societies should be measured, how they should be measured, and the consequences of doing so.

Urban Emotion Sensing Beyond 'Affective Capture': Advancing Critical Interdisciplinary Methods.

The use of mobile sensor methodologies in urban analytics to study 'urban emotions' is currently outpacing the science required to rigorously interpret the data generated. Interdisciplinary research on 'urban stress' could help inform urban wellbeing policies relating to healthier commuting and alleviation of work stress. The purpose of this paper is to address-through methodological experimentation-ethical, political and conceptual issues identified by critical social scientists with regards to emotion tracking, wearables and data analytics. We aim to encourage more dialogue between the critical approach and applied environmental health research. The definition of stress is not unambiguous or neutral and is mediated by the very technologies we use for research. We outline an integrative methodology in which we combine pilot field research using biosensing technologies, a novel method for identifying 'moments of stress' in a laboratory setting, psychometric surveys and narrative interviews on workplace and commuter stress in urban environments.

A data driven methodology for social science research with left-behind children as a case study.

For decades, traditional correlation analysis and regression models have been used in social science research. However, the development of machine learning algorithms makes it possible to apply machine learning techniques for social science research and social issues, which may outperform standard regression methods in some cases. Under the circumstances, this article proposes a methodological workflow for data analysis by machine learning techniques that have the possibility to be widely applied in social issues. Specifically, the workflow tries to uncover the natural mechanisms behind the social issues through a data-driven perspective from feature selection to model building. The advantage of data-driven techniques in feature selection is that the workflow can be built without so much restriction of related knowledge and theory in social science. The advantage of using machine learning techniques in modelling is to uncover non-linear and complex relationships behind social issues. The main purpose of our methodological workflow is to find important fields relevant to the target and provide appropriate predictions. However, to explain the result still needs theory and knowledge from social science. In this paper, we trained a methodological workflow with left-behind children as the social issue case, and all steps and full results are included.

Data analysis and modeling pipelines for controlled networked social science experiments.

There is large interest in networked social science experiments for understanding human behavior at-scale. Significant effort is required to perform data analytics on experimental outputs and for computational modeling of custom experiments. Moreover, experiments and modeling are often performed in a cycle, enabling iterative experimental refinement and data modeling to uncover interesting insights and to generate/refute hypotheses about social behaviors. The current practice for social analysts is to develop tailor-made computer programs and analytical scripts for experiments and modeling. This often leads to inefficiencies and duplication of effort. In this work, we propose a pipeline framework to take a significant step towards overcoming these challenges. Our contribution is to describe the design and implementation of a software system to automate many of the steps involved in analyzing social science experimental data, building models to capture the behavior of human subjects, and providing data to test hypotheses. The proposed pipeline framework consists of formal models, formal algorithms, and theoretical models as the basis for the design and implementation. We propose a formal data model, such that if an experiment can be described in terms of this model, then our pipeline software can be used to analyze data efficiently. The merits of the proposed pipeline framework is elaborated by several case studies of networked social science experiments.