Balancing media selections over time: Emotional valence, informational content, and time intervals of use.

The sequencing of information in media can influence processing of content via mechanisms like framing, mood management, and emotion regulation. This study examined three kinds of media sequences on smartphones: (1) balancing positive and negative emotional content; (2) balancing emotional content with informational content; and (3) balancing time spent on and off the media device. Actual media use was measured in natural settings using the Screenomics framework which gathers screenshots from smartphones every 5 s when devices are on. Time-series analyses of 223,531 smartphone sessions recorded from 94 participants showed that emotionally positive content was more likely to follow negative content, and that emotionally negative content was more likely to follow positive content; emotional content was more likely to follow informational content, and informational content was more likely to follow emotional content; and longer smartphone sessions were more likely to follow longer periods of non-use.

The Affective Dynamics of Everyday Digital Life: Opening Computational Possibility.

Up to now, there was no way to observe and track the affective impacts of the massive amount of complex visual stimuli that people encounter "in the wild" during their many hours of digital life. In this paper, we propose and illustrate how recent advances in AI-trained ensembles of deep neural networks-can be deployed on new data streams that are long sequences of screenshots of study participants' smartphones obtained unobtrusively during everyday life. We obtained affective valence and arousal ratings of hundreds of images drawn from existing picture repositories often used in psychological studies, and a new screenshot repository chronicling individuals' everyday digital life from both N = 832 adults and an affect computation model (Parry & Vuong, 2021). Results and analysis suggest that (a) our sample rates images similarly to other samples used in psychological studies, (b) the affect computation model is able to assign valence and arousal ratings similarly to humans, and (c) the resulting computational pipeline can be deployed at scale to obtain detailed maps of the affective space individuals travel through on their smartphones. Leveraging innovative methods for tracking the emotional content individuals encounter on their smartphones, we open the possibility for large-scale studies of how the affective dynamics of everyday digital life shape individuals' moment-to-moment experiences and well-being. Supplementary Information: The online version contains supplementary material available at 10.1007/s42761-023-00202-4.

Binding the Person-Specific Approach to Modern AI in the Human Screenome Project: Moving past Generalizability to Transferability.

Advances in ability to comprehensively record individuals' digital lives and in AI modeling of those data facilitate new possibilities for describing, predicting, and generating a wide variety of behavioral processes. In this paper, we consider these advances from a person-specific perspective, including whether the pervasive concerns about generalizability of results might be productively reframed with respect to transferability of models, and how self-supervision and new deep neural network architectures that facilitate transfer learning can be applied in a person-specific way to the super-intensive longitudinal data arriving in the Human Screenome Project. In developing the possibilities, we suggest Molenaar add a statement to the person-specific Manifesto - "In short, the concerns about generalizability commonly leveled at the person-specific paradigm are unfounded and can be fully and completely replaced with discussion and demonstrations of transferability."

Media Production on Smartphones: Analysis of the Timing, Content, and Context of Message Production Using Real-World Smartphone Use Data.

Although media production is a critical concept in communication theory, we know surprisingly little about the timing, content, and context of individuals' production behavior. Intensive observation and analysis of 94 American adults' smartphone use over 1 week showed that although time spent in producing content was on average only about 6 percent of the amount of time spent on smartphones, the production content was more purposeful, expressive, articulate, condensed, confident, personal, and emotionally charged than consumption content. Analysis of the temporal dynamics of production suggests that the content consumed in the minute before individuals' production began to resemble the subsequently produced content. Other results suggest that content production on smartphones was fragmented, idiosyncratic, and purposeful, highlighting the impact of individuals' quick interactions with media, and the need to develop user-centric theories about how, when, and why individuals produce digital content.

Virtual and real: Symbolic and natural experiences with social robots.

Interactions with social robots are symbolic experiences guided by the pretense that robots depict real people. But they can also be natural experiences that are direct, automatic, and independent of any thoughtful mapping between what is real and depicted. Both experiences are important, both may apply within the same interaction, and they may vary within a person over time.

Screenomics: A Framework to Capture and Analyze Personal Life Experiences and the Ways that Technology Shapes Them.

Digital experiences capture an increasingly large part of life, making them a preferred, if not required, method to describe and theorize about human behavior. Digital media also shape behavior by enabling people to switch between different content easily, and create unique threads of experiences that pass quickly through numerous information categories. Current methods of recording digital experiences provide only partial reconstructions of digital lives that weave - often within seconds - among multiple applications, locations, functions and media. We describe an end-to-end system for capturing and analyzing the "screenome" of life in media, i.e., the record of individual experiences represented as a sequence of screens that people view and interact with over time. The system includes software that collects screenshots, extracts text and images, and allows searching of a screenshot database. We discuss how the system can be used to elaborate current theories about psychological processing of technology, and suggest new theoretical questions that are enabled by multiple time scale analyses. Capabilities of the system are highlighted with eight research examples that analyze screens from adults who have generated data within the system. We end with a discussion of future uses, limitations, theory and privacy.

The Idiosyncrasies of Everyday Digital Lives: Using the Human Screenome Project to Study User Behavior on Smartphones.

Most methods used to make theory-relevant observations of technology use rely on self-report or application logging data where individuals' digital experiences are purposively summarized into aggregates meant to describe how the average individual engages with broadly defined segments of content. This aggregation and averaging masks heterogeneity in how and when individuals actually engage with their technology. In this study, we use screenshots (N > 6 million) collected every five seconds that were sequenced and processed using text and image extraction tools into content-, context-, and temporally-informative "screenomes" from 132 smartphone users over several weeks to examine individuals' digital experiences. Analyses of screenomes highlight extreme between-person and within-person heterogeneity in how individuals switch among and titrate their engagement with different content. Our simple quantifications of textual and graphical content and flow throughout the day illustrate the value screenomes have for the study of individuals' smartphone use and the cognitive and psychological processes that drive use. We demonstrate how temporal, textual, graphical, and topical features of people's smartphone screens can lay the foundation for expanding the Human Screenome Project with full-scale mining that will inform researchers' knowledge of digital life.

Screenomics: A New Approach for Observing and Studying Individuals' Digital Lives.

AIMS: This study describes when and how adolescents engage with their fast-moving and dynamic digital environment as they go about their daily lives. We illustrate a new approach - screenomics - for capturing, visualizing, and analyzing screenomes, the record of individuals' day-to-day digital experiences. SAMPLE: Over 500,000 smartphone screenshots provided by four Latino/Hispanic youth, age 14-15 years, from low-income, racial/ethnic minority neighborhoods. METHOD: Screenomes collected from smartphones for one to three months, as sequences of smartphone screenshots obtained every five seconds that the device is activated, are analyzed using computational machinery for processing images and text, machine learning algorithms, human-labeling, and qualitative inquiry. FINDINGS: Adolescents' digital lives differ substantially across persons, days, hours, and minutes. Screenomes highlight the extent of switching among multiple applications, and how each adolescent is exposed to different content at different times for different durations - with apps, food-related content, and sentiment as illustrative examples. IMPLICATIONS: We propose that the screenome provides the fine granularity of data needed to study individuals' digital lives, for testing existing theories about media use, and for generation of new theory about the interplay between digital media and development.

#Science: The Potential and the Challenges of Utilizing Social Media and Other Electronic Communication Platforms in Health Care.

Electronic communication is becoming increasingly popular worldwide, as evidenced by its widespread and rapidly growing use. In medicine, however, it remains a novel approach to reach out to patients. Yet, they have the potential for further improving current health care. Electronic platforms could support therapy adherence and communication between physicians and patients. The power of social media as well as other electronic devices can improve adherence as evidenced by the development of the app bant. Additionally, systemic analysis of social media content by Screenome can identify health events not always captured by regular health care. By better identifying these healthcare events we can improve our current healthcare system as we will be able to better tailor to the patients' needs. All these techniques are a valuable component of modern health care and will help us into the future of increasingly digital health care.

Psychological and physiological effects of applying self-control to the mobile phone.

This preregistered study examined the psychological and physiological consequences of exercising self-control with the mobile phone. A total of 125 participants were randomly assigned to sit in an unadorned room for six minutes and either (a) use their mobile phone, (b) sit alone with no phone, or (c) sit with their device but resist using it. Consistent with prior work, participants self-reported more concentration difficulty and more mind wandering with no device present compared to using the phone. Resisting the phone led to greater perceived concentration abilities than sitting without the device (not having external stimulation). Failing to replicate prior work, however, participants without external stimulation did not rate the experience as less enjoyable or more boring than having something to do. We also observed that skin conductance data were consistent across conditions for the first three-minutes of the experiment, after which participants who resisted the phone were less aroused than those who were without the phone. We discuss how the findings contribute to our understanding of exercising self-control with mobile media and how psychological consequences, such as increased mind wandering and focusing challenges, relate to periods of idleness or free thinking.