How good are large language models at product risk assessment?

Product safety professionals must assess the risks to consumers associated with the foreseeable uses and misuses of products. In this study, we investigate the utility of generative artificial intelligence (AI), specifically large language models (LLMs) such as ChatGPT, across a number of tasks involved in the product risk assessment process. For a set of six consumer products, prompts were developed related to failure mode identification, the construction and population of a failure mode and effects analysis (FMEA) table, risk mitigation identification, and guidance to product designers, users, and regulators. These prompts were input into ChatGPT and the outputs were recorded. A survey was administered to product safety professionals to ascertain the quality of the outputs. We found that ChatGPT generally performed better at divergent thinking tasks such as brainstorming potential failure modes and risk mitigations. However, there were errors and inconsistencies in some of the results, and the guidance provided was perceived as overly generic, occasionally outlandish, and not reflective of the depth of knowledge held by a subject matter expert. When tested against a sample of other LLMs, similar patterns in strengths and weaknesses were demonstrated. Despite these challenges, a role for LLMs may still exist in product risk assessment to assist in ideation, while experts may shift their focus to critical review of AI-generated content.

Defective Assistive Device Involvement in Older Adult Emergency Department Visits.

Background and Objectives: Many older adults adopt equipment to address physical limitations and reduce dependence on others to complete basic activities of daily living. Although a few prior studies have considered injuries associated with assistive devices for older adults, those studies focused on older adults' health and functional risks for injury. There is limited analysis of older adult injuries involving defective or malfunctioning assistive devices. Research Design and Methods: Data from this study are from the National Electronic Surveillance System All Injury Program which collected data on consumer product-related injuries from a probability sample of 66 hospital Emergency Departments across the United States. Data from 30 776 older adult Emergency Department (ED) injury narratives from 2016 to 2020 were coded according to the assistive device involved and whether malfunctioning led to the injury. The study team manually examined all narratives in which the assistive device was coded to have malfunctioned. Results: A total of 10 974 older adult ED cases were treated for 12 488 injuries involving a defective device. Injuries included 4 212 head and neck injuries (eg, concussion), 4 317 trunk injuries (eg, hip fractures), and 3 959 arm or leg injuries (eg, leg fracture). Of these patients, 4 586 were admitted to a hospital ward for further evaluation and treatment. Seventy percent of these patients were injured while using a walker; in contrast, wheelchairs were implicated in only 4% of the above cases. Design flaws were identified in 8 158 cases and part breakage/decoupling incidents in 2 816 cases. Discussion and Implications: Our findings provide evidence that assistive devices are actively involved in older adult injuries. Further research is needed to reduce injuries associated with assistive devices by educating patients and their careproviders about device use and assembly and developing effective methods for informing manufacturers about malfunctioning devices.

A semiautomated risk assessment method for consumer products.

In this study, we develop a model that assesses product risk using online reviews from Amazon.com. We first identify unique words and phrases capable of identifying hazards. Second, we estimate risk severity using hazard type weights and risk likelihood using total reviews as a proxy for sales volume. In addition, we obtain expert assessments of product hazard risk (risk likelihood and severity) from a sample of high- and low-risk consumer products identified by a computerized risk assessment model we have developed. Third, we assess the validity of our computerized product risk assessment scoring model by utilizing the experts' survey responses. We find that our model is especially consistent with expert judgments of hazard likelihood but not as consistent with expert judgments of hazard severity. This model helps organizations to determine the risk severity, risk likelihood, and overall risk level of a specific product. The model produced by this study is helpful for product safety practitioners in product risk identification, characterization, and mitigation.

Safety Concerns in Mobility-Assistive Products for Older Adults: Content Analysis of Online Reviews.

BACKGROUND: Older adults who have difficulty moving around are commonly advised to adopt mobility-assistive devices to prevent injuries. However, limited evidence exists on the safety of these devices. Existing data sources such as the National Electronic Injury Surveillance System tend to focus on injury description rather than the underlying context, thus providing little to no actionable information regarding the safety of these devices. Although online reviews are often used by consumers to assess the safety of products, prior studies have not explored consumer-reported injuries and safety concerns within online reviews of mobility-assistive devices. OBJECTIVE: This study aimed to investigate injury types and contexts stemming from the use of mobility-assistive devices, as reported by older adults or their caregivers in online reviews. It not only identified injury severities and mobility-assistive device failure pathways but also shed light on the development of safety information and protocols for these products. METHODS: Reviews concerning assistive devices were extracted from the "assistive aid" categories, which are typically intended for older adult use, on Amazon's US website. The extracted reviews were filtered so that only those pertaining to mobility-assistive devices (canes, gait or transfer belts, ramps, walkers or rollators, and wheelchairs or transport chairs) were retained. We conducted large-scale content analysis of these 48,886 retained reviews by coding them according to injury type (no injury, potential future injury, minor injury, and major injury) and injury pathway (device critical component breakage or decoupling; unintended movement; instability; poor, uneven surface handling; and trip hazards). Coding efforts were carried out across 2 separate phases in which the team manually verified all instances coded as minor injury, major injury, or potential future injury and established interrater reliability to validate coding efforts. RESULTS: The content analysis provided a better understanding of the contexts and conditions leading to user injury, as well as the severity of injuries associated with these mobility-assistive devices. Injury pathways-device critical component failures; unintended device movement; poor, uneven surface handling; instability; and trip hazards-were identified for 5 product types (canes, gait and transfer belts, ramps, walkers and rollators, and wheelchairs and transport chairs). Outcomes were normalized per 10,000 posting counts (online reviews) mentioning minor injury, major injury, or potential future injury by product category. Overall, per 10,000 reviews, 240 (2.4%) described mobility-assistive equipment-related user injuries, whereas 2318 (23.18%) revealed potential future injuries. CONCLUSIONS: This study highlights mobility-assistive device injury contexts and severities, suggesting that consumers who posted online reviews attribute most serious injuries to a defective item, rather than user misuse. It implies that many mobility-assistive device injuries may be preventable through patient and caregiver education on how to evaluate new and existing equipment for risk of potential future injury.

Injury prevention for older adults: A dataset of safety concern narratives from online reviews of mobility-related products.

Older adults are among the fastest-growing demographic groups in the United States, increasing by over a third this past decade. Consequently, the older adult consumer product market has quickly become a multi-billion-dollar industry in which millions of products are sold every year. However, the rapidly growing market raises the potential for an increasing number of product safety concerns and consumer product-related injuries among older adults. Recent manufacturer and consumer injury prevention efforts have begun to turn towards online reviews, as these provide valuable information from which actionable, timely intelligence can be derived and used to detect safety concerns and prevent injury. The presented dataset contains 1966 curated online product reviews from consumers, equally distributed between safety concerns and non-concerns, pertaining to product categories typically intended for older adults. Identified safety concerns were manually sub-coded across thirteen dimensions designed to capture relevant aspects of the consumer's experience with the purchased product, facilitate the safety concern identification and sub-classification process, and serve as a gold-standard, balanced dataset for text classifier learning.

Text Mining Approaches for Postmarket Food Safety Surveillance Using Online Media.

Food contamination and food poisoning pose enormous risks to consumers across the world. As discussions of consumer experiences have spread through online media, we propose the use of text mining to rapidly screen online media for mentions of food safety hazards. We compile a large data set of labeled consumer posts spanning two major websites. Utilizing text mining and supervised machine learning, we identify unique words and phrases in online posts that identify consumers' interactions with hazardous food products. We compare our methods to traditional sentiment-based text mining. We assess performance in a high-volume setting, utilizing a data set of over 4 million online reviews. Our methods were 77-90% accurate in top-ranking reviews, while sentiment analysis was just 11-26% accurate. Moreover, we aggregate review-level results to make product-level risk assessments. A panel of 21 food safety experts assessed our model's hazard-flagged products to exhibit substantially higher risk than baseline products. We suggest the use of these tools to profile food items and assess risk, building a postmarket decision support system to identify hazardous food products. Our research contributes to the literature and practice by providing practical and inexpensive means for rapidly monitoring food safety in real time.

An investigation into online videos as a source of safety hazard reports.

INTRODUCTION: Despite the advantages of video-based product reviews relative to text-based reviews in detecting possible safety hazard issues, video-based product reviews have received no attention in prior literature. This study focuses on online video-based product reviews as possible sources to detect safety hazards. METHODS: We use two common text mining methods - sentiment and smoke words - to detect safety issues mentioned in videos on the world's most popular video sharing platform, YouTube. RESULTS: 15,402 product review videos from YouTube were identified as containing either negative sentiment or smoke words, and were carefully manually viewed to verify whether hazards were indeed mentioned. 496 true safety issues (3.2%) were found. Out of 9,453 videos that contained smoke words, 322 (3.4%) mentioned safety issues, vs. only 174 (2.9%) of the 5,949 videos with negative sentiment words. Only 1% of randomly-selected videos mentioned safety hazards. CONCLUSIONS: Comparing the number of videos with true safety issues that contain sentiment words vs. smoke words in their title or description, we show that smoke words are a more accurate predictor of safety hazards in video-based product reviews than sentiment words. This research also discovers words that are indicative of true hazards versus false positives in online video-based product reviews. Practical applications: The smoke words lists and word sub-groups generated in this paper can be used by manufacturers and consumer product safety organizations to more efficiently identify product safety issues from online videos. This project also provides realistic baselines for resource estimates for future projects that aim to discover safety issues from online videos or reviews.

Automated discovery of safety and efficacy concerns for joint & muscle pain relief treatments from online reviews.

OBJECTIVES: Product issues can cost companies millions in lawsuits and have devastating effects on a firm's sales, image and goodwill, especially in the era of social media. The ability for a system to detect the presence of safety and efficacy (S&E) concerns early on could not only protect consumers from injuries due to safety hazards, but could also mitigate financial damage to the manufacturer. Prior studies in the field of automated defect discovery have found industry-specific techniques appropriate to the automotive, consumer electronics, home appliance, and toy industries, but have not investigated pain relief medicines and medical devices. In this study, we focus specifically on automated discovery of S&E concerns in over-the-counter (OTC) joint and muscle pain relief remedies and devices. METHODS: We select a dataset of over 32,000 records for three categories of Joint & Muscle Pain Relief treatments from Amazon's online product reviews, and train "smoke word" dictionaries which we use to score holdout reviews, for the presence of safety and efficacy issues. We also score using conventional sentiment analysis techniques. RESULTS: Compared to traditional sentiment analysis techniques, we found that smoke term dictionaries were better suited to detect product concerns from online consumer reviews, and significantly outperformed the sentiment analysis techniques in uncovering both efficacy and safety concerns, across all product subcategories. CONCLUSION: Our research can be applied to the healthcare and pharmaceutical industry in order to detect safety and efficacy concerns, reducing risks that consumers face using these products. These findings can be highly beneficial to improving quality assurance and management in joint and muscle pain relief.

Pelvic Breadth and Locomotor Kinematics in Human Evolution.

A broad pelvis is characteristic of most, if not all, pre-modern hominins. In at least some early australopithecines, most notably the female Australopithecus afarensis specimen known as "Lucy," it is very broad and coupled with very short lower limbs. In 1991, Rak suggested that Lucy's pelvic anatomy improved locomotor efficiency by increasing stride length through rotation of the wide pelvis in the axial plane. Compared to lengthening strides by increasing flexion and extension at the hips, this mechanism could avoid potentially costly excessive vertical oscillations of the body's center of mass (COM). Here, we test this hypothesis. We examined 3D kinematics of walking at various speeds in 26 adult subjects to address the following questions: Do individuals with wider pelves take longer strides, and do they use a smaller degree of hip flexion and extension? Is pelvic rotation greater in individuals with shorter legs, and those with narrower pelves? Our results support Rak's hypothesis. Subjects with wider pelves do take longer strides for a given velocity, and for a given stride length they flex and extend their hips less, suggesting a smoother pathway of the COM. Individuals with shorter legs do use more pelvic rotation when walking, but pelvic breadth was not related to pelvic rotation. These results suggest that a broad pelvis could benefit any bipedal hominin, but especially a short-legged australopithecine such as Lucy, by improving locomotor efficiency, particularly when carrying an infant or traveling in a foraging group with individuals of varying sizes. Anat Rec, 300:739-751, 2017. © 2017 Wiley Periodicals, Inc.

TOY SAFETY SURVEILLANCE FROM ONLINE REVIEWS.

Toy-related injuries account for a significant number of childhood injuries and the prevention of these injuries remains a goal for regulatory agencies and manufacturers. Text-mining is an increasingly prevalent method for uncovering the significance of words using big data. This research sets out to determine the effectiveness of text-mining in uncovering potentially dangerous children's toys. We develop a danger word list, also known as a 'smoke word' list, from injury and recall text narratives. We then use the smoke word lists to score over one million Amazon reviews, with the top scores denoting potential safety concerns. We compare the smoke word list to conventional sentiment analysis techniques, in terms of both word overlap and effectiveness. We find that smoke word lists are highly distinct from conventional sentiment dictionaries and provide a statistically significant method for identifying safety concerns in children's toy reviews. Our findings indicate that text-mining is, in fact, an effective method for the surveillance of safety concerns in children's toys and could be a gateway to effective prevention of toy-product-related injuries.