Health equity assessment of machine learning performance (HEAL): a framework and dermatology AI model case study.

Background: Artificial intelligence (AI) has repeatedly been shown to encode historical inequities in healthcare. We aimed to develop a framework to quantitatively assess the performance equity of health AI technologies and to illustrate its utility via a case study. Methods: Here, we propose a methodology to assess whether health AI technologies prioritise performance for patient populations experiencing worse outcomes, that is complementary to existing fairness metrics. We developed the Health Equity Assessment of machine Learning performance (HEAL) framework designed to quantitatively assess the performance equity of health AI technologies via a four-step interdisciplinary process to understand and quantify domain-specific criteria, and the resulting HEAL metric. As an illustrative case study (analysis conducted between October 2022 and January 2023), we applied the HEAL framework to a dermatology AI model. A set of 5420 teledermatology cases (store-and-forward cases from patients of 20 years or older, submitted from primary care providers in the USA and skin cancer clinics in Australia), enriched for diversity in age, sex and race/ethnicity, was used to retrospectively evaluate the AI model's HEAL metric, defined as the likelihood that the AI model performs better for subpopulations with worse average health outcomes as compared to others. The likelihood that AI performance was anticorrelated to pre-existing health outcomes was estimated using bootstrap methods as the probability that the negated Spearman's rank correlation coefficient (i.e., "R") was greater than zero. Positive values of R suggest that subpopulations with poorer health outcomes have better AI model performance. Thus, the HEAL metric, defined as p (R >0), measures how likely the AI technology is to prioritise performance for subpopulations with worse average health outcomes as compared to others (presented as a percentage below). Health outcomes were quantified as disability-adjusted life years (DALYs) when grouping by sex and age, and years of life lost (YLLs) when grouping by race/ethnicity. AI performance was measured as top-3 agreement with the reference diagnosis from a panel of 3 dermatologists per case. Findings: Across all dermatologic conditions, the HEAL metric was 80.5% for prioritizing AI performance of racial/ethnic subpopulations based on YLLs, and 92.1% and 0.0% respectively for prioritizing AI performance of sex and age subpopulations based on DALYs. Certain dermatologic conditions were significantly associated with greater AI model performance compared to a reference category of less common conditions. For skin cancer conditions, the HEAL metric was 73.8% for prioritizing AI performance of age subpopulations based on DALYs. Interpretation: Analysis using the proposed HEAL framework showed that the dermatology AI model prioritised performance for race/ethnicity, sex (all conditions) and age (cancer conditions) subpopulations with respect to pre-existing health disparities. More work is needed to investigate ways of promoting equitable AI performance across age for non-cancer conditions and to better understand how AI models can contribute towards improving equity in health outcomes. Funding: Google LLC.

The performance of 11 fingertip pulse oximeters during hypoxemia in healthy human participants with varied, quantified skin pigment.

BACKGROUND: Fingertip pulse oximeters are widely available, inexpensive, and commonly used to make clinical decisions in many settings. Device performance is largely unregulated and poorly characterised, especially in people with dark skin pigmentation. METHODS: Eleven popular fingertip pulse oximeters were evaluated using the US Food and Drug Administration (FDA) Guidance (2013) and International Organization for Standardization Standards (ISO, 2017) in 34 healthy humans with diverse skin pigmentation utilising a controlled desaturation study with arterial oxygen saturation (SaO 2) plateaus between 70% and 100%. Skin pigmentation was assessed subjectively using a perceived Fitzpatrick Scale (pFP) and objectively using the individual typology angle (ITA) via spectrophotometry at nine anatomical sites. FINDINGS: Five of 11 devices had a root mean square error (ARMS) > 3%, falling outside the acceptable FDA performance range. Nine devices demonstrated worse performance in participants in the darkest skin pigmentation category compared with those in the lightest category. A commonly used subjective skin colour scale frequently miscategorised participants as being darkly pigmented when compared to objective quantification of skin pigment by ITA. INTERPRETATION: Fingertip pulse oximeters have variable performance, frequently not meeting regulatory requirements for clinical use, and occasionally contradicting claims made by manufacturers. Most devices showed a trend toward worse performance in participants with darker skin pigment. Regulatory standards do not adequately account for the impact of skin pigmentation on device performance. We recommend that the pFP and other non-standardised subjective skin colour scales should no longer be used for defining diversity of skin pigmentation. Reliable methods for characterising skin pigmentation to improve diversity and equitable performance of pulse oximeters are needed. FUNDING: This study was conducted as part of the Open Oximetry Project funded by the Gordon and Betty Moore Foundation, Patrick J McGovern Foundation, and Robert Wood Johnson Foundation. The UCSF Hypoxia Research Laboratory receives funding from multiple industry sponsors to test the sponsors' devices for the purposes of product development and regulatory performance testing. Data in this paper do not include sponsor's study devices. All data were collected from devices procured by the Hypoxia Research Laboratory for the purposes of independent research. No company provided any direct funding for this study, participated in study design or analysis, or was involved in analysing data or writing the manuscript. None of the authors own stock or equity interests in any pulse oximeter companies. Dr Ellis Monk's time utilised for data analysis, reviewing and editing was funded by grant number: DP2MH132941.

Integrating skin color assessments into clinical practice and research: A review of current approaches.

Skin color classification can have importance in skin health, pigmentary disorders, and oncologic condition assessments. It is also critical for evaluating disease course and response to a variety of therapeutic interventions and aids in accurate classification of participants in clinical research studies. A panel of dermatologists conducted a literature review to assess the strengths and limitations of existing classification scales, as well as to compare their preferences and utilities. We identified 17 skin classification systems utilized in dermatologic settings. These systems include a range of parameters such as UV light reactivity, race, ethnicity, and degree of pigmentation. The Fitzpatrick skin type classification is most widely used and validated. However it has numerous limitations including its conflation with race, ethnicity, and skin color. There is a lack of validation data available for the remaining scales. There are significant deficiencies in current skin classification instruments. Consensus-based initiatives to drive the development of validated and reliable tools are critically needed.

Treatment for central centrifugal cicatricial alopecia-Delphi consensus recommendations.

BACKGROUND: There is no established standard of care for treating central centrifugal cicatricial alopecia (CCCA), and treatment approaches vary widely. OBJECTIVE: To develop consensus statements regarding the use of various pharmacological therapies in treating adults with CCCA. METHODS: We invited 27 dermatologists with expertise in hair and scalp disorders to participate in a 3-round modified Delphi study between January and March 2023. Statements met strong consensus if 75% of respondents agreed or disagreed. Statements met moderate consensus if 55% or more but less than 75% agreed or disagreed. RESULTS: In round 1, 5 of 33 (15.2%) statements met strong consensus, followed by 9 of 28 (32.1%) in round 2. After the final round 3 meeting, strong consensus was reached for 20 of 70 (28.6%) overall statements. Two statements achieved moderate consensus. LIMITATIONS: This study included only English-speaking, US-based dermatologists and did not consider nonpharmacological therapies. CONCLUSION: Despite varying opinions among dermatologists, consensus was reached for several statements to help clinicians manage CCCA. We also highlight areas that lack expert consensus with the goal of advancing research and therapeutic options for CCCA.

Large language models propagate race-based medicine.

Large language models (LLMs) are being integrated into healthcare systems; but these models may recapitulate harmful, race-based medicine. The objective of this study is to assess whether four commercially available large language models (LLMs) propagate harmful, inaccurate, race-based content when responding to eight different scenarios that check for race-based medicine or widespread misconceptions around race. Questions were derived from discussions among four physician experts and prior work on race-based medical misconceptions believed by medical trainees. We assessed four large language models with nine different questions that were interrogated five times each with a total of 45 responses per model. All models had examples of perpetuating race-based medicine in their responses. Models were not always consistent in their responses when asked the same question repeatedly. LLMs are being proposed for use in the healthcare setting, with some models already connecting to electronic health record systems. However, this study shows that based on our findings, these LLMs could potentially cause harm by perpetuating debunked, racist ideas.

Skin Tone Analysis for Representation in Educational Materials (STAR-ED) using machine learning.

Images depicting dark skin tones are significantly underrepresented in the educational materials used to teach primary care physicians and dermatologists to recognize skin diseases. This could contribute to disparities in skin disease diagnosis across different racial groups. Previously, domain experts have manually assessed textbooks to estimate the diversity in skin images. Manual assessment does not scale to many educational materials and introduces human errors. To automate this process, we present the Skin Tone Analysis for Representation in EDucational materials (STAR-ED) framework, which assesses skin tone representation in medical education materials using machine learning. Given a document (e.g., a textbook in .pdf), STAR-ED applies content parsing to extract text, images, and table entities in a structured format. Next, it identifies images containing skin, segments the skin-containing portions of those images, and estimates the skin tone using machine learning. STAR-ED was developed using the Fitzpatrick17k dataset. We then externally tested STAR-ED on four commonly used medical textbooks. Results show strong performance in detecting skin images (0.96 ± 0.02 AUROC and 0.90 ± 0.06 F1 score) and classifying skin tones (0.87 ± 0.01 AUROC and 0.91 ± 0.00 F1 score). STAR-ED quantifies the imbalanced representation of skin tones in four medical textbooks: brown and black skin tones (Fitzpatrick V-VI) images constitute only 10.5% of all skin images. We envision this technology as a tool for medical educators, publishers, and practitioners to assess skin tone diversity in their educational materials.

Qualitative exploration of melanoma awareness in black people in the USA.

OBJECTIVE: Although black patients are more likely to have advanced melanomas at diagnosis, with a 5-year survival rate among black patients of 70% compared with 92% for white patients, black people are generally not the focus of melanoma public health campaigns. We sought to explore awareness and perspectives of melanoma among black people to inform the development of relevant and valued public health messages to promote early detection of melanoma. DESIGN: Inductive thematic analysis of in-depth semistructured interviews. SETTING: Interviews were conducted with participants via video software or telephone in the USA. PARTICIPANTS: Participants were adults from the USA who self-identified as African American or black. Recruitment flyers were posted around the San Francisco Bay Area and shared on our team Facebook page, with further participants identified through snowball sampling. RESULTS: We interviewed 26 participants from 10 different states. Overall, 12 were men and 14 were women, with a mean age of 43 years (range 18-85). We identified five key themes regarding melanoma awareness in black people: (1) lack of understanding of term 'melanoma' and features of skin cancer; (2) do not feel at risk of melanoma skin cancer; (3) surprise that melanoma can occur on palms, soles and nails; (4) skin cancer awareness messages do not apply to or include black people; and (5) Importance of relationship with healthcare and habits of utilisation. CONCLUSIONS: Analysis of these in-depth semistructured interviews illuminate the pressing need for health information on melanoma designed specifically for black people. We highlight two key points for focused public health messaging: (1) melanoma skin cancer does occur in black people and (2) high-risk sites for melanoma in black people include the palms, soles and nail beds. Therefore, public health messages for black people and their healthcare providers may involve productively checking these body surface areas.

Diversity and Career Goals of Graduating Allopathic Medical Students Pursuing Careers in Dermatology.

Importance: Dermatology is one of the least diverse specialties, while patients from minority racial and ethnic groups and other underserved populations continue to face numerous dermatology-specific health and health care access disparities in the US. Objectives: To examine the demographic characteristics and intended career goals of graduating US allopathic medical students pursuing careers in dermatology compared with those pursuing other specialties and whether these differ by sex, race and ethnicity, and/or sexual orientation. Design, Setting, and Participants: This secondary analysis of a repeated cross-sectional study included 58 077 graduating allopathic medical students using data from the 2016 to 2019 Association of American Medical Colleges Graduation Questionnaires. Main Outcomes and Measures: The proportion of female students, students from racial and ethnic groups underrepresented in medicine (URM), and sexual minority (SM) students pursuing dermatology vs pursuing other specialties. The proportions and multivariable-adjusted odds of intended career goals between students pursuing dermatology and those pursuing other specialties and by sex, race and ethnicity, and sexual orientation among students pursuing dermatology. Results: A total of 58 077 graduating students were included, with 28 489 (49.0%) female students, 8447 (14.5%) URM students, and 3641 (6.3%) SM students. Female students pursuing dermatology were less likely than female students pursuing other specialties to identify as URM (96 of 829 [11.6%] vs 4760 of 27 660 [17.2%]; P < .001) or SM (16 [1.9%] vs 1564 [5.7%]; P < .001). In multivariable-adjusted analyses, students pursuing dermatology compared with other specialties had decreased odds of intending to care for underserved populations (247 of 1350 [18.3%] vs 19 142 of 56 343 [34.0%]; adjusted odd ratio [aOR], 0.40; 95% CI, 0.35-0.47; P < .001), practice in underserved areas (172 [12.7%] vs 14 570 [25.9%]; aOR, 0.40; 95% CI, 0.34-0.47; P < .001), and practice public health (230 [17.0%] vs 17 028 [30.2%]; aOR, 0.44; 95% CI, 0.38-0.51; P < .001) but increased odds of pursuing research (874 [64.7%] vs 29 121 [51.7%]; aOR, 1.76; 95% CI, 1.57-1.97; P < .001) in their careers. Among students pursuing dermatology, female, URM, and SM identities were independently associated with increased odds of caring for underserved populations (eg, URM: aOR, 4.05; 95% CI, 2.83-5.80) and practicing public health (eg, SM: aOR, 2.55; 95% CI, 1.51-4.31). URM students compared with non-URM students pursuing dermatology had increased odds of intending to practice in underserved areas (aOR, 3.93; 95% CI, 2.66-5.80), and SM students compared with heterosexual students pursuing dermatology had increased odds of intending to become medical school faculty (aOR, 1.60; 95% CI, 1.01-2.57), to pursue administrative roles (aOR, 1.60; 95% CI, 1.01-2.59), and to conduct research (aOR, 1.73; 95% CI, 1.01-2.98). Conclusions and Relevance: The findings of this cross-sectional study suggest that diversity gaps continue to exist in the dermatology workforce pipeline. Efforts are needed to increase racial and ethnic and sexual orientation diversity and interest in careers focused on underserved care and public health among students pursuing dermatology.

Evaluation of Hidradenitis Suppurativa Diagnostic Criteria in Pediatric Patients.

Importance: Hidradenitis suppurativa (HS) is associated with considerable diagnostic delay. Although most patients report adolescent onset, existing HS diagnostic criteria may not adequately capture disease in pediatric populations. Objectives: To determine the proportion of physician-diagnosed pediatric patients with HS who met diagnostic criteria, and describe demographics, disease characteristics, and diagnostic patterns among pediatric patients with HS. Design, Setting, and Participants: In this retrospective, cross-sectional study, electronic medical records from 2 sites of a single academic tertiary care center were included. Eligible patients were those born after January 1, 1993, and assigned International Classification of Diseases, Ninth and Tenth Revisions (ICD-9/10) codes for HS (ICD-9 705.83/ICD-10 L73.2) between January 1, 2012, and July 1, 2021. Patients were excluded if they were older than 18 years at diagnosis, had inaccessible diagnostic visit notes, or were unintentionally assigned an HS ICD code. Exposures: Pediatric patients with HS. Main Outcomes and Measures: Fulfillment of diagnostic criteria in pediatric patients with HS. Results: A total of 297 adolescents with HS were included in the study; 123 patients were female (78.1%), 78 self-identified as Black (26.3%), and 116 self-identified as Hispanic (39.1%). The median (IQR) age at diagnosis was 14.0 (13.0-16.0) years. Documentation from the diagnostic visit demonstrated that 127 (42.8%) patients did not meet all 3 major HS diagnostic criteria. Of these patients, 122 (96.1%) did not meet the recurrence interval criterion (≥2 lesions within 6 months). Overall, 96 patients who did not meet the recurrence interval criterion had documentation from additional visits in the health system; 59 (61.5%) had documentation of 1 or more additional lesions consistent with HS. Review of these additional records demonstrated that 26 of these 59 (44.1%) patients met the recurrence interval criterion after diagnosis, and 44 (74.6%) had recurrent lesions within a 1-year interval (median, 6.5 months; interquartile range, 3.5-12.2 months). Medical chart review was conducted from November 22, 2021, to January 12, 2022. Analysis was conducted from January 12, 2022, to January 15, 2022. Conclusions and Relevance: Overall, 118 (40%) of 297 pediatric patients with HS in this retrospective cross-sectional study did not meet all major diagnostic criteria at the time of diagnosis, largely due to failure to fulfill the 6-month recurrence interval criterion. Future studies are needed to determine the appropriate recurrence interval to facilitate timely diagnosis and promote clinical trial eligibility for pediatric patients with HS.

Associations between COVID-19 risk perceptions, behavior intentions and worry.

BACKGROUND: Risk perceptions are central to health behavior, but some types of risk perceptions may be more strongly connected to behavior than others. This research examined different risk perceptions of COVID-19 and their respective associations with behavior intentions and worry. PARTICIPANTS AND PROCEDURE: U.S. college students (N = 248) and general adults (N = 300) reported their risk perceptions of COVID-19 - including absolute numerical, verbal, comparative, and feelings of risk - as well as their worry and intentions to do things such as get vaccinated. RESULTS: Although most risk perceptions related to intentions and worry, feelings of risk were the most strongly and consistently related. The associations showed that the higher people's feelings of risk were, the greater were their intentions and worry. CONCLUSIONS: Assessing feelings of risk of COVID-19 may provide the best insight into people's perceived threat of this virus.

Skin Cancer in People of Color: A Systematic Review.

BACKGROUND: People of African, Asian, Hispanic or Latino, Pacific Islander, and Native Indian descent are considered people of color by the Skin of Color Society (SOCS). OBJECTIVES: In this study, we assess incidence, risk factors, clinical characteristics, histopathology, treatment, and survival for skin malignancies in people of color as defined by the SOCS, by systematically reviewing the literature. METHODS: An electronic literature search of the PubMed, EMBASE, and MEDLINE databases was performed. Articles published from 1 January 1990 through 12 December 2020 were included in the search. RESULTS: We identified 2666 publications potentially meeting the study criteria. Titles and abstracts of these studies were reviewed and 2353 were excluded. The full text of 313 articles were evaluated and 251 were included in this review. CONCLUSION: Differences in incidence, patterns, treatment, and survival exist among people of color for cutaneous malignancies. Further research and initiatives are needed to account for and mitigate these differences.