Performance of GPT-4 on the American College of Radiology In-training Examination: Evaluating Accuracy, Model Drift, and Fine-tuning.

RATIONALE AND OBJECTIVES: In our study, we evaluate GPT-4's performance on the American College of Radiology (ACR) 2022 Diagnostic Radiology In-Training Examination (DXIT). We perform multiple experiments across time points to assess for model drift, as well as after fine-tuning to assess for differences in accuracy. MATERIALS AND METHODS: Questions were sequentially input into GPT-4 with a standardized prompt. Each answer was recorded and overall accuracy was calculated, as was logic-adjusted accuracy, and accuracy on image-based questions. This experiment was repeated several months later to assess for model drift, then again after the performance of fine-tuning to assess for changes in GPT's performance. RESULTS: GPT-4 achieved 58.5% overall accuracy, lower than the PGY-3 average (61.9%) but higher than the PGY-2 average (52.8%). Adjusted accuracy was 52.8%. GPT-4 showed significantly higher (p = 0.012) confidence for correct answers (87.1%) compared to incorrect (84.0%). Performance on image-based questions was significantly poorer (p < 0.001) at 45.4% compared to text-only questions (80.0%), with adjusted accuracy for image-based questions of 36.4%. When the questions were repeated, GPT-4 chose a different answer 25.5% of the time and there was no change in accuracy. Fine-tuning did not improve accuracy. CONCLUSION: GPT-4 performed between PGY-2 and PGY-3 levels on the 2022 DXIT, significantly poorer on image-based questions, and with large variability in answer choices across time points. Exploratory experiments in fine-tuning did not improve performance. This study underscores the potential and risks of using minimally-prompted general AI models in interpreting radiologic images as a diagnostic tool. Implementers of general AI radiology systems should exercise caution given the possibility of spurious yet confident responses.

Novel Validated Index for the Measurement of Disinformation Susceptibility at the County Level.

In the past decade, disinformation has become an increasingly dangerous enemy of public health, scientific advancement, and social stability. To address and counter this trend, it is essential to first identify communities most at risk for disinformation. The Jin-Hafiz Disinformation Index (JHDI) is developed and validated as a tool to counter disinformation and address deficits of good information on a county level in the United States. Once vulnerable communities are identified with the JHDI, targeted interventions with community partnerships can be conducted to address knowledge concerns.

Gray's paradox: a fluid mechanical perspective.

Nearly eighty years ago, Gray reported that the drag power experienced by a dolphin was larger than the estimated muscle power - this is termed as Gray's paradox. We provide a fluid mechanical perspective of this paradox. The viewpoint that swimmers necessarily spend muscle energy to overcome drag in the direction of swimming needs revision. For example, in undulatory swimming most of the muscle energy is directly expended to generate lateral undulations of the body, and the drag power is balanced not by the muscle power but by the thrust power. Depending on drag model utilized, the drag power may be greater than muscle power without being paradoxical.

Energy efficiency and allometry of movement of swimming and flying animals.

Which animals use their energy better during movement? One metric to answer this question is the energy cost per unit distance per unit weight. Prior data show that this metric decreases with mass, which is considered to imply that massive animals are more efficient. Although useful, this metric also implies that two dynamically equivalent animals of different sizes will not be considered equally efficient. We resolve this longstanding issue by first determining the scaling of energy cost per unit distance traveled. The scale is found to be M(2/3) or M(1/2), where M is the animal mass. Second, we introduce an energy-consumption coefficient (CE) defined as energy per unit distance traveled divided by this scale. CE is a measure of efficiency of swimming and flying, analogous to how drag coefficient quantifies aerodynamic drag on vehicles. Derivation of the energy-cost scale reveals that the assumption that undulatory swimmers spend energy to overcome drag in the direction of swimming is inappropriate. We derive allometric scalings that capture trends in data of swimming and flying animals over 10-20 orders of magnitude by mass. The energy-consumption coefficient reveals that swimmers beyond a critical mass, and most fliers are almost equally efficient as if they are dynamically equivalent; increasingly massive animals are not more efficient according to the proposed metric. Distinct allometric scalings are discovered for large and small swimmers. Flying animals are found to require relatively more energy compared with swimmers.