Dyport: dynamic importance-based biomedical hypothesis generation benchmarking technique.

BACKGROUND: Automated hypothesis generation (HG) focuses on uncovering hidden connections within the extensive information that is publicly available. This domain has become increasingly popular, thanks to modern machine learning algorithms. However, the automated evaluation of HG systems is still an open problem, especially on a larger scale. RESULTS: This paper presents a novel benchmarking framework Dyport for evaluating biomedical hypothesis generation systems. Utilizing curated datasets, our approach tests these systems under realistic conditions, enhancing the relevance of our evaluations. We integrate knowledge from the curated databases into a dynamic graph, accompanied by a method to quantify discovery importance. This not only assesses hypotheses accuracy but also their potential impact in biomedical research which significantly extends traditional link prediction benchmarks. Applicability of our benchmarking process is demonstrated on several link prediction systems applied on biomedical semantic knowledge graphs. Being flexible, our benchmarking system is designed for broad application in hypothesis generation quality verification, aiming to expand the scope of scientific discovery within the biomedical research community. CONCLUSIONS: Dyport is an open-source benchmarking framework designed for biomedical hypothesis generation systems evaluation, which takes into account knowledge dynamics, semantics and impact. All code and datasets are available at: https://github.com/IlyaTyagin/Dyport .

Diagnosing patient flow issues in the emergency department: an Australasian hospital case study.

PURPOSE: This study investigates how a hospital can increase the flow of patients through its emergency department by using benchmarking and process improvement techniques borrowed from the manufacturing sector. DESIGN/METHODOLOGY/APPROACH: An in-depth case study of an Australasian public hospital utilises rigorous, multi-method data collection procedures with systems thinking to benchmark an emergency department (ED) value stream and identify the performance inhibitors. FINDINGS: High levels of value stream uncertainty result from inefficient processes and weak controls. Reduced patient flow arises from senior management's commitment to simplistic government targets, clinical staff that lack basic operations management skills, and fragmented information systems. High junior/senior staff ratios aggravate the lack of inter-functional integration and poor use of time and material resources, increasing the risk of a critical patient incident. RESEARCH LIMITATIONS/IMPLICATIONS: This research is limited to a single case; hence, further research should assess value stream maturity and associated performance enablers and inhibitors in other emergency departments experiencing patient flow delays. PRACTICAL IMPLICATIONS: This study illustrates how hospital managers can use systems thinking and a context-free performance benchmarking measure to identify needed interventions and transferable best practices for achieving seamless patient flow. ORIGINALITY/VALUE: This study is the first to operationalise the theoretical concept of the seamless healthcare system to acute care as defined by Parnaby and Towill (2008). It is also the first to use the uncertainty circle model in an Australasian public healthcare setting to objectively benchmark an emergency department's value stream maturity.

In and out: Benchmarking in vitro, in vivo, ex vivo, and xenografting approaches for an integrative brain disease modeling pipeline.

Human cellular models and their neuronal derivatives have afforded unprecedented advances in elucidating pathogenic mechanisms of neuropsychiatric diseases. Notwithstanding their indispensable contribution, animal models remain the benchmark in neurobiological research. In an attempt to harness the best of both worlds, researchers have increasingly relied on human/animal chimeras by xenografting human cells into the animal brain. Despite the unparalleled potential of xenografting approaches in the study of the human brain, literature resources that systematically examine their significance and advantages are surprisingly lacking. We fill this gap by providing a comprehensive account of brain diseases that were thus far subjected to all three modeling approaches (transgenic rodents, in vitro human lineages, human-animal xenografting) and provide a critical appraisal of the impact of xenografting approaches for advancing our understanding of those diseases and brain development. Next, we give our perspective on integrating xenografting modeling pipeline with recent cutting-edge technological advancements.

Beyond benchmarking and towards predictive models of dataset-specific single-cell RNA-seq pipeline performance.

BACKGROUND: The advent of single-cell RNA-sequencing (scRNA-seq) has driven significant computational methods development for all steps in the scRNA-seq data analysis pipeline, including filtering, normalization, and clustering. The large number of methods and their resulting parameter combinations has created a combinatorial set of possible pipelines to analyze scRNA-seq data, which leads to the obvious question: which is best? Several benchmarking studies compare methods but frequently find variable performance depending on dataset and pipeline characteristics. Alternatively, the large number of scRNA-seq datasets along with advances in supervised machine learning raise a tantalizing possibility: could the optimal pipeline be predicted for a given dataset? RESULTS: Here, we begin to answer this question by applying 288 scRNA-seq analysis pipelines to 86 datasets and quantifying pipeline success via a range of measures evaluating cluster purity and biological plausibility. We build supervised machine learning models to predict pipeline success given a range of dataset and pipeline characteristics. We find that prediction performance is significantly better than random and that in many cases pipelines predicted to perform well provide clustering outputs similar to expert-annotated cell type labels. We identify characteristics of datasets that correlate with strong prediction performance that could guide when such prediction models may be useful. CONCLUSIONS: Supervised machine learning models have utility for recommending analysis pipelines and therefore the potential to alleviate the burden of choosing from the near-infinite number of possibilities. Different aspects of datasets influence the predictive performance of such models which will further guide users.

Estimation of urinary cadmium benchmark dose thresholds for preschool children in a cadmium-polluted area based on Bayesian model averaging.

Urinary cadmium (U-Cd) values are indicators for determining chronic cadmium toxicity, and previous studies have calculated U-Cd indicators using renal injury biomarkers. However, most of these studies have been conducted in adult populations, and there is a lack of research on U-Cd thresholds in preschool children. We aimed to apply benchmark dose (BMD) analysis to estimate the U-Cd threshold level associated with renal impairment in preschool children in the cadmium-polluted area. 518 preschool children aged 3-5 years were selected by systematic sampling (275 boys, 243 girls). Urinary cadmium and three biomarkers of early renal injury (urinary N-acetyl-ß-D-glucosaminidase, UNAG; urinary ß2-microglobulin, Uß2-MG; urinary retinol-binding protein, URBP) were determined. Bayesian model averaging estimated the BMD and lower confidence interval limit (BMDL) of U-Cd. The medians U-Cd levels in both boys and girls exceeded the recommended national standard threshold (5 µg/g cr) and U-Cd levels were higher in girls than in boys. Urinary N-acetyl-ß-D-glucosaminidase (UNAG) was the most sensitive biomarker of renal effects in preschool children. The overall BMDL5 (BMDL at a benchmark response value of 5) was 2.76 µg/g cr. In the gender analysis, the BMDL5 values were 1.92 µg/g cr for boys and 4.12 µg/g cr for girls. This study shows that the U-Cd threshold (BMDL5) is lower than the national standard (5 µg/g cr) and boys' BMDL5 was lower than the limit set by the European Parliament and Council in 2019 (2 µg/g cr), which provides a reference point for making U-Cd thresholds for preschool children.

Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation.

PURPOSE: The quality of radiotherapy auto-segmentation training data, primarily derived from clinician observers, is of utmost importance. However, the factors influencing the quality of clinician-derived segmentations are poorly understood; our study aims to quantify these factors. METHODS: Organ at risk (OAR) and tumor-related segmentations provided by radiation oncologists from the Contouring Collaborative for Consensus in Radiation Oncology data set were used. Segmentations were derived from five disease sites: breast, sarcoma, head and neck (H&N), gynecologic (GYN), and GI. Segmentation quality was determined on a structure-by-structure basis by comparing the observer segmentations with an expert-derived consensus, which served as a reference standard benchmark. The Dice similarity coefficient (DSC) was primarily used as a metric for the comparisons. DSC was stratified into binary groups on the basis of structure-specific expert-derived interobserver variability (IOV) cutoffs. Generalized linear mixed-effects models using Bayesian estimation were used to investigate the association between demographic variables and the binarized DSC for each disease site. Variables with a highest density interval excluding zero were considered to substantially affect the outcome measure. RESULTS: Five hundred seventy-four, 110, 452, 112, and 48 segmentations were used for the breast, sarcoma, H&N, GYN, and GI cases, respectively. The median percentage of segmentations that crossed the expert DSC IOV cutoff when stratified by structure type was 55% and 31% for OARs and tumors, respectively. Regression analysis revealed that the structure being tumor-related had a substantial negative impact on binarized DSC for the breast, sarcoma, H&N, and GI cases. There were no recurring relationships between segmentation quality and demographic variables across the cases, with most variables demonstrating large standard deviations. CONCLUSION: Our study highlights substantial uncertainty surrounding conventionally presumed factors influencing segmentation quality relative to benchmarks.

BioCoder: a benchmark for bioinformatics code generation with large language models.

SUMMARY: Pretrained large language models (LLMs) have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target bioinformatics due to the amount of domain knowledge, algorithms, and data operations this discipline requires. We present BioCoder, a benchmark developed to evaluate LLMs in generating bioinformatics-specific code. BioCoder spans much of the field, covering cross-file dependencies, class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling, we show that the overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate various models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our training dataset can enhance the performance on our testing benchmark (by >15% in terms of Pass@K under certain prompt configurations and always >3%). The results highlight two key aspects of successful models: (i) Successful models accommodate a long prompt (>2600 tokens) with full context, including functional dependencies. (ii) They contain domain-specific knowledge of bioinformatics, beyond just general coding capability. This is evident from the performance gain of GPT-3.5/4 compared to the smaller models on our benchmark (50% versus up to 25%). AVAILABILITY AND IMPLEMENTATION: All datasets, benchmark, Docker images, and scripts required for testing are available at: https://github.com/gersteinlab/biocoder and https://biocoder-benchmark.github.io/.

Collecting routine and timely cancer stage at diagnosis by implementing a cancer staging tiered framework: the Western Australian Cancer Registry experience.

BACKGROUND: Current processes collecting cancer stage data in population-based cancer registries (PBCRs) lack standardisation, resulting in difficulty utilising diverse data sources and incomplete, low-quality data. Implementing a cancer staging tiered framework aims to improve stage collection and facilitate inter-PBCR benchmarking. OBJECTIVE: Demonstrate the application of a cancer staging tiered framework in the Western Australian Cancer Staging Project to establish a standardised method for collecting cancer stage at diagnosis data in PBCRs. METHODS: The tiered framework, developed in collaboration with a Project Advisory Group and applied to breast, colorectal, and melanoma cancers, provides business rules - procedures for stage collection. Tier 1 represents the highest staging level, involving complete American Joint Committee on Cancer (AJCC) tumour-node-metastasis (TNM) data collection and other critical staging information. Tier 2 (registry-derived stage) relies on supplementary data, including hospital admission data, to make assumptions based on data availability. Tier 3 (pathology stage) solely uses pathology reports. FINDINGS: The tiered framework promotes flexible utilisation of staging data, recognising various levels of data completeness. Tier 1 is suitable for all purposes, including clinical and epidemiological applications. Tiers 2 and 3 are recommended for epidemiological analysis alone. Lower tiers provide valuable insights into disease patterns, risk factors, and overall disease burden for public health planning and policy decisions. Capture of staging at each tier depends on data availability, with potential shifts to higher tiers as new data sources are acquired. CONCLUSIONS: The tiered framework offers a dynamic approach for PBCRs to record stage at diagnosis, promoting consistency in population-level staging data and enabling practical use for benchmarking across jurisdictions, public health planning, policy development, epidemiological analyses, and assessing cancer outcomes. Evolution with staging classifications and data variable changes will futureproof the tiered framework. Its adaptability fosters continuous refinement of data collection processes and encourages improvements in data quality.

Optimizing in silico drug discovery: simulation of connected differential expression signatures and applications to benchmarking.

BACKGROUND: We present a novel simulation method for generating connected differential expression signatures. Traditional methods have struggled with the lack of reliable benchmarking data and biases in drug-disease pair labeling, limiting the rigorous benchmarking of connectivity-based approaches. OBJECTIVE: Our aim is to develop a simulation method based on a statistical framework that allows for adjustable levels of parametrization, especially the connectivity, to generate a pair of interconnected differential signatures. This could help to address the issue of benchmarking data availability for connectivity-based drug repurposing approaches. METHODS: We first detailed the simulation process and how it reflected real biological variability and the interconnectedness of gene expression signatures. Then, we generated several datasets to enable the evaluation of different existing algorithms that compare differential expression signatures, providing insights into their performance and limitations. RESULTS: Our findings demonstrate the ability of our simulation to produce realistic data, as evidenced by correlation analyses and the log2 fold-change distribution of deregulated genes. Benchmarking reveals that methods like extreme cosine similarity and Pearson correlation outperform others in identifying connected signatures. CONCLUSION: Overall, our method provides a reliable tool for simulating differential expression signatures. The data simulated by our tool encompass a wide spectrum of possibilities to challenge and evaluate existing methods to estimate connectivity scores. This may represent a critical gap in connectivity-based drug repurposing research because reliable benchmarking data are essential for assessing and advancing in the development of new algorithms. The simulation tool is available as a R package (General Public License (GPL) license) at https://github.com/cgonzalez-gomez/cosimu.

Benchmarking NGS-Based HLA Typing Algorithms.

Knowledge of the expected accuracy of HLA typing algorithms is important when choosing between algorithms and when evaluating the HLA typing predictions of an algorithm. This chapter guides the reader through an example benchmarking study that evaluates the performances of four NGS-based HLA typing algorithms as well as outlining factors to consider, when designing and running such a benchmarking study. The code related to this benchmarking workflow can be found at https://github.com/nikolasthuesen/springers-hla-benchmark/ .

Robotic performance metrics model fellow proficiency in living donor nephrectomy.

We investigated the use of robotic objective performance metrics (OPM) to predict number of cases to proficiency and independence among abdominal transplant fellows performing robot-assisted donor nephrectomy (RDN). 101 RDNs were performed by 5 transplant fellows from September 2020 to October 2023. OPM included fellow percent active control time (%ACT) and handoff counts (HC). Proficiency was defined as ACT ≥ 80% and HC ≤ 2, and independence as ACT ≥ 99% and HC ≤ 1. Case number was significantly associated with increasing fellow %ACT, with proficiency estimated at 14 cases and independence at 32 cases (R2 = 0.56, p < 0.001). Similarly, case number was significantly associated with decreasing HC, with proficiency at 18 cases and independence at 33 cases (R2 = 0.29, p < 0.001). Case number was not associated with total active console time (p = 0.91). Patient demographics, operative characteristics, and outcomes were not associated with OPM, except for donor estimated blood loss (EBL), which positively correlated with HC. Abdominal transplant fellows demonstrated proficiency at 14-18 cases and independence at 32-33 cases. Total active console time remained unchanged, suggesting that increasing fellow autonomy does not impede operative efficiency. These findings may serve as a benchmark for training abdominal transplant surgery fellows independently and safely in RDN.

Benchmarking short-, long- and hybrid-read assemblers for metagenome sequencing of complex microbial communities.

Metagenome community analyses, driven by the continued development in sequencing technology, is rapidly providing insights in many aspects of microbiology and becoming a cornerstone tool. Illumina, Oxford Nanopore Technologies (ONT) and Pacific Biosciences (PacBio) are the leading technologies, each with their own advantages and drawbacks. Illumina provides accurate reads at a low cost, but their length is too short to close bacterial genomes. Long reads overcome this limitation, but these technologies produce reads with lower accuracy (ONT) or with lower throughput (PacBio high-fidelity reads). In a critical first analysis step, reads are assembled to reconstruct genomes or individual genes within the community. However, to date, the performance of existing assemblers has never been challenged with a complex mock metagenome. Here, we evaluate the performance of current assemblers that use short, long or both read types on a complex mock metagenome consisting of 227 bacterial strains with varying degrees of relatedness. We show that many of the current assemblers are not suited to handle such a complex metagenome. In addition, hybrid assemblies do not fulfil their potential. We conclude that ONT reads assembled with CANU and Illumina reads assembled with SPAdes offer the best value for reconstructing genomes and individual genes of complex metagenomes, respectively.

Multicenter benchmarking of short and long read wet lab protocols for clinical viral metagenomics.

Metagenomics is gradually being implemented for diagnosing infectious diseases. However, in-depth protocol comparisons for viral detection have been limited to individual sets of experimental workflows and laboratories. In this study, we present a benchmark of metagenomics protocols used in clinical diagnostic laboratories initiated by the European Society for Clinical Virology (ESCV) Network on NGS (ENNGS). A mock viral reference panel was designed to mimic low biomass clinical specimens. The panel was used to assess the performance of twelve metagenomic wet lab protocols currently in use in the diagnostic laboratories of participating ENNGS member institutions. Both Illumina and Nanopore, shotgun and targeted capture probe protocols were included. Performance metrics sensitivity, specificity, and quantitative potential were assessed using a central bioinformatics pipeline. Overall, viral pathogens with loads down to 104 copies/ml (corresponding to CT values of 31 in our PCR assays) were detected by all the evaluated metagenomic wet lab protocols. In contrast, lower abundant mixed viruses of CT values of 35 and higher were detected only by a minority of the protocols. Considering the reference panel as the gold standard, optimal thresholds to define a positive result were determined per protocol, based on the horizontal genome coverage. Implementing these thresholds, sensitivity and specificity of the protocols ranged from 67 to 100 % and 87 to 100 %, respectively. A variety of metagenomic protocols are currently in use in clinical diagnostic laboratories. Detection of low abundant viral pathogens and mixed infections remains a challenge, implying the need for standardization of metagenomic analysis for use in clinical settings.

OpenMedLM: prompt engineering can out-perform fine-tuning in medical question-answering with open-source large language models.

LLMs can accomplish specialized medical knowledge tasks, however, equitable access is hindered by the extensive fine-tuning, specialized medical data requirement, and limited access to proprietary models. Open-source (OS) medical LLMs show performance improvements and provide the transparency and compliance required in healthcare. We present OpenMedLM, a prompting platform delivering state-of-the-art (SOTA) performance for OS LLMs on medical benchmarks. We evaluated OS foundation LLMs (7B-70B) on medical benchmarks (MedQA, MedMCQA, PubMedQA, MMLU medical-subset) and selected Yi34B for developing OpenMedLM. Prompting strategies included zero-shot, few-shot, chain-of-thought, and ensemble/self-consistency voting. OpenMedLM delivered OS SOTA results on three medical LLM benchmarks, surpassing previous best-performing OS models that leveraged costly and extensive fine-tuning. OpenMedLM displays the first results to date demonstrating the ability of OS foundation models to optimize performance, absent specialized fine-tuning. The model achieved 72.6% accuracy on MedQA, outperforming the previous SOTA by 2.4%, and 81.7% accuracy on MMLU medical-subset, establishing itself as the first OS LLM to surpass 80% accuracy on this benchmark. Our results highlight medical-specific emergent properties in OS LLMs not documented elsewhere to date and validate the ability of OS models to accomplish healthcare tasks, highlighting the benefits of prompt engineering to improve performance of accessible LLMs for medical applications.

Computational evaluation and benchmark study of 342 crystallographic holo-structures of SARS-CoV-2 Mpro enzyme.

The coronavirus disease 19 pandemic, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has led to a global health crisis with millions of confirmed cases and related deaths. The main protease (Mpro) of SARS-CoV-2 is crucial for viral replication and presents an attractive target for drug development. Despite the approval of some drugs, the search for effective treatments continues. In this study, we systematically evaluated 342 holo-crystal structures of Mpro to identify optimal conformations for structure-based virtual screening (SBVS). Our analysis revealed limited structural flexibility among the structures. Three docking programs, AutoDock Vina, rDock, and Glide were employed to assess the efficiency of virtual screening, revealing diverse performances across selected Mpro structures. We found that the structures 5RHE, 7DDC, and 7DPU (PDB Ids) consistently displayed the lowest EF, AUC, and BEDROCK scores. Furthermore, these structures demonstrated the worst pose prediction results in all docking programs. Two structural differences contribute to variations in docking performance: the absence of the S1 subsite in 7DDC and 7DPU, and the presence of a subpocket in the S2 subsite of 7DDC, 7DPU, and 5RHE. These findings underscore the importance of selecting appropriate Mpro conformations for SBVS, providing valuable insights for advancing drug discovery efforts.

Metabolomics Simultaneously Derives Benchmark Dose Estimates and Discovers Metabolic Biotransformations in a Rat Bioassay.

Benchmark dose (BMD) modeling estimates the dose of a chemical that causes a perturbation from baseline. Transcriptional BMDs have been shown to be relatively consistent with apical end point BMDs, opening the door to using molecular BMDs to derive human health-based guidance values for chemical exposure. Metabolomics measures the responses of small-molecule endogenous metabolites to chemical exposure, complementing transcriptomics by characterizing downstream molecular phenotypes that are more closely associated with apical end points. The aim of this study was to apply BMD modeling to in vivo metabolomics data, to compare metabolic BMDs to both transcriptional and apical end point BMDs. This builds upon our previous application of transcriptomics and BMD modeling to a 5-day rat study of triphenyl phosphate (TPhP), applying metabolomics to the same archived tissues. Specifically, liver from rats exposed to five doses of TPhP was investigated using liquid chromatography-mass spectrometry and 1H nuclear magnetic resonance spectroscopy-based metabolomics. Following the application of BMDExpress2 software, 2903 endogenous metabolic features yielded viable dose-response models, confirming a perturbation to the liver metabolome. Metabolic BMD estimates were similarly sensitive to transcriptional BMDs, and more sensitive than both clinical chemistry and apical end point BMDs. Pathway analysis of the multiomics data sets revealed a major effect of TPhP exposure on cholesterol (and downstream) pathways, consistent with clinical chemistry measurements. Additionally, the transcriptomics data indicated that TPhP activated xenobiotic metabolism pathways, which was confirmed by using the underexploited capability of metabolomics to detect xenobiotic-related compounds. Eleven biotransformation products of TPhP were discovered, and their levels were highly correlated with multiple xenobiotic metabolism genes. This work provides a case study showing how metabolomics and transcriptomics can estimate mechanistically anchored points-of-departure. Furthermore, the study demonstrates how metabolomics can also discover biotransformation products, which could be of value within a regulatory setting, for example, as an enhancement of OECD Test Guideline 417 (toxicokinetics).

PhyloBench: A Benchmark for Evaluating Phylogenetic Programs.

Phylogenetic inference based on protein sequence alignment is a widely used procedure. Numerous phylogenetic algorithms have been developed, most of which have many parameters and options. Choosing a program, options, and parameters can be a nontrivial task. No benchmark for comparison of phylogenetic programs on real protein sequences was publicly available. We have developed PhyloBench, a benchmark for evaluating the quality of phylogenetic inference, and used it to test a number of popular phylogenetic programs. PhyloBench is based on natural, not simulated, protein sequences of orthologous evolutionary domains. The measure of accuracy of an inferred tree is its distance to the corresponding species tree. A number of tree-to-tree distance measures were tested. The most reliable results were obtained using the Robinson-Foulds distance. Our results confirmed recent findings that distance methods are more accurate than maximum likelihood (ML) and maximum parsimony. We tested the bayesian program MrBayes on natural protein sequences and found that, on our datasets, it performs better than ML, but worse than distance methods. Of the methods we tested, the Balanced Minimum Evolution method implemented in FastME yielded the best results on our material. Alignments and reference species trees are available at https://mouse.belozersky.msu.ru/tools/phylobench/ together with a web-interface that allows for a semi-automatic comparison of a user's method with a number of popular programs.

Scaling neural machine translation to 200 languages.

The development of neural techniques has opened up new avenues for research in machine translation. Today, neural machine translation (NMT) systems can leverage highly multilingual capacities and even perform zero-shot translation, delivering promising results in terms of language coverage and quality. However, scaling quality NMT requires large volumes of parallel bilingual data, which are not equally available for the 7,000+ languages in the world1. Focusing on improving the translation qualities of a relatively small group of high-resource languages comes at the expense of directing research attention to low-resource languages, exacerbating digital inequities in the long run. To break this pattern, here we introduce No Language Left Behind-a single massively multilingual model that leverages transfer learning across languages. We developed a conditional computational model based on the Sparsely Gated Mixture of Experts architecture2-7, which we trained on data obtained with new mining techniques tailored for low-resource languages. Furthermore, we devised multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. We evaluated the performance of our model over 40,000 translation directions using tools created specifically for this purpose-an automatic benchmark (FLORES-200), a human evaluation metric (XSTS) and a toxicity detector that covers every language in our model. Compared with the previous state-of-the-art models, our model achieves an average of 44% improvement in translation quality as measured by BLEU. By demonstrating how to scale NMT to 200 languages and making all contributions in this effort freely available for non-commercial use, our work lays important groundwork for the development of a universal translation system.

United Kingdom oral and maxillofacial surgeons support changes to current specialty recruitment process, which could include 'Walport' style local selection and national benchmarking.

Recruitment to oral and maxillofacial Surgical (OMFS) specialty training was centralised in 2010. The 'flexibility' for OMFS to respond to specialty specific recruitment issues is reducing and many Specialty Trainees' (ST) posts are left unfilled. The National Institute for Health and Care Research (NIHR) appointment process designed to address the problem of recruiting and appointing academic surgeons with local selection with national benchmarking has worked. Using a database of all UK OMFS consultants/trainees, an electronic questionnaire was shared by e-mail, WhatsApp, and other social media. Of 306 replies, 125 (41%) were Consultants/post-certificate of completion training (CCT) individuals, 66 (22%) ST, 61 (20%) second degree students, 27 (9%) pre-second degree, 26 (9%) dual degree pre-ST trainees, and one did not indicate their status. A total of 249 (76%) studied dentistry first and 230 (75%) were male. Of those replying, 147 (48%) had no direct experience of national selection. 120 (39%) had experience as a candidate, 20 (7%) as a selector only, 17 (6%) as a candidate and selector, and two did not record their experience. Of 250 expressing an opinion, 156 (62%) supported local selection with 140 (56%) supporting local selection and national benchmarking, which is a process used for research training posts by the NIHR. Geographical continuity was most important for 78% of pre-second-degree trainees, 45% of STs, and 54% of second-degree students. A total of 57 respondents completed free text comments. There is support for changes in OMFS ST selection including creating OMFS posts which include Foundation and second-degree training in NIHR style locally recruited nationally benchmarked posts.