Validation of prediction models of severe disease course and non-achievement of remission in juvenile idiopathic arthritis part 2: results of the Nordic model in the Canadian cohort.

BACKGROUND: Validated clinical prediction models to identify children with poor prognosis at the time of juvenile idiopathic arthritis (JIA) diagnosis would be very helpful for tailoring treatments, and avoiding under- or over-treatment. Our objective was to externally validate Nordic clinical prediction models in Canadian patients with JIA. METHODS: We used data from 513 subjects at the 3-year follow-up from the Research in Arthritis in Canadian Children emphasizing Outcomes (ReACCh-Out) cohort. The predicted outcomes were non-achievement of remission, severe disease course, and functional disability. The Nordic models were evaluated exactly as published and after fine-tuning the logistic regression coefficients using multiple data splits of the Canadian cohort. Missing data was handled with multiple imputation, and prediction ability was assessed with C-indices. C-index values > 0.7 were deemed to reflect helpful prediction. RESULTS: Overall, 81% of evaluable patients did not achieve remission off medications, 15% experienced a severe disease course, and 38% reported disability (CHAQ score > 0). The Nordic model for predicting non-achievement of remission had a C-index of 0.68 (95% CI 0.62-0.74), and 0.74 (0.67-0.80) after fine-tuning. For prediction of severe disease course, it had a C-index of 0.69 (0.61-0.78), and 0.79 (0.68-0.91) after fine-tuning. The fine-tuned Nordic model identified 85% of the cohort as low risk for a severe disease course (< 20% chance) and 7% as high risk (> 60% chance). The Nordic model to predict functional disability had a C-index of 0.57 (0.50-0.63), and 0.51 (0.39-0.63) after fine-tuning. CONCLUSIONS: Fine-tuned Nordic models, combining active joint count, physician global assessment of disease activity, morning stiffness, and ankle involvement, predicted well non-achievement of remission and severe disease course in Canadian patients with JIA. The Nordic model for predicting disability could not predict functional disability in Canadian patients.

Validation of prediction models of severe disease course and non-achievement of remission in juvenile idiopathic arthritis: part 1-results of the Canadian model in the Nordic cohort.

BACKGROUND: Models to predict disease course and long-term outcome based on clinical characteristics at disease onset may guide early treatment strategies in juvenile idiopathic arthritis (JIA). Before a prediction model can be recommended for use in clinical practice, it needs to be validated in a different cohort than the one used for building the model. The aim of the current study was to validate the predictive performance of the Canadian prediction model developed by Guzman et al. and the Nordic model derived from Rypdal et al. to predict severe disease course and non-achievement of remission in Nordic patients with JIA. METHODS: The Canadian and Nordic multivariable logistic regression models were evaluated in the Nordic JIA cohort for prediction of non-achievement of remission, and the data-driven outcome denoted severe disease course. A total of 440 patients in the Nordic cohort with a baseline visit and an 8-year visit were included. The Canadian prediction model was first externally validated exactly as published. Both the Nordic and Canadian models were subsequently evaluated with repeated fine-tuning of model coefficients in training sets and testing in disjoint validation sets. The predictive performances of the models were assessed with receiver operating characteristic curves and C-indices. A model with a C-index above 0.7 was considered useful for clinical prediction. RESULTS: The Canadian prediction model had excellent predictive ability and was comparable in performance to the Nordic model in predicting severe disease course in the Nordic JIA cohort. The Canadian model yielded a C-index of 0.85 (IQR 0.83-0.87) for prediction of severe disease course and a C-index of 0.66 (0.63-0.68) for prediction of non-achievement of remission when applied directly. The median C-indices after fine-tuning were 0.85 (0.80-0.89) and 0.69 (0.65-0.73), respectively. Internal validation of the Nordic model for prediction of severe disease course resulted in a median C-index of 0.90 (0.86-0.92). CONCLUSIONS: External validation of the Canadian model and internal validation of the Nordic model with severe disease course as outcome confirm their predictive abilities. Our findings suggest that predicting long-term remission is more challenging than predicting severe disease course.

Classic motor chunking theory fails to account for behavioural diversity and speed in a complex naturalistic task.

In tasks that demand rapid performance, actions must be executed as efficiently as possible. Theories of expert motor performance such as the motor chunking framework suggest that efficiency is supported by automatization, where many serial actions are automatized into smaller chunks, or groups of commonly co-occuring actions. We use the fast-paced, professional eSport StarCraft 2 as a test case of the explanatory power of the motor chunking framework and assess the importance of chunks in explaining expert performance. To do so, we test three predictions motivated by a simple motor chunking framework. (1) StarCraft 2 players should exhibit an increasing number of chunks with expertise. (2) The proportion of actions falling within a chunk should increase with skill. (3) Chunks should be faster than non-chunks containing the same atomic behaviours. Although our findings support the existence of chunks, they also highlight two problems for existing accounts of rapid motor execution and expert performance. First, while better players do use more chunks, the proportion of actions within a chunks is stable across expertise and expert sequences are generally more varied (the diversity problem). Secondly, chunks, which are supposed to enjoy the most extreme automatization, appear to save little or no time overall (the time savings problem). Instead, the most parsimonious description of our latency analysis is that players become faster overall regardless of chunking.

Predicting Which Children with Juvenile Idiopathic Arthritis Will Not Attain Early Remission with Conventional Treatment: Results from the ReACCh-Out Cohort.

OBJECTIVE: To estimate the probability of early remission with conventional treatment for each child with juvenile idiopathic arthritis (JIA). Children with a low chance of remission may be candidates for initial treatment with biologics or triple disease-modifying antirheumatic drugs (DMARD). METHODS: We used data from 1074 subjects in the Research in Arthritis in Canadian Children emphasizing Outcomes (ReACCh-Out) cohort. The predicted outcome was clinically inactive disease for ≥ 6 months starting within 1 year of JIA diagnosis in patients who did not receive early biologic agents or triple DMARD. Models were developed in 200 random splits of 75% of the cohort and tested on the remaining 25% of subjects, calculating expected and observed frequencies of remission and c-index values. RESULTS: Our best Cox logistic model combining 18 clinical variables a median of 2 days after diagnosis had a c-index of 0.69 (95% CI 0.67-0.71), better than using JIA category alone (0.59, 95% CI 0.56-0.63). Children in the lowest probability decile had a 20% chance of remission and 21% attained remission; children in the highest decile had a 69% chance of remission and 73% attained remission. Compared to 5% of subjects identified by JIA category alone, the model identified 14% of subjects as low chance of remission (probability < 0.25), of whom 77% failed to attain remission. CONCLUSION: Although the model did not meet our a priori performance threshold (c-index > 0.70), it identified 3 times more subjects with low chance of remission than did JIA category alone, and it may serve as a benchmark for assessing value added by future laboratory/imaging biomarkers.

Practical considerations when analyzing discrete survival times using the grouped relative risk model.

The grouped relative risk model (GRRM) is a popular semi-parametric model for analyzing discrete survival time data. The maximum likelihood estimators (MLEs) of the regression coefficients in this model are often asymptotically efficient relative to those based on a more restrictive, parametric model. However, in settings with a small number of sampling units, the usual properties of the MLEs are not assured. In this paper, we discuss computational issues that can arise when fitting a GRRM to small samples, and describe conditions under which the MLEs can be ill-behaved. We find that, overall, estimators based on a penalized score function behave substantially better than the MLEs in this setting and, in particular, can be far more efficient. We also provide methods of assessing the fit of a GRRM to small samples.

Predicting Which Children with Juvenile Idiopathic Arthritis Will Have a Severe Disease Course: Results from the ReACCh-Out Cohort.

OBJECTIVE: We studied an inception cohort of children with juvenile idiopathic arthritis (JIA) to (1) identify distinct disease courses based on changes over 5 years in 5 variables prioritized by patients, parents, and clinicians; and (2) estimate the probability of a severe disease course for each child at diagnosis. METHODS: Assessments of quality of life, pain, medication requirements, patient-reported side effects, and active joint counts were scheduled at 0, 6, 12, 18, 24, 36, 48, and 60 months. Patients who attended at least 6 assessments were included. Multivariable cluster analysis, r2, and silhouette statistics were used to identify distinct disease courses. One hundred candidate prediction models were developed in random samples of 75% of the cohort; their reliability and accuracy were tested in the 25% not used in their development. RESULTS: Four distinct courses were identified in 609 subjects. They differed in prioritized variables, disability scores, and probabilities of attaining inactive disease and remission. We named them Mild (43.8% of children), Moderate (35.6%), Severe Controlled (9%), and Severe Persisting (11.5%). A logistic regression model using JIA category, active joint count, and pattern of joint involvement at enrollment best predicted a severe disease course (Controlled + Persisting, c-index = 0.87); 91% of children in the highest decile of risk actually experienced a severe disease course, compared to 5% of those in the lowest decile. CONCLUSION: Children in this JIA cohort followed 1 of 4 disease courses and the probability of a severe disease course could be estimated with information available at diagnosis.

Over the hill at 24: persistent age-related cognitive-motor decline in reaction times in an ecologically valid video game task begins in early adulthood.

Typically studies of the effects of aging on cognitive-motor performance emphasize changes in elderly populations. Although some research is directly concerned with when age-related decline actually begins, studies are often based on relatively simple reaction time tasks, making it impossible to gauge the impact of experience in compensating for this decline in a real world task. The present study investigates age-related changes in cognitive motor performance through adolescence and adulthood in a complex real world task, the real-time strategy video game StarCraft 2. In this paper we analyze the influence of age on performance using a dataset of 3,305 players, aged 16-44, collected by Thompson, Blair, Chen & Henrey [1]. Using a piecewise regression analysis, we find that age-related slowing of within-game, self-initiated response times begins at 24 years of age. We find no evidence for the common belief expertise should attenuate domain-specific cognitive decline. Domain-specific response time declines appear to persist regardless of skill level. A second analysis of dual-task performance finds no evidence of a corresponding age-related decline. Finally, an exploratory analyses of other age-related differences suggests that older participants may have been compensating for a loss in response speed through the use of game mechanics that reduce cognitive load.

Video game telemetry as a critical tool in the study of complex skill learning.

Cognitive science has long shown interest in expertise, in part because prediction and control of expert development would have immense practical value. Most studies in this area investigate expertise by comparing experts with novices. The reliance on contrastive samples in studies of human expertise only yields deep insight into development where differences are important throughout skill acquisition. This reliance may be pernicious where the predictive importance of variables is not constant across levels of expertise. Before the development of sophisticated machine learning tools for data mining larger samples, and indeed, before such samples were available, it was difficult to test the implicit assumption of static variable importance in expertise development. To investigate if this reliance may have imposed critical restrictions on the understanding of complex skill development, we adopted an alternative method, the online acquisition of telemetry data from a common daily activity for many: video gaming. Using measures of cognitive-motor, attentional, and perceptual processing extracted from game data from 3360 Real-Time Strategy players at 7 different levels of expertise, we identified 12 variables relevant to expertise. We show that the static variable importance assumption is false--the predictive importance of these variables shifted as the levels of expertise increased--and, at least in our dataset, that a contrastive approach would have been misleading. The finding that variable importance is not static across levels of expertise suggests that large, diverse datasets of sustained cognitive-motor performance are crucial for an understanding of expertise in real-world contexts. We also identify plausible cognitive markers of expertise.