CholecTriplet2022: Show me a tool and tell me the triplet - An endoscopic vision challenge for surgical action triplet detection.

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery.

Learning vs. minding: How subjective costs can mask motor learning.

When learning new movements some people make larger kinematic errors than others, interpreted as a reduction in motor-learning ability. Consider a learning task where error-cancelling strategies incur higher effort costs, specifically where subjects reach to targets in a force field. Concluding that those with greater error have learned less has a critical assumption: everyone uses the same error-canceling strategy. Alternatively, it could be that those with greater error may be choosing to sacrifice error reduction in favor of a lower effort movement. Here, we test this hypothesis in a dataset that includes both younger and older adults, where older adults exhibited greater kinematic errors. Utilizing the framework of optimal control theory, we infer subjective costs (i.e., strategies) and internal model accuracy (i.e., proportion of the novel dynamics learned) by fitting a model to each population's trajectory data. Our results demonstrate trajectories are defined by a combination of the amount learned and strategic differences represented by relative cost weights. Based on the model fits, younger adults could have learned between 65-90% of the novel dynamics. Critically, older adults could have learned between 60-85%. Each model fit produces trajectories that match the experimentally observed data, where a lower proportion learned in the model is compensated for by increasing costs on kinematic errors relative to effort. This suggests older and younger adults could be learning to the same extent, but older adults have a higher relative cost on effort compared to younger adults. These results call into question the proposition that older adults learn less than younger adults and provide a potential explanation for the equivocal findings in the literature. Importantly, our findings suggest that the metrics commonly used to probe motor learning paint an incomplete picture, and that to accurately quantify the learning process the subjective costs of movements should be considered.

Reaching Movements With Limb-Based Visual Feedback.

Reaches in experimental settings are commonly found to be straight. This straightness is robust to physical, but not visual, perturbations. Here, we question whether typical visual feedback contributes to this finding by implicitly promoting straight movements. To do so, we replaced the conventional feedback depicting the hand's location with feedback depicting the limb's orientation. Reaching movements with three different visual feedback conditions were examined. In the final condition, the subject's arm was depicted as two rotating links, and targets were depicted as two links indicating a desired arm posture. We found that by replacing standard cursor feedback, reaches became curved and arched to the target. Our findings further demonstrate that depicted feedback influences movements, and feedback depicting the limb, in particular, may elicit curved reaches.

A Probabilistic Approach to Surgical Tasks and Skill Metrics.

Identifying and quantifying the activities that compose surgery is essential for effective interventions, computer-aided analyses and the advancement of surgical data science. For example, recent studies have shown that objective metrics (referred to as objective performance indicators, OPIs) computed during key surgical tasks correlate with surgeon skill and clinical outcomes. Unambiguous identification of these surgical tasks can be particularly challenging for both human annotators and algorithms. Each surgical procedure has multiple approaches, each surgeon has their own level of skill, and the initiation and termination of surgical tasks can be subject to interpretation. As such, human annotators and machine learning models face the same basic problem, accurately identifying the boundaries of surgical tasks despite variable and unstructured information. For use in surgeon feedback, OPIs should also be robust to the variability and diversity in this data. To mitigate this difficulty, we propose a probabilistic approach to surgical task identification and calculation of OPIs. Rather than relying on tasks that are identified by hard temporal boundaries, we demonstrate an approach that relies on distributions of start and stop times, for a probabilistic interpretation of when the task was performed. We first use hypothetical data to outline how this approach is superior to other conventional approaches. Then we present similar analyses on surgical data. We find that when surgical tasks are identified by their individual probabilities, the resulting OPIs are less sensitive to noise in the identification of the start and stop times. These results suggest that this probabilistic approach holds promise for the future of surgical data science.

Correction: Force field generalization and the internal representation of motor learning.

[This corrects the article DOI: 10.1371/journal.pone.0225002.].

Planned Straight or Biased to Be So? The Influence of Visual Feedback on Reaching Movements.

Behavioral studies consistently find that subjects move their hand along straight paths despite considerations that suggest reaches should be curved. Literature on this topic makes it clear that the experimentally displayed feedback influences how subjects reach. Could the standard visual feedback, a displayed cursor, explain the lack of path curvature in experimental results? To address this question, we conducted three experiments to examine reach behavior in the absence of the standard visual feedback. In the first experiment, we found significant increases in curvature as visual feedback was progressively extinguished across groups. A second experiment revealed that practiced reaches became curved after the standard visual feedback was removed. A final experiment found that subjects' reaches made before and after a brief display of visual feedback were similar, indicating a preference for specific curved trajectories. Our results suggest that the consistently straight reaches often observed could be due to a bias to move the displayed cursor straight, which when removed reveal subject-specific preferences for reaches that are often curved.

Force field generalization and the internal representation of motor learning.

When learning a new motor behavior, e.g. reaching in a force field, the nervous system builds an internal representation. Examining how subsequent reaches in unpracticed directions generalize reveals this representation. Although often studied, it is not known how this representation changes across training directions, or how changes in reach direction and the corresponding changes in limb impedance, influence these measurements. We ran a force field adaptation experiment using eight groups of subjects each trained on one of eight standard directions and then tested for generalization in the remaining seven directions. Generalization in all directions was local and asymmetric, providing limited and unequal transfer to the left and right side of the trained target. These asymmetries were not consistent in either magnitude or direction, even after correcting for changes in limb impedance. Relying on a standard model for generalization the inferred representations inconsistently shifted to one side or the other of their respective training direction. A second model that accounted for limb impedance and variations in baseline trajectories explained more data and the inferred representations were centered on their respective training directions. Our results highlight the influence of limb mechanics and impedance on psychophysical measurements and their interpretations for motor learning.

A normative approach to neuromotor control.

While we can readily observe and model the dynamics of our limbs, analyzing the neurons that drive movement is not nearly as straightforward. As a result, their role in motor behavior (e.g., forward models, state estimators, controllers, etc.) remains elusive. Computational explanations of electrophysiological data often rely on firing rate models or deterministic spiking models. Yet neither can accurately describe the interactions of neurons that issue spikes, probabilistically. Here we take a normative approach by designing a probabilistic spiking network to implement LQR control for a limb model. We find typical results: cosine tuning curves, population vectors that correlate with reaching directions, low-dimensional oscillatory activity for reaches that have no oscillatory movement, and changes in neuron's tuning curves after force field adaptation. Importantly, while the model is consistent with these empirically derived correlations, we can also analyze it in terms of the known causal mechanism: an LQR controller and the probability distributions of the neurons that encode it. Redesigning the system under a different set of assumptions (e.g. a different controller, or network architecture) would yield a new set of testable predictions. We suggest this normative approach can be a framework for examining the motor system, providing testable links between observed neural activity and motor behavior.

Visual feedback of hand and target location does not explain the tendency for straight adapted reaches.

Subjects in laboratory settings exhibit straight hand paths-typified by the minimum jerk path-even in the presence of a learned but disturbing force field. At the same time it is known that in this setting, visual feedback strongly influences reaches, biasing them to be straight. Here we examine whether or not this bias can account for the straightness of movements made in a force field. We ran three curl field experiments to investigate how the lack of visual feedback influences adapted reaches. In a first experiment, hand position was displayed at the beginning and at the end of each trial, but extinguished during movement, and the hand was passively brought back to the home location. In the second experiment, visual feedback of neither the hand nor the target was provided, and targets were haptically rendered as "dimples." In order to provide extended practice, a third experiment was run with a single target and an active reach back to the home location. In all three cases we found minor changes in the adapted reaches relative to control groups that had full visual feedback. Our subjects adopted trajectories that were better explained by minimum jerk paths over those that minimize effort. The results indicate that for point-to-point reaching movements the visual feedback, or lack there of, cannot explain why reaches appear to be straight, even after adapting to a perturbing force field.

Chunking as the result of an efficiency computation trade-off.

How to move efficiently is an optimal control problem, whose computational complexity grows exponentially with the horizon of the planned trajectory. Breaking a compound movement into a series of chunks, each planned over a shorter horizon can thus reduce the overall computational complexity and associated costs while limiting the achievable efficiency. This trade-off suggests a cost-effective learning strategy: to learn new movements we should start with many short chunks (to limit the cost of computation). As practice reduces the impediments to more complex computation, the chunking structure should evolve to allow progressively more efficient movements (to maximize efficiency). Here we show that monkeys learning a reaching sequence over an extended period of time adopt this strategy by performing movements that can be described as locally optimal trajectories. Chunking can thus be understood as a cost-effective strategy for producing and learning efficient movements.

Deep networks for motor control functions.

The motor system generates time-varying commands to move our limbs and body. Conventional descriptions of motor control and learning rely on dynamical representations of our body's state (forward and inverse models), and control policies that must be integrated forward to generate feedforward time-varying commands; thus these are representations across space, but not time. Here we examine a new approach that directly represents both time-varying commands and the resulting state trajectories with a function; a representation across space and time. Since the output of this function includes time, it necessarily requires more parameters than a typical dynamical model. To avoid the problems of local minima these extra parameters introduce, we exploit recent advances in machine learning to build our function using a stacked autoencoder, or deep network. With initial and target states as inputs, this deep network can be trained to output an accurate temporal profile of the optimal command and state trajectory for a point-to-point reach of a non-linear limb model, even when influenced by varying force fields. In a manner that mirrors motor babble, the network can also teach itself to learn through trial and error. Lastly, we demonstrate how this network can learn to optimize a cost objective. This functional approach to motor control is a sharp departure from the standard dynamical approach, and may offer new insights into the neural implementation of motor control.

Using psychophysics to ask if the brain samples or maximizes.

The two-alternative forced-choice (2AFC) task is the workhorse of psychophysics and is used to measure the just-noticeable difference, generally assumed to accurately quantify sensory precision. However, this assumption is not true for all mechanisms of decision making. Here we derive the behavioral predictions for two popular mechanisms, sampling and maximum a posteriori, and examine how they affect the outcome of the 2AFC task. These predictions are used in a combined visual 2AFC and estimation experiment. Our results strongly suggest that subjects use a maximum a posteriori mechanism. Further, our derivations and experimental paradigm establish the already standard 2AFC task as a behavioral tool for measuring how humans make decisions under uncertainty.

The effects of training breadth on motor generalization.

To generate new movements, we have to generalize what we have learned from previously practiced movements. An important question, therefore, is how the breadth of training affects generalization: does practicing a broad or narrow range of movements lead to better generalization? We address this question with a force field learning experiment. One group adapted while making many reaches in a small region (narrow group), and another group adapted while making reaches in a large region (broad group). Subsequently, both groups were tested for their ability to generalize without visual feedback. Not surprisingly, the narrow group exhibited smaller adaptation errors, yet they did not generalize any better than the broad group. Path errors during generalization were indistinguishable across the two groups, whereas the broad group exhibited reduced terminal errors. These findings indicate that overall, practicing a variety of movements is advantageous for performance during generalization; movement paths are not hindered, and terminal errors are superior. Moreover, the evidence suggests a dissociation between the ability to generalize information about a novel dynamic disturbance, which generalizes narrowly, and the ability to locate the limb accurately in space, which generalizes broadly.

Motor learning of novel dynamics is not represented in a single global coordinate system: evaluation of mixed coordinate representations and local learning.

Successful motor performance requires the ability to adapt motor commands to task dynamics. A central question in movement neuroscience is how these dynamics are represented. Although it is widely assumed that dynamics (e.g., force fields) are represented in intrinsic, joint-based coordinates (Shadmehr R, Mussa-Ivaldi FA. J Neurosci 14: 3208-3224, 1994), recent evidence has questioned this proposal. Here we reexamine the representation of dynamics in two experiments. By testing generalization following changes in shoulder, elbow, or wrist configurations, the first experiment tested for extrinsic, intrinsic, or object-centered representations. No single coordinate frame accounted for the pattern of generalization. Rather, generalization patterns were better accounted for by a mixture of representations or by models that assumed local learning and graded, decaying generalization. A second experiment, in which we replicated the design of an influential study that had suggested encoding in intrinsic coordinates (Shadmehr and Mussa-Ivaldi 1994), yielded similar results. That is, we could not find evidence that dynamics are represented in a single coordinate system. Taken together, our experiments suggest that internal models do not employ a single coordinate system when generalizing and may well be represented as a mixture of coordinate systems, as a single system with local learning, or both.

An examination of the generalizability of motor costs.

Most approaches to understanding human motor control assume that people maximize their rewards while minimizing their motor efforts. This tradeoff between potential rewards and a sense of effort is quantified with a cost function. While the rewards can change across tasks, our sense of effort is assumed to remain constant and characterize how the nervous system organizes motor control. As such, when a proposed cost function compares well with data it is argued to be the underlying cause of a motor behavior, and not simply a fit to the data. Implicit in this proposition is the assumption that this cost function can then predict new motor behaviors. Here we examined this idea and asked whether an inferred cost function in one setting could explain subject's behavior in settings that differed dynamically but had identical rewards. We found that the pattern of behavior observed across settings was similar to our predictions of optimal behavior. However, we could not conclude that this behavior was consistent with a conserved sense of effort. These results suggest that the standard forms for quantifying cost may not be sufficient to accurately examine whether or not human motor behavior abides by optimality principles.

FES control of isometric forces in the rat hindlimb using many muscles.

Functional electrical stimulation (FES) attempts to restore motor behaviors to paralyzed limbs by electrically stimulating nerves and/or muscles. This restoration of behavior requires specifying commands to a large number of muscles, each making an independent contribution to the ongoing behavior. Efforts to develop FES systems in humans have generally been limited to preprogrammed, fixed muscle activation patterns. The development and evaluation of more sophisticated FES control strategies is difficult to accomplish in humans, mainly because of the limited access of patients for FES experiments. Here, we developed an in vivo FES test platform using a rat model that is capable of using many muscles for control and that can therefore be used to evaluate potential strategies for developing flexible FES control strategies. We first validated this FES test platform by showing consistent force responses to repeated stimulation, monotonically increasing muscle recruitment with constant force directions, and linear summation of costimulated muscles. These results demonstrate that we are able to differentially control the activation of many muscles, despite the small size of the rat hindlimb. We then demonstrate the utility of this platform to test potential FES control strategies, using it to test our ability to effectively produce open-loop control of isometric forces. We show that we are able to use this preparation to produce a range of endpoint forces flexibly and with good accuracy. We suggest that this platform will aid in FES controller design, development, and evaluation, thus accelerating the development of effective FES applications for the restoration of movement in paralyzed patients.

Estimating the relevance of world disturbances to explain savings, interference and long-term motor adaptation effects.

Recent studies suggest that motor adaptation is the result of multiple, perhaps linear processes each with distinct time scales. While these models are consistent with some motor phenomena, they can neither explain the relatively fast re-adaptation after a long washout period, nor savings on a subsequent day. Here we examined if these effects can be explained if we assume that the CNS stores and retrieves movement parameters based on their possible relevance. We formalize this idea with a model that infers not only the sources of potential motor errors, but also their relevance to the current motor circumstances. In our model adaptation is the process of re-estimating parameters that represent the body and the world. The likelihood of a world parameter being relevant is then based on the mismatch between an observed movement and that predicted when not compensating for the estimated world disturbance. As such, adapting to large motor errors in a laboratory setting should alert subjects that disturbances are being imposed on them, even after motor performance has returned to baseline. Estimates of this external disturbance should be relevant both now and in future laboratory settings. Estimated properties of our bodies on the other hand should always be relevant. Our model demonstrates savings, interference, spontaneous rebound and differences between adaptation to sudden and gradual disturbances. We suggest that many issues concerning savings and interference can be understood when adaptation is conditioned on the relevance of parameters.

Bayesian approaches to sensory integration for motor control.

The processing of sensory information is fundamental to the basic operation of the nervous system. Our nervous system uses this sensory information to gain knowledge of our bodies and the world around us. This knowledge is of great importance as it provides the coherent and accurate information necessary for successful motor control. Yet, all this knowledge is of an uncertain nature because we obtain information only through our noisy sensors. We are thus faced with the problem of integrating many uncertain pieces of information into estimates of the properties of our bodies and the surrounding world. Bayesian approaches to estimation formalize the problem of how this uncertain information should be integrated. Utilizing this approach, many studies make predictions that faithfully predict human sensorimotor behavior. WIREs Cogni Sci 2011 2 419-428 DOI: 10.1002/wcs.125 For further resources related to this article, please visit the WIREs website.

Learning priors for Bayesian computations in the nervous system.

Our nervous system continuously combines new information from our senses with information it has acquired throughout life. Numerous studies have found that human subjects manage this by integrating their observations with their previous experience (priors) in a way that is close to the statistical optimum. However, little is known about the way the nervous system acquires or learns priors. Here we present results from experiments where the underlying distribution of target locations in an estimation task was switched, manipulating the prior subjects should use. Our experimental design allowed us to measure a subject's evolving prior while they learned. We confirm that through extensive practice subjects learn the correct prior for the task. We found that subjects can rapidly learn the mean of a new prior while the variance is learned more slowly and with a variable learning rate. In addition, we found that a Bayesian inference model could predict the time course of the observed learning while offering an intuitive explanation for the findings. The evidence suggests the nervous system continuously updates its priors to enable efficient behavior.

Simplified and effective motor control based on muscle synergies to exploit musculoskeletal dynamics.

The basic hypothesis of producing a range of behaviors using a small set of motor commands has been proposed in various forms to explain motor behaviors ranging from basic reflexes to complex voluntary movements. Yet many fundamental questions regarding this long-standing hypothesis remain unanswered. Indeed, given the prominent nonlinearities and high dimensionality inherent in the control of biological limbs, the basic feasibility of a low-dimensional controller and an underlying principle for its creation has remained elusive. We propose a principle for the design of such a controller, that it endeavors to control the natural dynamics of the limb, taking into account the nature of the task being performed. Using this principle, we obtained a low-dimensional model of the hindlimb and a set of muscle synergies to command it. We demonstrate that this set of synergies was capable of producing effective control, establishing the viability of this muscle synergy hypothesis. Finally, by combining the low-dimensional model and the muscle synergies we were able to build a relatively simple controller whose overall performance was close to that of the system's full-dimensional nonlinear controller. Taken together, the results of this study establish that a low-dimensional controller is capable of simplifying control without degrading performance.