Classifying the emotional speech content of participants in group meetings using convolutional long short-term memory network.

Emotion is a central component of verbal communication between humans. Due to advances in machine learning and the development of affective computing, automatic emotion recognition is increasingly possible and sought after. To examine the connection between emotional speech and significant group dynamics perceptions, such as leadership and contribution, a new dataset (14 group meetings, 45 participants) is collected for analyzing collaborative group work based on the lunar survival task. To establish a training database, each participant's audio is manually annotated both categorically and along a three-dimensional scale with axes of activation, dominance, and valence and then converted to spectrograms. The performance of several neural network architectures for predicting speech emotion are compared for two tasks: categorical emotion classification and 3D emotion regression using multitask learning. Pretraining each neural network architecture on the well-known IEMOCAP (Interactive Emotional Dyadic Motion Capture) corpus improves the performance on this new group dynamics dataset. For both tasks, the two-dimensional convolutional long short-term memory network achieves the highest overall performance. By regressing the annotated emotions against post-task questionnaire variables for each participant, it is shown that the emotional speech content of a meeting can predict 71% of perceived group leaders and 86% of major contributors.

A Systematic Evaluation and Benchmark for Person Re-Identification: Features, Metrics, and Datasets.

Person re-identification (re-id) is a critical problem in video analytics applications such as security and surveillance. The public release of several datasets and code for vision algorithms has facilitated rapid progress in this area over the last few years. However, directly comparing re-id algorithms reported in the literature has become difficult since a wide variety of features, experimental protocols, and evaluation metrics are employed. In order to address this need, we present an extensive review and performance evaluation of single- and multi-shot re-id algorithms. The experimental protocol incorporates the most recent advances in both feature extraction and metric learning. To ensure a fair comparison, all of the approaches were implemented using a unified code library that includes 11 feature extraction algorithms and 22 metric learning and ranking techniques. All approaches were evaluated using a new large-scale dataset that closely mimics a real-world problem setting, in addition to 16 other publicly available datasets: VIPeR, GRID, CAVIAR, DukeMTMC4ReID, 3DPeS, PRID, V47, WARD, SAIVT-SoftBio, CUHK01, CHUK02, CUHK03, RAiD, iLIDSVID, HDA+, and Market1501. The evaluation codebase and results will be made publicly available for community use.

Viewpoint Invariant Human Re-Identification in Camera Networks Using Pose Priors and Subject-Discriminative Features.

Human re-identification across cameras with non-overlapping fields of view is one of the most important and difficult problems in video surveillance and analysis. However, current algorithms are likely to fail in real-world scenarios for several reasons. For example, surveillance cameras are typically mounted high above the ground plane, causing serious perspective changes. Also, most algorithms approach matching across images using the same descriptors, regardless of camera viewpoint or human pose. Here, we introduce a re-identification algorithm that addresses both problems. We build a model for human appearance as a function of pose, using training data gathered from a calibrated camera. We then apply this "pose prior" in online re-identification to make matching and identification more robust to viewpoint. We further integrate person-specific features learned over the course of tracking to improve the algorithm's performance. We evaluate the performance of the proposed algorithm and compare it to several state-of-the-art algorithms, demonstrating superior performance on standard benchmarking datasets as well as a challenging new airport surveillance scenario.

Estimating contact rates at a mass gathering by using video analysis: a proof-of-concept project.

BACKGROUND: Current approaches for estimating social mixing patterns and infectious disease transmission at mass gatherings have been limited by various constraints, including low participation rates for volunteer-based research projects and challenges in quantifying spatially and temporally accurate person-to-person interactions. We developed a proof-of-concept project to assess the use of automated video analysis for estimating contact rates of attendees of the GameFest 2013 event at Rensselaer Polytechnic Institute (RPI) in Troy, New York. METHODS: Video tracking and analysis algorithms were used to estimate the number and duration of contacts for 5 attendees during a 3-minute clip from the RPI video. Attendees were considered to have a contact event if the distance between them and another person was ≤1 meter. Contact duration was estimated in seconds. We also simulated 50 attendees assuming random mixing using a geo-spatially accurate representation of the same GameFest location. RESULTS: The 5 attendees had an overall median of 2 contact events during the 3-minute video clip (range: 0-6). Contact events varied from less than 5 seconds to the full duration of the 3-minute clip. The random mixing simulation was visualized and presented as a contrasting example. CONCLUSION: We were able to estimate the number and duration of contacts for 5 GameFest attendees from a 3-minute video clip that can be compared to a random mixing simulation model at the same location. The next phase will involve scaling the system for simultaneous analysis of mixing patterns from hours-long videos and comparing our results with other approaches for collecting contact data from mass gathering attendees.

Keeping a pan-tilt-zoom camera calibrated.

Pan-tilt-zoom (PTZ) cameras are pervasive in modern surveillance systems. However, we demonstrate that the (pan, tilt) coordinates reported by PTZ cameras become inaccurate after many hours of operation, endangering tracking and 3D localization algorithms that rely on the accuracy of such values. To solve this problem, we propose a complete model for a PTZ camera that explicitly reflects how focal length and lens distortion vary as a function of zoom scale. We show how the parameters of this model can be quickly and accurately estimated using a series of simple initialization steps followed by a nonlinear optimization. Our method requires only 10 images to achieve accurate calibration results. Next, we show how the calibration parameters can be maintained using a one-shot dynamic correction process; this ensures that the camera returns the same field of view every time the user requests a given (pan, tilt, zoom), even after hundreds of hours of operation. The dynamic calibration algorithm is based on matching the current image against a stored feature library created at the time the PTZ camera is mounted. We evaluate the calibration and dynamic correction algorithms on both experimental and real-world datasets, demonstrating the effectiveness of the techniques.

Reduced-order constrained optimization (ROCO): clinical application to head-and-neck IMRT.

PURPOSE: The authors present the application of the reduced order constrained optimization (ROCO) method, previously successfully applied to the prostate and lung sites, to the head-and-neck (H&N) site, demonstrating that it can quickly and automatically generate clinically competitive IMRT plans. We provide guidelines for applying ROCO to larynx, oropharynx, and nasopharynx cases, and report the results of a live experiment that demonstrates how an expert planner can save several hours of trial-and-error interaction using the proposed approach. METHODS: The ROCO method used for H&N IMRT planning consists of three major steps. First, the intensity space of treatment plans is sampled by solving a series of unconstrained optimization problems with a parameter range based on previously treated patient data. Second, the dominant modes in the intensity space are estimated by dimensionality reduction using principal component analysis (PCA). Third, a constrained optimization problem over this basis is quickly solved to find an IMRT plan that meets organ-at-risk (OAR) and target coverage constraints. The quality of the plan is assessed using evaluation tools within Memorial Sloan-Kettering Cancer Center (MSKCC)'s treatment planning system (TPS). RESULTS: The authors generated ten H&N IMRT plans for previously treated patients using the ROCO method and processed them for deliverability by a dynamic multileaf collimator (DMLC). The authors quantitatively compared the ROCO plans to the previously achieved clinical plans using the TPS tools used at MSKCC, including DVH and isodose contour analysis, and concluded that the ROCO plans would be clinically acceptable. In our current implementation, ROCO H&N plans can be generated using about 1.6 h of offline computation followed by 5-15 min of semiautomatic planning time. Additionally, the authors conducted a live session for a plan designated by MSKCC performed together with an expert H&N planner. A technical assistant set up the first two steps, which were performed without further human interaction, and then collaborated in a virtual meeting with the expert planner to perform the third (constrained optimization) step. The expert planner performed in-depth analysis of the resulting ROCO plan and deemed it to be clinically acceptable and in some aspects superior to the clinical plan. This entire process took 135 min including two constrained optimization runs, in comparison to the estimated 4 h that would have been required using traditional clinical planning tools. CONCLUSIONS: The H&N site is very challenging for IMRT planning, due to several levels of prescription and a large, variable number (6-20) of OARs that depend on the location of the tumor. ROCO for H&N shows promise in generating clinically acceptable plans both more quickly and with substantially less human interaction.

Reduced order constrained optimization (ROCO): clinical application to lung IMRT.

PURPOSE: The authors use reduced-order constrained optimization (ROCO) to create clinically acceptable IMRT plans quickly and automatically for advanced lung cancer patients. Their new ROCO implementation works with the treatment planning system and full dose calculation used at Memorial Sloan-Kettering Cancer Center (MSKCC). The authors have implemented mean dose hard constraints, along with the point-dose and dose-volume constraints that the authors used for our previous work on the prostate. METHODS: ROCO consists of three major steps. First, the space of treatment plans is sampled by solving a series of optimization problems using penalty-based quadratic objective functions. Next, an efficient basis for this space is found via principal component analysis (PCA); this reduces the dimensionality of the problem. Finally, a constrained optimization problem is solved over this basis to find a clinically acceptable IMRT plan. Dimensionality reduction makes constrained optimization computationally efficient. RESULTS: The authors apply ROCO to 12 stage III non-small-cell lung cancer (NSCLC) cases, generating IMRT plans that meet all clinical constraints and are clinically acceptable, and demonstrate that they are competitive with the clinical treatment plans. The authors also test how many samples and PCA modes are necessary to achieve an adequate lung plan, demonstrate the importance of long-range dose calculation for ROCO, and evaluate the performance of nonspecific normal tissue ("rind") constraints in ROCO treatment planning for the lung. Finally, authors show that ROCO can save time for planners, and they estimate that in the clinic, planning using their approach would save a median of 105 min for the patients in the study. CONCLUSIONS: New challenges arise when applying ROCO to the lung site, which include the lack of a class solution, a larger treatment site, an increased number of parameters and beamlets, a variable number of beams and beam arrangement, and the customary use of rinds in clinical plans to avoid high-dose areas outside the PTV. In the authors previous work, use of an approximate dose calculation in the hard constraint optimization sometimes meant that clinical constraints were not met when evaluated with the full dose calculation. This difficulty has been removed in the current work by using the full dose calculation in the hard constraint optimization. The authors have demonstrated that ROCO offers a fast and automatic way to create IMRT plans for advanced NSCLC, which extends their previous application of ROCO to prostate cancer IMRT planning.

Automated kymograph analysis for profiling axonal transport of secretory granules.

This paper describes an automated method to profile the velocity patterns of small organelles (BDNF granules) being transported along a selected section of axon of a cultured neuron imaged by time-lapse fluorescence microscopy. Instead of directly detecting the granules as in conventional tracking, the proposed method starts by generating a two-dimensional spatio-temporal map (kymograph) of the granule traffic along an axon segment. Temporal sharpening during the kymograph creation helps to highlight granule movements while suppressing clutter due to stationary granules. A voting algorithm defined over orientation distribution functions is used to refine the locations and velocities of the granules. The refined kymograph is analyzed using an algorithm inspired from the minimum set cover framework to generate multiple motion trajectories of granule transport paths. The proposed method is computationally efficient, robust to significant levels of noise and clutter, and can be used to capture and quantify trends in transport patterns quickly and accurately. When evaluated on a collection of image sequences, the proposed method was found to detect granule movement events with 94% recall rate and 82% precision compared to a time-consuming manual analysis. Further, we present a study to evaluate the efficacy of velocity profiling by analyzing the impact of oxidative stress on granule transport in which the fully automated analysis correctly reproduced the biological conclusion generated by manual analysis.

Segmenting the prostate and rectum in CT imagery using anatomical constraints.

The automatic segmentation of the prostate and rectum from 3D computed tomography (CT) images is still a challenging problem, and is critical for image-guided therapy applications. We present a new, automatic segmentation algorithm based on deformable organ models built from previously segmented training data. The major contributions of this work are a new segmentation cost function based on a Bayesian framework that incorporates anatomical constraints from surrounding bones and a new appearance model that learns a nonparametric distribution of the intensity histograms inside and outside organ contours. We report segmentation results on 185 datasets of the prostate site, demonstrating improved performance over previous models.

Reduced-order constrained optimization in IMRT planning.

This paper presents a new algorithm for constrained intensity-modulated radiotherapy (IMRT) planning, made tractable by a dimensionality reduction using a set of plans obtained by fast, unconstrained optimizations. The main result is to reduce planning time by an order of magnitude, producing viable five field prostate IMRT plans in about 5 min. Broadly, the algorithm has three steps. First, we solve a series of independent unconstrained minimization problems based on standard penalty-based objective functions, 'probing' the space of reasonable beamlet intensities. Next, we apply principal component analysis (PCA) to this set of plans, revealing that the high-dimensional intensity space can be spanned by only a few basis vectors. Finally, we parameterize an IMRT plan as a linear combination of these few basis vectors, enabling the fast solution of a constrained optimization problem for the desired intensities. We describe a simple iterative process for handling the dose-volume constraints that are typically required for clinical evaluation, and demonstrate that the resulting plans meet all clinical constraints based on an approximate dose calculation algorithm.

Reslicing axially sampled 3D shapes using elliptic Fourier descriptors.

We propose a new method that interpolates between parallel slices from a 3D shape for the purposes of reslicing and putting into correspondence organ shapes acquired from volumetric medical imagery. By interpolating the coefficients of elliptic Fourier descriptors for a set of parallel contours, a new set of slices can be directly generated at desired axial locations. Neither an explicit correspondence between points on adjacent contours nor a 3D interpolating surface needs to be obtained. We apply the proposed reslicing method to experimental datasets of both synthetic 3D shapes and real prostate contours, and demonstrate that it performs as well as a common method based on variational implicit surfaces, for a much lower computational cost. We also show that reslicing and putting into correspondence an ensemble of axially sampled 3D organs enables the construction of shape models for accurate 3D segmentation.

Reduced-order parameter optimization for simplifying prostate IMRT planning.

Intensity-modulated radiotherapy (IMRT) has become an effective tool for cancer treatment with radiation. However, even expert radiation planners still need to spend a substantial amount of time manually adjusting IMRT optimization parameters such as dose limits and costlet weights in order to obtain a clinically acceptable plan. In this paper, we describe two main advances that simplify the parameter adjustment process for five-field prostate IMRT planning. First, we report the results of a sensitivity analysis that quantifies the effect of each hand-tunable parameter of the IMRT cost function on each clinical objective and the overall quality of the resulting plan. Second, we show that a recursive random search over the six most sensitive parameters as an outer loop in IMRT planning can quickly and automatically determine parameters for the cost function that lead to a plan meeting the clinical requirements. Our experiments on a ten-patient dataset show that for 70% of the cases, we can automatically determine a plan in 10 min (on the average) that is either clinically acceptable or requires only minor adjustment by the planner. The outer-loop optimization can be easily integrated into a traditional IMRT planning system.

Automated semantic analysis of changes in image sequences of neurons in culture.

Quantitative studies of dynamic behaviors of live neurons are currently limited by the slowness, subjectivity, and tedium of manual analysis of changes in time-lapse image sequences. Challenges to automation include the complexity of the changes of interest, the presence of obfuscating and uninteresting changes due to illumination variations and other imaging artifacts, and the sheer volume of recorded data. This paper describes a highly automated approach that not only detects the interesting changes selectively, but also generates quantitative analyses at multiple levels of detail. Detailed quantitative neuronal morphometry is generated for each frame. Frame-to-frame neuronal changes are measured and labeled as growth, shrinkage, merging, or splitting, as would be done by a human expert. Finally, events unfolding over longer durations, such as apoptosis and axonal specification, are automatically inferred from the short-term changes. The proposed method is based on a Bayesian model selection criterion that leverages a set of short-term neurite change models and takes into account additional evidence provided by an illumination-insensitive change mask. An automated neuron tracing algorithm is used to identify the objects of interest in each frame. A novel curve distance measure and weighted bipartite graph matching are used to compare and associate neurites in successive frames. A separate set of multi-image change models drives the identification of longer term events. The method achieved frame-to-frame change labeling accuracies ranging from 85% to 100% when tested on 8 representative recordings performed under varied imaging and culturing conditions, and successfully detected all higher order events of interest. Two sequences were used for training the models and tuning their parameters; the learned parameter settings can be applied to hundreds of similar image sequences, provided imaging and culturing conditions are similar to the training set. The proposed approach is a substantial innovation over manual annotation and change analysis, accomplishing in minutes what it would take an expert hours to complete.

Learning the relationship between patient geometry and beam intensity in breast intensity-modulated radiotherapy.

Intensity modulated radiotherapy (IMRT) has become an effective tool for cancer treatment with radiation. However, even expert radiation planners still need to spend a substantial amount of time adjusting IMRT optimization parameters in order to get a clinically acceptable plan. We demonstrate that the relationship between patient geometry and radiation intensity distributions can be automatically inferred using a variety of machine learning techniques in the case of two-field breast IMRT. Our experiments show that given a small number of human-expert-generated clinically acceptable plans, the machine learning predictions produce equally acceptable plans in a matter of seconds. The machine learning approach has the potential for greater benefits in sites where the IMRT planning process is more challenging or tedious.

Automated cell lineage construction: a rapid method to analyze clonal development established with murine neural progenitor cells.

Understanding cell lineage relationships is fundamental to understanding development, and can shed light on disease etiology and progression. We present a method for automated tracking of lineages of proliferative, migrating cells from a sequence of images. The method is applicable to image sequences gathered either in vitro or in vivo. Currently, generating lineage trees from progenitor cells over time is a tedious, manual process, which limits the number of cell measurements that can be practically analyzed. In contrast, the automated method is rapid and easily applied, and produces a wealth of measurements including the precise position, shape, cell-cell contacts, motility and ancestry of each cell in every frame, and accurate timings of critical events, e.g., mitosis and cell death. Furthermore, it automatically produces graphical output that is immediately accessible. Application to clonal development of mouse neural progenitor cells growing in cell culture reveals complex changes in cell cycle rates during neuron and glial production. The method enables a level of quantitative analysis of cell behavior over time that was previously infeasible.

Image change detection algorithms: a systematic survey.

Detecting regions of change in multiple images of the same scene taken at different times is of widespread interest due to a large number of applications in diverse disciplines, including remote sensing, surveillance, medical diagnosis and treatment, civil infrastructure, and underwater sensing. This paper presents a systematic survey of the common processing steps and core decision rules in modern change detection algorithms, including significance and hypothesis testing, predictive models, the shading model, and background modeling. We also discuss important preprocessing methods, approaches to enforcing the consistency of the change mask, and principles for evaluating and comparing the performance of change detection algorithms. It is hoped that our classification of algorithms into a relatively small number of categories will provide useful guidance to the algorithm designer.

Model-based segmentation of medical imagery by matching distributions.

The segmentation of deformable objects from three-dimensional (3-D) images is an important and challenging problem, especially in the context of medical imagery. We present a new segmentation algorithm based on matching probability distributions of photometric variables that incorporates learned shape and appearance models for the objects of interest. The main innovation over similar approaches is that there is no need to compute a pixelwise correspondence between the model and the image. This allows for a fast, principled algorithm. We present promising results on difficult imagery for 3-D computed tomography images of the male pelvis for the purpose of image-guided radiotherapy of the prostate.