Point Light Source Position Estimation From RGB-D Images by Learning Surface Attributes.

Light source position (LSP) estimation is a difficult yet an important problem in computer vision. A common approach for estimating the LSP assumes Lambert's law. However, in real-world scenes, Lambert's law does not hold for all different types of surfaces. Instead of assuming all that surfaces follow Lambert's law, our approach classifies image surface segments based on their photometric and geometric surface attributes (i.e. glossy, matte, curved, and so on) and assigns weights to image surface segments based on their suitability for LSP estimation. In addition, we propose the use of the estimated camera pose to globally constrain LSP for RGB-D video sequences. Experiments on Boom and a newly collected RGB-D video data sets show that the state-of-the-art methods are outperformed by the proposed method. The results demonstrate that weighting image surface segments based on their attributes outperform the state-of-the-art methods in which the image surface segments are considered to equally contribute. In particular, by using the proposed surface weighting, the angular error for LSP estimation is reduced from 12.6° to 8.2° and 24.6° to 4.8° for Boom and RGB-D video data sets, respectively. Moreover, using the camera pose to globally constrain LSP provides higher accuracy (4.8°) compared with using single frames (8.5°).

Visual dictionaries as intermediate features in the human brain.

The human visual system is assumed to transform low level visual features to object and scene representations via features of intermediate complexity. How the brain computationally represents intermediate features is still unclear. To further elucidate this, we compared the biologically plausible HMAX model and Bag of Words (BoW) model from computer vision. Both these computational models use visual dictionaries, candidate features of intermediate complexity, to represent visual scenes, and the models have been proven effective in automatic object and scene recognition. These models however differ in the computation of visual dictionaries and pooling techniques. We investigated where in the brain and to what extent human fMRI responses to short video can be accounted for by multiple hierarchical levels of the HMAX and BoW models. Brain activity of 20 subjects obtained while viewing a short video clip was analyzed voxel-wise using a distance-based variation partitioning method. Results revealed that both HMAX and BoW explain a significant amount of brain activity in early visual regions V1, V2, and V3. However, BoW exhibits more consistency across subjects in accounting for brain activity compared to HMAX. Furthermore, visual dictionary representations by HMAX and BoW explain significantly some brain activity in higher areas which are believed to process intermediate features. Overall our results indicate that, although both HMAX and BoW account for activity in the human visual system, the BoW seems to more faithfully represent neural responses in low and intermediate level visual areas of the brain.

Visual Tracking: An Experimental Survey.

There is a large variety of trackers, which have been proposed in the literature during the last two decades with some mixed success. Object tracking in realistic scenarios is a difficult problem, therefore, it remains a most active area of research in computer vision. A good tracker should perform well in a large number of videos involving illumination changes, occlusion, clutter, camera motion, low contrast, specularities, and at least six more aspects. However, the performance of proposed trackers have been evaluated typically on less than ten videos, or on the special purpose datasets. In this paper, we aim to evaluate trackers systematically and experimentally on 315 video fragments covering above aspects. We selected a set of nineteen trackers to include a wide variety of algorithms often cited in literature, supplemented with trackers appearing in 2010 and 2011 for which the code was publicly available. We demonstrate that trackers can be evaluated objectively by survival curves, Kaplan Meier statistics, and Grubs testing. We find that in the evaluation practice the F-score is as effective as the object tracking accuracy (OTA) score. The analysis under a large variety of circumstances provides objective insight into the strengths and weaknesses of trackers.

Stages as models of scene geometry.

Reconstruction of 3D scene geometry is an important element for scene understanding, autonomous vehicle and robot navigation, image retrieval, and 3D television. We propose accounting for the inherent structure of the visual world when trying to solve the scene reconstruction problem. Consequently, we identify geometric scene categorization as the first step toward robust and efficient depth estimation from single images. We introduce 15 typical 3D scene geometries called stages, each with a unique depth profile, which roughly correspond to a large majority of broadcast video frames. Stage information serves as a first approximation of global depth, narrowing down the search space in depth estimation and object localization. We propose different sets of low-level features for depth estimation, and perform stage classification on two diverse data sets of television broadcasts. Classification results demonstrate that stages can often be efficiently learned from low-dimensional image representations.

Visual word ambiguity.

This paper studies automatic image classification by modeling soft assignment in the popular codebook model. The codebook model describes an image as a bag of discrete visual words selected from a vocabulary, where the frequency distributions of visual words in an image allow classification. One inherent component of the codebook model is the assignment of discrete visual words to continuous image features. Despite the clear mismatch of this hard assignment with the nature of continuous features, the approach has been successfully applied for some years. In this paper, we investigate four types of soft assignment of visual words to image features. We demonstrate that explicitly modeling visual word assignment ambiguity improves classification performance compared to the hard assignment of the traditional codebook model. The traditional codebook model is compared against our method for five well-known data sets: 15 natural scenes, Caltech-101, Caltech-256, and Pascal VOC 2007/2008. We demonstrate that large codebook vocabulary sizes completely deteriorate the performance of the traditional model, whereas the proposed model performs consistently. Moreover, we show that our method profits in high-dimensional feature spaces and reaps higher benefits when increasing the number of image categories.

Brain responses strongly correlate with Weibull image statistics when processing natural images.

The visual appearance of natural scenes is governed by a surprisingly simple hidden structure. The distributions of contrast values in natural images generally follow a Weibull distribution, with beta and gamma as free parameters. Beta and gamma seem to structure the space of natural images in an ecologically meaningful way, in particular with respect to the fragmentation and texture similarity within an image. Since it is often assumed that the brain exploits structural regularities in natural image statistics to efficiently encode and analyze visual input, we here ask ourselves whether the brain approximates the beta and gamma values underlying the contrast distributions of natural images. We present a model that shows that beta and gamma can be easily estimated from the outputs of X-cells and Y-cells. In addition, we covaried the EEG responses of subjects viewing natural images with the beta and gamma values of those images. We show that beta and gamma explain up to 71% of the variance of the early ERP signal, substantially outperforming other tested contrast measurements. This suggests that the brain is strongly tuned to the image's beta and gamma values, potentially providing the visual system with an efficient way to rapidly classify incoming images on the basis of omnipresent low-level natural image statistics.

Spatio-temporal context for robust multitarget tracking.

In multitarget tracking, the main challenge is to maintain the correct identity of targets even under occlusions or when differences between the targets are small. The paper proposes a new approach to this problem by incorporating the context information. The context of a target in an image sequence has two components: the spatial context including the local background and nearby targets and the temporal context including all appearances of the targets that have been seen previously. The paper considers both aspects. We propose a new model for multitarget tracking based on the classification of each target against its spatial context. The tracker searches a region similar to the target while avoiding nearby targets. The temporal context is included by integrating the entire history of target appearance based on probabilistic principal component analysis (PPCA). We have developed a new incremental scheme that can learn the full set of PPCA parameters accurately online. The experiments show robust tracking performance under the condition of severe clutter, occlusions, and pose changes.

The semantic pathfinder: using an authoring metaphor for generic multimedia indexing.

This paper presents the semantic pathfinder architecture for generic indexing of multimedia archives. The semantic pathfinder extracts semantic concepts from video by exploring different paths through three consecutive analysis steps, which we derive from the observation that produced video is the result of an authoring-driven process. We exploit this authoring metaphor for machine-driven understanding. The pathfinder starts with the content analysis step. In this analysis step, we follow a data-driven approach of indexing semantics. The style analysis step is the second analysis step. Here, we tackle the indexing problem by viewing a video from the perspective of production. Finally, in the context analysis step, we view semantics in context. The virtue of the semantic pathfinder is its ability to learn the best path of analysis steps on a per-concept basis. To show the generality of this novel indexing approach, we develop detectors for a lexicon of 32 concepts and we evaluate the semantic pathfinder against the 2004 NIST TRECVID video retrieval benchmark, using a news archive of 64 hours. Top ranking performance in the semantic concept detection task indicates the merit of the semantic pathfinder for generic indexing of multimedia archives.

Sparse representation for coarse and fine object recognition.

This paper offers a sparse, multiscale representation of objects. It captures the object appearance by selection from a very large dictionary of Gaussian differential basis functions. The learning procedure results from the matching pursuit algorithm, while the recognition is based on polynomial approximation to the bases, turning image matching into a problem of polynomial evaluation. The method is suited for coarse recognition between objects and, by adding more bases, also for fine recognition of the object pose. The advantages over the common representation using PCA include storing sampled points for recognition is not required, adding new objects to an existing data set is trivial because retraining other object models is not needed, and significantly in the important case where one has to scan an image over multiple locations in search for an object, the new representation is readily available as opposed to PCA projection at each location. The experimental result on the COIL-100 data set demonstrates high recognition accuracy with real-time performance.

Robust photometric invariant features from the color tensor.

Luminance-based features are widely used as low-level input for computer vision applications, even when color data is available. The extension of feature detection to the color domain prevents information loss due to isoluminance and allows us to exploit the photometric information. To fully exploit the extra information in the color data, the vector nature of color data has to be taken into account and a sound framework is needed to combine feature and photometric invariance theory. In this paper, we focus on the structure tensor, or color tensor, which adequately handles the vector nature of color images. Further, we combine the features based on the color tensor with photometric invariant derivatives to arrive at photometric invariant features. We circumvent the drawback of unstable photometric invariants by deriving an uncertainty measure to accompany the photometric invariant derivatives. The uncertainty is incorporated in the color tensor, hereby allowing the computation of robust photometric invariant features. The combination of the photometric invariance theory and tensor-based features allows for detection of a variety of features such as photometric invariant edges, corners, optical flow, and curvature. The proposed features are tested for noise characteristics and robustness to photometric changes. Experiments show that the proposed features are robust to scene incidental events and that the proposed uncertainty measure improves the applicability of full invariants.

Combining strings and necklaces for interactive three-dimensional segmentation of spinal images using an integral deformable spine model.

Segmentation of the spine directly from three-dimensional (3-D) image data is desirable to accurately capture its morphological properties. We describe a method that allows true 3-D spinal image segmentation using a deformable integral spine model. The method learns the appearance of vertebrae from multiple continuous features recorded along vertebra boundaries in a given training set of images. Important summarizing statistics are encoded into a necklace model on which landmarks are differentiated on their free dimensions. The landmarks are used within a priority segmentation scheme to reduce the complexity of the segmentation problem. Necklace models are coupled by string models. The string models describe in detail the biological variability in the appearance of spinal curvatures from multiple continuous features recorded in the training set. In the segmentation phase, the necklace and string models are used to interactively detect vertebral structures in new image data via elastic deformation reminiscent of a marionette with strings allowing for movement between interrelated structures. Strings constrain the deformation of the spine model within feasible solutions. The driving application in this work is analysis of computed tomography scans of the human lumbar spine. An illustration of the segmentation process shows that the method is promising for segmentation of the spine and for assessment of its morphological properties.

Population-based incremental interactive concept learning for image retrieval by stochastic string segmentations.

We propose a method for concept-based medical image retrieval that is a superset of existing semantic-based image retrieval methods. We conceive of a concept as an incremental and interactive formalization of the user's conception of an object in an image. The premise is that such a concept is closely related to a user's specific preferences and subjectivity and, thus, allows to deal with the complexity and content-dependency of medical image content. We describe an object in terms of multiple continuous boundary features and represent an object concept by the stochastic characteristics of an object population. A population-based incrementally learning technique, in combination with relevance feedback, is then used for concept customization. The user determines the speed and direction of concept customization using a single parameter that defines the degree of exploration and exploitation of the search space. Images are retrieved from a database in a limited number of steps based upon the customized concept. To demonstrate our method we have performed concept-based image retrieval on a database of 292 digitized X-ray images of cervical vertebrae with a variety of abnormalities. The results show that our method produces precise and accurate results when doing a direct search. In an open-ended search our method efficiently and effectively explores the search space.

Fast occluded object tracking by a robust appearance filter.

We propose a new method for object tracking in image sequences using template matching. To update the template, appearance features are smoothed temporally by robust Kalman filters, one to each pixel. The resistance of the resulting template to partial occlusions enables the accurate detection and handling of more severe occlusions. Abrupt changes of lighting conditions can also be handled, especially when photometric invariant color features are used. The method has only a few parameters and is computationally fast enough to track objects in real time.

Fast anisotropic Gauss filtering.

We derive the decomposition of the anisotropic Gaussian in a one-dimensional (1-D) Gauss filter in the x-direction followed by a 1-D filter in a nonorthogonal direction phi. So also the anisotropic Gaussian can be decomposed by dimension. This appears to be extremely efficient from a computing perspective. An implementation scheme for normal convolution and for recursive filtering is proposed. Also directed derivative filters are demonstrated. For the recursive implementation, filtering an 512 x 512 image is performed within 40 msec on a current state of the art PC, gaining over 3 times in performance for a typical filter, independent of the standard deviations and orientation of the filter. Accuracy of the filters is still reasonable when compared to truncation error or recursive approximation error. The anisotropic Gaussian filtering method allows fast calculation of edge and ridge maps, with high spatial and angular accuracy. For tracking applications, the normal anisotropic convolution scheme is more advantageous, with applications in the detection of dashed lines in engineering drawings. The recursive implementation is more attractive in feature detection applications, for instance in affine invariant edge and ridge detection in computer vision. The proposed computational filtering method enables the practical applicability of orientation scale-space analysis.

Tracking nonparameterized object contours in video.

We propose a new method for contour tracking in video. The inverted distance transform of the edge map is used as an edge indicator function for contour detection. Using the concept of topographical distance, the watershed segmentation can be formulated as a minimization. This new viewpoint gives a way to combine the results of the watershed algorithm on different surfaces. In particular, our algorithm determines the contour as a combination of the current edge map and the contour, predicted from the tracking result in the previous frame. We also show that the problem of background clutter can be relaxed by taking the object motion into account. The compensation with object motion allows to detect and remove spurious edges in background. The experimental results confirm the expected advantages of the proposed method over the existing approaches.

Electropermeabilization and fluorescent tracer exchange: the role of whole-cell capacitance.

Transmembrane crossing of charged fluorescent tracers such as propidium iodide (PI) and carboxyfluorescein+ (CF) can be used to quantitate membrane permeabilization. Murine myeloma Sp2/0-Ag14 cells were loaded with CF (0.1 fmol/cell) before electropulsation (0.5-3.0 kV/cm, 40 microseconds) in medium containing 25-50 micrograms/ml PI at 21-23 degrees C. Cytograms of PI vs. CF fluorescence showed three readily distinguishable subpopulations: 1) intact living cells with CF but without PI (these form > 95% of the prepulsed population), 2) transiently electropermeabilized but resealed cells showing both CF and low-level PI fluorescence, and 3) permanently permeabilized cells without CF but with very high PI fluorescence. Despite the ready influx of PI, the efflux of CF from transiently permeabilized cells was negligible and was insensitive to pulse parameters; however, electrically killed cells (subpopulation 3) lost all CF fluorescence and probably lost their cytoplasm. This difference in transmembrane passage of the dyes is best explained by binding of intracellular CF to macromolecules (and/or organelles). In isotonic "pulse medium," the membranes resealed after electropulsing with a time constant (tau R) of about 2 min. In 150 mOsm medium, resealing was faster (typically tau R approximately 0.5 min). The population distribution of PI uptake [coefficient of variation (CV) > 40%] was very broad and could not be accounted for by the radius dependence of pulse-induced voltage (CVradius approximately 10%). The variability in PI uptake could be explained if the electrical energy of the charged membrane, which depends on the whole-cell capacitance (Cc), was taken into account. Evaluation of the Cc values with single-cell resolution was based on measurement of the electrical charging time constant of the plasma membrane by electrorotation.

Radio-frequency microtools for particle and liver cell manipulation.

Single particles can be manipulated by applying high frequencies to ultramicro electrode arrays fabricated on planar structures. Heat production can be reduced to the extent that intense electric fields can be applied even to unmodified cell culture media. Animal cells grow normally in the high field (up to 100 kV/m) between such continuously energized multielectrodes. As with laser tweezers [1-3], this technique can capture particles and cells in field traps, generate linear movement, and permit cell cultivation. It can also produce micropatterns of pH gradients, field-cast objects, and control cell adhesion. These microtools may be combined to develop cell separators, microsensors, and controlled-biocompatibility surfaces.

DNA, protein, and plasma-membrane incorporation by arrested mammalian cells.

Incorporation of DNA, protein, and plasma membrane during blockage by aphidicolin or by doxorubicin was studied by flow cytometry and electrorotation of three cell lines (mouse-myeloma Sp2/0-Ag14, hybridoma H73C11, and fibroblast-like L929 cells). Drug-mediated arrest at the G1-S boundary (aphidicolin) or in G2/M (doxorubicin) did not arrest synthesis of either protein or total membrane area, the increases in which outstripped growth in cell volume and apparent cell area, respectively. Measurements of membrane capacity in normal and hypo-osmotic media showed that the drugs had not changed the fundamental bilayer, but that an increase in the number or size of microvilli must have occurred. Aphidicolin-arrested cells withstood hypo-osmotic stress better than untreated cells could, indicating that the membrane excess can be utilized as a reserve during rapid cell expansion. Hypo-osmotically treated cell populations exhibited only about half the coefficient of variance (CV) in membrane properties of cells at physiological osmolality. Populations of arrested cells exhibited the same high CV as asynchronous cells, indicating that chemical arrest does not give uniformly villated cell populations. However, the lowest CV values were given by some synchronized (aphidicolin-blocked, then released) populations. Removal of aphidicolin allowed most cells to progress through S and G2, and then divide. During these processes, the membrane excess was reduced. After removal of doxorubicin, the cells did not divide: some continued protein synthesis, grew abnormally large, and further increased their membrane excess. Membrane breakdown by electric pulsing (3 x 5kV/cm, 40 microseconds decay time) of aphidicolin-synchronized L cells in G2/M led to a 22% loss of plasma membrane (both the area-specific and the whole-cell capacitance were reduced), presumably via endocytosis-like vesiculation.

Effect of electric field pulses on the viability and on the membrane-bound immunoglobulins of LPS-activated murine B-lymphocytes: correlation with the cell cycle.

The effects of microsecond electropulses (1-5 kV/cm) on the viability of murine B lymphocytes and on their binding of antibodies by surface immunoglobulin (Ig) were studied in relation to the cell cycle. Before electropulsing, cultures given 48 h mitogenic stimulation showed at least two cell subpopulations, which were distinguishable by their levels of surface-Ig expression as assessed with FITC-labelled antibodies against mouse Ig. The immunofluorescence intensity of cells in S and G2/M phases was higher than that of G0/G1 cells. After exposure of the mitogen-stimulated lymphocytes to three exponentially decaying (time constant tau = 5-40 microseconds) electric field pulses, dye exclusion assay showed that pulsing at 1 or 2 kV/cm (at 4 degrees C or 20 degrees C) did not cause permeabilization. Field strengths of 3, 4, or 5 kV/cm resulted in 20%, 45%, or 70% of dye-permeable cells, respectively, if the pulsed cells were transferred to phosphate-buffered saline on ice for 30 min. Incubation in full medium at 37 degrees C for 30 min ("resealing") significantly decreased the percentage of permeabilized cells. Electropulsed G0/G1 cells were not only more resistant to direct electric exposure (tolerated higher field strengths) than S + G2/M cells but also responded better to resealing. The surface Ig of lymphocytes pulsed at higher fields and low temperature (4 or 5 kV/cm, tau = 5 microseconds, three pulses, 4 degrees C) was less easily immunostained than in controls or in cells pulsed at 2 kV/cm or less. At 5 kV/cm those cells that were not permeabilized showed a greater reduction in immunostaining, especially if resealed.