Haptic perception using optoelectronic robotic flesh for embodied artificially intelligent agents.

Flesh encodes a variety of haptic information including deformation, temperature, vibration, and damage stimuli using a multisensory array of mechanoreceptors distributed on the surface of the human body. Currently, soft sensors are capable of detecting some haptic stimuli, but whole-body multimodal perception at scales similar to a human adult (surface area ~17,000 square centimeters) is still a challenge in artificially intelligent agents due to the lack of encoding. This encoding is needed to reduce the wiring required to send the vast amount of information transmitted to the processor. We created a robotic flesh that could be further developed for use in these agents. This engineered flesh is an optical, elastomeric matrix "innervated" with stretchable lightguides that encodes haptic stimuli into light: temperature into wavelength due to thermochromic dyes and forces into intensity due to mechanical deformation. By exploiting the optical properties of the constitutive materials and using machine learning, we infer spatiotemporal, haptic information from light that is read by an image sensor. We demonstrate the capabilities of our system in various assemblies to estimate temperature, contact location, normal and shear force, gestures, and damage from temporal snapshots of light coming from the entire haptic sensor with errors <5%.

Watch-n-Patch: Unsupervised Learning of Actions and Relations.

There is a large variation in the activities that humans perform in their everyday lives. We consider modeling these composite human activities which comprises multiple basic level actions in a completely unsupervised setting. Our model learns high-level co-occurrence and temporal relations between the actions. We consider the video as a sequence of short-term action clips, which contains human-words and object-words. An activity is about a set of action-topics and object-topics indicating which actions are present and which objects are interacting with. We then propose a new probabilistic model relating the words and the topics. It allows us to model long-range action relations that commonly exist in the composite activities, which is challenging in previous works. We apply our model to the unsupervised action segmentation and clustering, and to a novel application that detects forgotten actions, which we call action patching. For evaluation, we contribute a new challenging RGB-D activity video dataset recorded by the new Kinect v2, which contains several human daily activities as compositions of multiple actions interacting with different objects. Moreover, we develop a robotic system that watches and reminds people using our action patching algorithm. Our robotic setup can be easily deployed on any assistive robots.

Tracking evolving communities in large linked networks.

We are interested in tracking changes in large-scale data by periodically creating an agglomerative clustering and examining the evolution of clusters (communities) over time. We examine a large real-world data set: the NEC CiteSeer database, a linked network of >250,000 papers. Tracking changes over time requires a clustering algorithm that produces clusters stable under small perturbations of the input data. However, small perturbations of the CiteSeer data lead to significant changes to most of the clusters. One reason for this is that the order in which papers within communities are combined is somewhat arbitrary. However, certain subsets of papers, called natural communities, correspond to real structure in the CiteSeer database and thus appear in any clustering. By identifying the subset of clusters that remain stable under multiple clustering runs, we get the set of natural communities that we can track over time. We demonstrate that such natural communities allow us to identify emerging communities and track temporal changes in the underlying structure of our network data.