Self-Organizing a Latent Hierarchy of Sketch Patterns for Controllable Sketch Synthesis.

Encoding sketches as Gaussian mixture model (GMM)-distributed latent codes is an effective way to control sketch synthesis. Each Gaussian component represents a specific sketch pattern, and a code randomly sampled from the Gaussian can be decoded to synthesize a sketch with the target pattern. However, existing methods treat the Gaussians as individual clusters, which neglects the relationships between them. For example, the giraffe and horse sketches heading left are related to each other by their face orientation. The relationships between sketch patterns are important messages to reveal cognitive knowledge in sketch data. Thus, it is promising to learn accurate sketch representations by modeling the pattern relationships into a latent structure. In this article, we construct a tree-structured taxonomic hierarchy over the clusters of sketch codes. The clusters with the more specific descriptions of sketch patterns are placed at the lower levels, while the ones with the more general patterns are ranked at the higher levels. The clusters at the same rank relate to each other through the inheritance of features from common ancestors. We propose a hierarchical expectation-maximization (EM)-like algorithm to explicitly learn the hierarchy, jointly with the training of encoder-decoder network. Moreover, the learned latent hierarchy is utilized to regularize sketch codes with structural constraints. Experimental results show that our method significantly improves controllable synthesis performance and obtains effective sketch analogy results.

Controllable stroke-based sketch synthesis from a self-organized latent space.

Learning to synthesize free-hand sketches controllably according to specified categories and sketching styles is a challenging task, due to the lack of training data with category labels and style labels. One choice to control the synthesis is by self-organizing a latent coding space to preserve the similarity of structural patterns of the observed data. A practical way is introducing a Gaussian mixture prior over the latent codes, where each Gaussian component represents a specific categorical or stylistic pattern. As a result, we can generate sketches by sampling the latent variables from the Gaussian components or continuously manipulating the latent representations by interpolation. To achieve robust controllable sketch synthesis, it is critical to determine an appropriate Gaussian number. An underestimated Gaussian number cannot fully represent all the sketch patterns, i.e., some clusters have to contain sketches with more than one pattern. An overestimated one introduces redundant components, usually representing a chaotic collection of sketches with diverse patterns featured by other components. Both cases disturb pattern clustering over the coding space and make the internal code generation difficult to control for specific patterns. However, the Gaussian number is unavailable in this unsupervised task. In this paper, we present Rival Penalized Competitive Learning pixel to sequence (RPCL-pix2seq) to automatically determine the Gaussian number. Both quantitative and qualitative experimental results show RPCL-pix2seq can partition the codes for the sketches into an approximate stable number of clusters. Hence, we are able to do synthesis reasoning over the latent space, generating novel but reasonable sketches which neither appear in the training dataset nor exist in real life.