A Robust Approach to Multimodal Deepfake Detection.

The widespread use of deep learning techniques for creating realistic synthetic media, commonly known as deepfakes, poses a significant threat to individuals, organizations, and society. As the malicious use of these data could lead to unpleasant situations, it is becoming crucial to distinguish between authentic and fake media. Nonetheless, though deepfake generation systems can create convincing images and audio, they may struggle to maintain consistency across different data modalities, such as producing a realistic video sequence where both visual frames and speech are fake and consistent one with the other. Moreover, these systems may not accurately reproduce semantic and timely accurate aspects. All these elements can be exploited to perform a robust detection of fake content. In this paper, we propose a novel approach for detecting deepfake video sequences by leveraging data multimodality. Our method extracts audio-visual features from the input video over time and analyzes them using time-aware neural networks. We exploit both the video and audio modalities to leverage the inconsistencies between and within them, enhancing the final detection performance. The peculiarity of the proposed method is that we never train on multimodal deepfake data, but on disjoint monomodal datasets which contain visual-only or audio-only deepfakes. This frees us from leveraging multimodal datasets during training, which is desirable given their lack in the literature. Moreover, at test time, it allows to evaluate the robustness of our proposed detector on unseen multimodal deepfakes. We test different fusion techniques between data modalities and investigate which one leads to more robust predictions by the developed detectors. Our results indicate that a multimodal approach is more effective than a monomodal one, even if trained on disjoint monomodal datasets.

Modal analysis of free archtop guitar top plates.

We analyze the modal response of the top plates of archtop guitars using free boundary conditions. Starting from the three-dimensional scan of a real archtop guitar, we build a mesh of its top plate using a non-invasive process. Once the mesh of the plate is built, we compute its vibrational response by finite element method simulations and perform many different analyses. The outer surface of the mesh matches the scan, while we retain the freedom to control the shape of the inner surface. This way we can change some of its aspects (e.g., thickness distribution) depending on what we intend to study. We investigate the similarities of its mode shapes with those of similar instruments (e.g., violin and classical guitar), analyze the carving process of the plate's inner surface and study the influence of resonant holes on its final vibratory response.

A data-driven approach to violin making.

Of all the characteristics of a violin, those that concern its shape are probably the most important ones, as the violin maker has complete control over them. Contemporary violin making, however, is still based more on tradition than understanding, and a definitive scientific study of the specific relations that exist between shape and vibrational properties is yet to come and sorely missed. In this article, using standard statistical learning tools, we show that the modal frequencies of violin tops can, in fact, be predicted from geometric parameters, and that artificial intelligence can be successfully applied to traditional violin making. We also study how modal frequencies vary with the thicknesses of the plate (a process often referred to as plate tuning) and discuss the complexity of this dependency. Finally, we propose a predictive tool for plate tuning, which takes into account material and geometric parameters.

Eigenfrequency optimisation of free violin plates.

We discuss how the modal response of violin plates changes as their shape varies. Starting with an accurate 3D scan of the top plate of a historic violin, we develop a parametric model that controls a smooth shaping of the interior of the plate, while guaranteeing that the boundary is the same as the original violin. This allows us to generate a family of violin tops whose shape can be smoothly controlled through various parameters that are meaningful to a violin maker: from the thickness in different areas of the top to the location, angle, and dimensions of the bass bar. We show that the interplay between the different parameters affects the eigenmodes of the plate frequencies in a nonlinear fashion. We also show that, depending on the parameters, the ratio between the fifth and the second eigenfrequencies can be set to match that used by celebrated violin makers of the Cremonese school. As the parameterisation that we define can be readily understood by violin makers, we believe that our findings can have a relevant impact on the violin making community, as they show how to steer geometric modifications of the violin to balance the eigenfrequencies of the free plates.