Localizing, describing, interpreting: effects of different audio text structures on attributing meaning to digital pictures.

Based on previous research on multimedia learning and text comprehension, an eye-tracking study was conducted to examine the influence of audio text coherence on visual attention and memory in a multimedia learning situation with a focus on picture comprehension. Audio text coherence was manipulated by the type of LDI structure, that is, whether localization, description, and interpretation followed in immediate succession for each pictorial detail or whether localizations and description of details were separated from their interpretation. Results show that with a LDI integrated structure compared to a LDI separated structure the referred-to picture elements were fixated longer during interpretation parts, and linkages between descriptions and interpretations were better recalled and recognized. The effects on recall and recognition of linkages were fully mediated by fixation times. This pattern of results can be explained by an interplay between audio text coherence and dual coding processes. It points out the importance of local coherence and the provision of localization information in audio explanations as well as visual attention to allow for dual coding processes that can be used to better attribute meaning to picture details. Practical implications for the design of educational videos, audio texts on websites, and audio guides are discussed.

Aesthetic Experience of Representational Art: Liking Is Affected by Audio-Information Naming and Explaining Inaccuracies of Historical Paintings.

Paintings in museums are often accompanied by additional information, such as titles or audio-texts. Previous research has reported mostly positive effects of additional information on the liking and subjective understanding of a painting. However, some studies have also reported negative effects when additional information introduces inconsistencies between the painting's content and the represented reality. Therefore, the present study examined the negative effects of naming a painting's historical inaccuracies, which are inconsistencies between the content of the painting and the real historic event, and whether these negative effects can be compensated by an explanation for the inaccuracies. The results revealed that liking was lower with inaccuracies named and that this effect was compensated by an explanation for the inaccuracies. No significant effects were observed for subjective understanding and aesthetic emotions. The results corroborate parts of the Vienna integrated model of art perception and have practical implications for the design of audio-texts in museums.

The Closer, The Better? Processing Relations Between Picture Elements in Historical Paintings.

The present eye-tracking study investigated how audio explanations influence perception and the cognitive processing of historical paintings. Spatially close and distant pairs of picture elements and their semantic relations were named in an audio text either immediately after each other or with descriptions of other elements in between. It was assumed that the number of backward fixation counts on the first of the two mentioned related picture elements should be higher if they are spatially close rather than spatially distant. There should also be more backward fixation counts if the elements are named temporally close rather than temporally distant. Similar predictions were made for the retention of these picture elements and their relations. A 2x2x2 within-subject design (n=36) with spatial distance (close vs. distant), temporal distance (close vs. distant) and painting (Leutze vs. West) revealed more background fixation counts for spatially close compared to spatially distant elements but just for the Leutze painting. Accordingly, the relations between the spatially close pairs were retained better than between the spatially distant pairs in the Leutze painting but vice versa for the West painting. The results are discussed with regard to the spatial contiguity principle of multimedia learning and research on text coherence.