Reflections on Cross-Modal Correspondences: Current Understanding and Issues for Future Research.

The past two decades have seen an explosion of research on cross-modal correspondences. Broadly speaking, this term has been used to encompass associations between and among features, dimensions, or attributes across the senses. There has been an increasing interest in this topic amongst researchers from multiple fields (psychology, neuroscience, music, art, environmental design, etc.) and, importantly, an increasing breadth of the topic's scope. Here, this narrative review aims to reflect on what cross-modal correspondences are, where they come from, and what underlies them. We suggest that cross-modal correspondences are usefully conceived as relative associations between different actual or imagined sensory stimuli, many of these correspondences being shared by most people. A taxonomy of correspondences with four major kinds of associations (physiological, semantic, statistical, and affective) characterizes cross-modal correspondences. Sensory dimensions (quantity/quality) and sensory features (lower perceptual/higher cognitive) correspond in cross-modal correspondences. Cross-modal correspondences may be understood (or measured) from two complementary perspectives: the phenomenal view (perceptual experiences of subjective matching) and the behavioural response view (observable patterns of behavioural response to multiple sensory stimuli). Importantly, we reflect on remaining questions and standing issues that need to be addressed in order to develop an explanatory framework for cross-modal correspondences. Future research needs (a) to understand better when (and why) phenomenal and behavioural measures are coincidental and when they are not, and, ideally, (b) to determine whether different kinds of cross-modal correspondence (quantity/quality, lower perceptual/higher cognitive) rely on the same or different mechanisms.

Interactions of Lemon, Sucrose and Citric Acid in Enhancing Citrus, Sweet and Sour Flavors.

Flavorants such as lemon extract that activate olfactory receptors may also evoke or enhance flavor qualities such as sour and sweet that are typically considered gustatory. Similarly, flavorants such as sucrose and citric acid that activate gustatory receptors may enhance flavors such as citrus that are typically considered olfactory. Here, we ask how lemon extract, sucrose, and citric acid, presented separately and together, affect sweet, sour, and citrus flavors. We accomplished this by testing, in the same 12 subjects, lemon extract and sucrose (Experiment 1), lemon extract and citric acid (Experiment 2), and lemon extract, sucrose, and citric acid (Experiment 3). Results showed that both lemon extract and citric acid increased the ratings of citrus and sour intensity. Lemon extract did not affect sweet, but citric acid did, mainly in Experiment 3. Sucrose systematically increased only sweet intensity and modulated the effect of lemon extract on sour. The most robust multiquality effect was the enhancement of sour by lemon extract. These outcomes suggest, first, a role played by experience with the statistical associations of gustatory and olfactory flavorants and, second, that lemon flavor is complex, having citrus and sour qualities that may not be fully separable in perception.

Contextual Effects in Judgments of Taste Intensity: No Assimilation, Sometimes Contrast.

Judgments of taste intensity often show contextual contrast but not assimilation, even though both effects of stimulus context appear in other sense modalities, such as hearing. Four experiments used a paradigm that shifts the stimulus context within a test session in order to seek evidence of assimilation in judgments of the taste intensity of sucrose and, for comparison, the loudness of 500-Hz tones. Experiment 1 found no assimilation in taste using three response scales, magnitude estimation, labeled magnitude, and visual analog, but did find evidence of contrast. Experiments 2 and 3 found no clear evidence of either assimilation or contrast in taste, but found consistent evidence of assimilation in loudness. Experiment 4 found no assimilation in loudness, however, when the intervals between successive stimuli increased from about 6 to 30 s in order to match the interval used with sucrose in Experiments 1 to 3. Taken together, these findings suggest that the assimilation found in intensity judgments in other sensory modalities may not appear in taste perception because of the slower rates presenting of taste stimuli.

The semantic basis of taste-shape associations.

Previous research shows that people systematically match tastes with shapes. Here, we assess the extent to which matched taste and shape stimuli share a common semantic space and whether semantically congruent versus incongruent taste/shape associations can influence the speed with which people respond to both shapes and taste words. In Experiment 1, semantic differentiation was used to assess the semantic space of both taste words and shapes. The results suggest a common semantic space containing two principal components (seemingly, intensity and hedonics) and two principal clusters, one including round shapes and the taste word "sweet," and the other including angular shapes and the taste words "salty," "sour," and "bitter." The former cluster appears more positively-valenced whilst less potent than the latter. In Experiment 2, two speeded classification tasks assessed whether congruent versus incongruent mappings of stimuli and responses (e.g., sweet with round versus sweet with angular) would influence the speed of participants' responding, to both shapes and taste words. The results revealed an overall effect of congruence with congruent trials yielding faster responses than their incongruent counterparts. These results are consistent with previous evidence suggesting a close relation (or crossmodal correspondence) between tastes and shape curvature that may derive from common semantic coding, perhaps along the intensity and hedonic dimensions.

Flavor Identification and Intensity: Effects of Stimulus Context.

Two experiments presented oral mixtures containing different proportions of the gustatory flavorant sucrose and an olfactory flavorant, either citral (Experiment 1) or lemon (Experiment 2). In 4 different sessions of each experiment, subjects identified each mixture as "mostly sugar" or "mostly citrus/lemon" or rated the perceived intensities of the sweet and citrus components. Different sessions also presented the mixtures in different contexts, with mixtures containing relatively high concentrations of sucrose or citral/lemon presented more often (skew sucrose or skew citral/lemon). As expected, in both experiments, varying stimulus context affected both identification and perceived intensity: Skewing to sucrose versus citral/lemon decreased the probability of identifying the stimuli as "mostly sugar" and reduced the ratings of sweet intensity relative to citrus intensity. Across both contextual conditions of both experiments, flavor identification associated closely with the ratio of the perceived sweet and citrus intensities. The results accord with a model, extrapolated from signal-detection theory, in which sensory events are represented as multisensory-multidimensional distributions in perceptual space. Changing stimulus context can shift the locations of the distributions relative to response criteria, Decision rules guide judgments based on both sensory events and criteria, these rules not necessarily being identical in tasks of identification and intensity rating.

Response Times to Gustatory-Olfactory Flavor Mixtures: Role of Congruence.

A mixture of perceptually congruent gustatory and olfactory flavorants (sucrose and citral) was previously shown to be detected faster than predicted by a model of probability summation that assumes stochastically independent processing of the individual gustatory and olfactory signals. This outcome suggests substantial integration of the signals. Does substantial integration also characterize responses to mixtures of incongruent flavorants? Here, we report simple response times (RTs) to detect brief pulses of 3 possible flavorants: monosodium glutamate, MSG (gustatory: "umami" quality), citral (olfactory: citrus quality), and a mixture of MSG and citral (gustatory-olfactory). Each stimulus (and, on a fraction of trials, water) was presented orally through a computer-operated, automated flow system, and subjects were instructed to press a button as soon as they detected any of the 3 non-water stimuli. Unlike responses previously found to the congruent mixture of sucrose and citral, responses here to the incongruent mixture of MSG and citral took significantly longer (RTs were greater) and showed lower detection rates than the values predicted by probability summation. This outcome suggests that the integration of gustatory and olfactory flavor signals is less extensive when the component flavors are perceptually incongruent rather than congruent, perhaps because incongruent flavors are less familiar.

Lower pitch is larger, yet falling pitches shrink.

Experiments using diverse paradigms, including speeded discrimination, indicate that pitch and visually-perceived size interact perceptually, and that higher pitch is congruent with smaller size. While nearly all of these studies used static stimuli, here we examine the interaction of dynamic pitch and dynamic size, using Garner's speeded discrimination paradigm. Experiment 1 examined the interaction of continuous rise/fall in pitch and increase/decrease in object size. Experiment 2 examined the interaction of static pitch and size (steady high/low pitches and large/small visual objects), using an identical procedure. Results indicate that static and dynamic auditory and visual stimuli interact in opposite ways. While for static stimuli (Experiment 2), higher pitch is congruent with smaller size (as suggested by earlier work), for dynamic stimuli (Experiment 1), ascending pitch is congruent with growing size, and descending pitch with shrinking size. In addition, while static stimuli (Experiment 2) exhibit both congruence and Garner effects, dynamic stimuli (Experiment 1) present congruence effects without Garner interference, a pattern that is not consistent with prevalent interpretations of Garner's paradigm. Our interpretation of these results focuses on effects of within-trial changes on processing in dynamic tasks and on the association of changes in apparent size with implied changes in distance. Results suggest that static and dynamic stimuli can differ substantially in their cross-modal mappings, and may rely on different processing mechanisms.

Identification of gustatory-olfactory flavor mixtures: effects of linguistic labeling.

Two experiments, using different ranges and numbers of stimuli, examined how linguistic labels affect the identification of flavor mixtures containing different proportions of sucrose (gustatory flavorant) and citral (olfactory flavorant). Both experiments asked subjects to identify each stimulus as having either "mostly sugar" or "mostly citrus." In one condition, no labels preceded the flavor stimuli. In another condition, each flavor stimulus followed a label, either SUGAR or CITRUS, which, the subjects were informed, usually though not always named the stronger flavor component; that is, the labels were probabilistically valid. The results of both experiments showed that the labels systematically modified the identification responses: Subjects responded "sugar" or "citrus" more often when the flavor stimulus followed the corresponding label, SUGAR or CITRUS. But the labels hardly affected overall accuracy of identification. Accuracy was possibly limited, however, by both the confusability of the flavor stimuli per se and the way that confusability could limit the opportunity to discern the probabilistic associations between labels and individual flavor stimuli. We describe the results in terms of a decision-theoretic model, in which labels induce shifts in response criteria governing the identification responses, or possibly effect changes in the sensory representations of the flavorants themselves.

Flavor-intensity perception: effects of stimulus context.

Stimulus context affects judgments of intensity of both gustatory and olfactory flavors, and the contextual effects are modality-specific. Does context also exert separate effects on the gustatory and olfactory components of flavor mixtures? To answer this question, in each of 4 experiments, subjects rated the perceived intensity of 16 mixtures constructed by combining 4 concentrations of the gustatory flavorant sucrose with 4 concentrations of the retronasal olfactory flavorant citral. In 1 contextual condition of each experiment, concentrations of sucrose were relatively high and those of citral low; in the other condition, the relative concentrations of sucrose and citral reversed. There were 2 main results: First, consistent with earlier findings, in 5 of the 8 conditions, the ratings were consistent with linear addition of perceived sucrose and citral; departures from additivity appeared, however, in 3 conditions where the relative concentrations of citral were high. Second, changes in context produced contrast (adaptation-like changes) in perceived intensity: The contribution to perceived intensity of a given concentration of a flavorant was smaller when the contextual concentrations of that flavorant were high rather than low. A notable exception was the absence of contextual effects on the perceived intensity of near-threshold citral. These findings suggest that the contextual effects may arise separately in the gustatory and olfactory channels, prior to the integration of perceived flavor intensity.

Detecting gustatory-olfactory flavor mixtures: models of probability summation.

Odorants and flavorants typically contain many components. It is generally easier to detect multicomponent stimuli than to detect a single component, through either neural integration or probability summation (PS) (or both). PS assumes that the sensory effects of 2 (or more) stimulus components (e.g., gustatory and olfactory components of a flavorant) are detected in statistically independent channels, that each channel makes a separate decision whether a component is detected, and that the behavioral response depends solely on the separate decisions. Models of PS traditionally assume high thresholds for detecting each component, noise being irrelevant. The core assumptions may be adapted, however, to signal-detection theory, where noise limits detection. The present article derives predictions of high-threshold and signal-detection models of independent-decision PS in detecting gustatory-olfactory flavorants, comparing predictions in yes/no and 2-alternative forced-choice tasks using blocked and intermixed stimulus designs. The models also extend to measures of response times to suprathreshold flavorants. Predictions derived from high-threshold and signal-detection models differ markedly. Available empirical evidence on gustatory-olfactory flavor detection suggests that neither the high-threshold nor the signal-detection versions of PS can readily account for the results, which likely reflect neural integration in the flavor system.

Post-transient shifts in auditory lateralization: effects of target level.

Exposing one ear to a series of brief 80 dB sound pressure level (SPL) inducing tones reduces the tendency to lateralize subsequent target tones to that ear and shifts the point of subjective equality (PSE) toward the unexposed ear. Furthermore, targets with average SPLs of 60 and 80 dB at the two ears showed similar changes in PSE. These results support and extend earlier findings of Arieh and Marks (2007). Percept. Psychophys. 69, 523-528 and suggest the presence of a mechanism, located at least partially before the site of binaural integration, that depresses the magnitude of intensity information from the ear in response to moderately intense transient stimuli. Assuming that lateralization depends on the ratio of the magnitude of intensity information from the two ears that reach the central integrator, the results imply that the inducer reduced the representations of magnitude of 60 and 80 dB test tones in equal proportion.

Coactivation of gustatory and olfactory signals in flavor perception.

It is easier to detect mixtures of gustatory and olfactory flavorants than to detect either component alone. But does the detection of mixtures exceed the level predicted by probability summation, assuming independent detection of each component? To answer this question, we measured simple response times (RTs) to detect brief pulses of one of 3 flavorants (sucrose [gustatory], citral [olfactory], sucrose-citral mixture) or water, presented into the mouth by a computer-operated, automated flow system. Subjects were instructed to press a button as soon as they detected any of the 3 nonwater stimuli. Responses to the mixtures were faster (RTs smaller) than predicted by a model of probability summation of independently detected signals, suggesting positive coactivation (integration) of gustation and retronasal olfaction in flavor perception. Evidence for integration appeared mainly in the fastest 60% of the responses, indicating that integration arises relatively early in flavor processing. Results were similar when the 3 possible flavorants, and water, were interleaved within the same session (experimental condition), and when each flavorant was interleaved with water only (control conditions). This outcome suggests that subjects did not attend selectively to one flavor component or the other in the experimental condition and further supports the conclusion that (late) decisional or attentional strategies do not exert a large influence on the gustatory-olfactory flavor integration.

Absolute pitch in children prior to the beginning of musical training.

Absolute pitch (AP) is a rare skill, historically defined as the ability to name notes. Until now, methodologic limitations made it impossible to directly test the extent to which the development of AP depends on musical training. Using a new paradigm, we tested children with minimal musical experience. Although most children performed poorly, two performed comparably to adult possessors of AP. Follow-up testing showed that the performance of both children progressed to that of "classic" AP. These data support the theory that AP can result from differences in the encoding of stimulus frequency that are independent of musical experience.

Taste coding after selective inhibition by chlorhexidine.

Coding of the complex tastes of ionic stimuli in humans was studied by combining taste confusion matrix (TCM) methodology and treatment with chlorhexidine gluconate. The TCM evaluates discrimination of multiple stimuli simultaneously. Chlorhexidine, a bis-biguanide antiseptic, reversibly inhibits salty taste and tastes of a subset of bitter stimuli, including quinine hydrochloride. Identifications of salty (NaCl, "salt"), bitter (quinine.HCl, "quinine"), sweet (sucrose, "sugar"), and sour (citric acid, "acid") prototypes, alone and as components of binary mixtures, were measured under 4 conditions. One was a water-rinse control and the others had the salt and quinine tastes progressively reduced by treatment with 1 mM chlorhexidine, 3 mM chlorhexidine, and ultimately to zero by elimination of NaCl and quinine.HCl. Treatment with chlorhexidine perturbed identification of salt more than quinine; both were thereafter more often confused with "water" and unidentified when mixed with sucrose or citric acid. All pairwise discriminations that depended on the tastes of NaCl and quinine.HCl deteriorated, and although H(2)O was mistakenly identified as quinine after chlorhexidine, this may have been a decisional bias. Other confusions reflected "unprompted mixture analysis" and an obscuring of salt taste by a less-inhibited stronger quinine or sugar or acid tastes in mixtures. Partial inhibition of the tastes of NaCl and quinine.HCl by chlorhexidine was considered in the context of multiple receptors for the 2 compounds. Discrimination among prototypic stimuli with varying strengths was consistent with a gustatory system that evaluates a small number of independent tastes.

Intramodal and cross-modal discrimination of curvature: Haptic touch versus vision.

A total of 60 subjects, 20 in each experimental condition, gave 'same-different' judgments to pairs of stimuli differing in radius of curvature. Stimuli were presented intramodally to vision, intramodally to haptic touch, and cross-modally to vision and haptic touch. Results showed that performance, quantified by the measure d', differed among the three modality conditions, being best in vision and poorest in haptics, with cross-modal performance falling roughly mid-way between. Unimodal visual performance exceeded cross-modal performance by about one d' unit, and cross-modal performance similarly exceeded unimodal haptic performance by about one d' unit. The study reveals the relative differences in the discrimination of curvatures of objects in vision and haptics.

Cross-modal interaction between vision and hearing: a speed-accuracy analysis.

Cross-modal facilitation of response time (RT) is said to occur in a selective attention task when the introduction of an irrelevant sound increases the speed at which visual stimuli are detected and identified. To investigate the source of the facilitation in RT, we asked participants to rapidly identify the color of lights in the quiet and when accompanied by a pulse of noise. The resulting measures of accuracy and RT were used to derive speed-accuracy trade-off functions (SATFs) separately for the noise and the no-noise conditions. The two resulting SATFs have similar slopes and intercepts and, thus, can be treated as overlapping segments of a single function. That speeded identification of color with and without the presence of noise can be described by one SATF suggests, in turn, that cross-modal facilitation of RT represents a change in decision criterion induced by the auditory stimulus. Analogous changes in decision criteria might also underlie other measures of cross-modal interactions, such as auditory enhancement of brightness judgments.

Posttransient shifts in auditory lateralization.

Presenting an intense (e.g., 80-dB [SPL]) "transient" (e.g., 50-msec) inducer to the ear reduces the loudness of subsequent signals at or near the frequency of the inducer. In this study, we ask whether similar inducers also affect lateralization. In two experiments, we asked how inducing tones presented to one ear (the exposed ear) affect judgments of the lateral position of subsequent target tones having various interaural intensity differences. In Experiment 1, inducers had the same frequency as the targets, and, as predicted, reduced the tendency to lateralize the targets to the exposed ear. In Experiment 2, the frequency of the inducers and the target differed (different critical bands), thereby eliminating the effect on lateralization. These results are consistent with the hypothesis that inducers temporarily reduce the magnitude of the representation of intensity signals in the frequency region around them and that this reduction occurs, at least partly, peripherally to the site at which binaural intensity differences are encoded. The results imply further that the reduction in loudness previously reported under similar stimulus conditions reflects a more general reduction of intensity-based information in hearing.

Memory for curvature of objects: haptic touch vs. vision.

The present study examined the role of vision and haptics in memory for stimulus objects that vary along the dimension of curvature. Experiment 1 measured haptic-haptic (T-T) and haptic-visual (T-V) discrimination of curvature in a short-term memory paradigm, using 30-second retention intervals containing five different interpolated tasks. Results showed poorest performance when the interpolated tasks required spatial processing or movement, thereby suggesting that haptic information about shape is encoded in a spatial-motor representation. Experiment 2 compared visual-visual (V-V) and visual-haptic (V-T) short-term memory, again using 30-second delay intervals. The results of the ANOVA failed to show a significant effect of intervening activity. Intra-modal visual performance and cross-modal performance were similar. Comparing the four modality conditions (inter-modal V-T, T-V; intra-modal V-V, T-T, by combining the data of Experiments 1 and 2), in a global analysis, showed a reliable interaction between intervening activity and experiment (modality). Although there appears to be a general tendency for spatial and movement activities to exert the most deleterious effects overall, the patterns are not identical when the initial stimulus is encoded haptically (Experiment 1) and visually (Experiment 2).