Influence of visual information during the percepction of prosody of actuated emotions
Keywords:
emotions, visual information, speech perceptionAbstract
The fact of the audiovisual integration during the speech perception has been stated in several researches. It has been argued that access to visual information during speech perception provides intelligibility in the perception of sound message process. From this perspective, this research is to determinate, through perceptive analysis, the influence that the access to the visual information performs during the process of perception of the prosody of emotions. To reach this aim, we have built a corpus of controlled audiovisual speech samples. We got these samples from professional actors' registers. Later, parts of those samples were manipulated. We designed an experiment for perception with 152 participants. They had to listen and look at the audiovisual samples in the perception test so we could determinate the emotion that they perceived and select it among the options we present to them. Through the analysis of the result we could determinate that the listerners evaluate the information from the visual and auditory source. The availability of visual information changes the perception of acoustic information. To conclude, the product of visual perception process is a correlation of visual information processing and sound.
References
ASSMANN, P. y Q. SUMMERFIELD. (2004): «The perception of speech under adverse conditions», en S. Greenberg y W. Ainsworth (eds.): Speech processing in the auditory system, Nueva York, Springer Verlag, pp. 231–308.
BARBOSA, A.; E. VATIKIOTIS-BATESON y H. YEHIA (2002): «Modeling the relation between speech acoustics and 3D facial motion», Technical Report of the Institute of Electronics, Information and Communication Engineers, 102 (735), pp. 13-18.
BERTELSON, P.; J. VROOMEN; B. DE GELDER y J. DRIVER (2000): «The ventrilo-quist effect does not depend on the direction of deliberate visual attention», Perception and Psychophysics, 62, pp. 321-332.
BIRDWHISTELL, R. (1979): El lenguaje de la expresión corporal, Barcelona, Gustavo Gili SA.
BLONDET, M. (2006): Variaciones de la velocidad de habla en español: patrones fonéticos y estrategias fonológicas. Un estudio desde la producción, tesis doctoral, Universidad de Los Andes (Mérida-Venezuela).
BORRÀS-COMES, J. y P. PRIETO. (2011): «‘Seeing tunes.' The role of visual gestures in tune interpretation», Laboratory Phonology, 2 (2), pp. 355-380.
BRUNER, J. (1984): Acción, pensamiento y lenguaje, Madrid, Alianza.
CHÓLIZ, M. (1995): «La expresión de las emociones en la obra de Darwin», en F. Tortosa, M. Civera y C. Catalayud (comps.): Prácticas de Historia de Psicología, Valencia, Promolibro.
COLLIGNON, O.; S. GIRARD; F. GOSSELIN; S. ROY; D. SAINT-AMOUR; M. LASSON-DE y F. LEPORE. (2008): «Audio-visual integration of emotion expression», Brain research, 1242, pp. 126-135.
CONDE LÓPEZ, R. (2005): Mapeo facial de emociones sintéticas.
http://tierra.aslab.upm.es/documents/PFC/PFC_RPConde.pdf. [10/10/11].
CLEARY, M. y D. PISONI. (1998): «Speech perception and spoken word recognition: Research and theory», Research on spoken language processing, 22, pp. 56-78.
DE GELDER, B. y J. Vroomen. (2000): «The perception of emotions by ear and by eye», Cognition & Emotion, 14, pp. 289-311.
DIJKSTRA, C.; E. KRAHMER y M. SWERTS (2006): «Manipulating uncertainty: The contribution of different audiovisual prosodic cues to the perception of confidence», Proceedings of the 3rd International Conference on Speech Prosody, Desden (Alemania). http://arno.uvt.nl/show.cgi?fid=95687 [12/03/2012].
EKMAN, P. y H. OSTER. (1981): «Expresión facial de las emociones», Estudios de Psicología, 7, pp. 116-144.
FAGEL, S. (2006): «Emotional McGurk effect», Proceedings of the 3rd International Conference on Speech Prosody, Desden (Alemania). http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.138.3943&rep=rep1&type=pdf. [14/10/11]. http://sprosig.isle.illinois.edu/sp2006/contents/papers/SPS2-6_0006.pdf [12/03/2012]
FRICKE, A. (2008): «Relating emotional content to speech rate in Brazilian Portuguese», en P. Barbosa, S. Madureira y C. Reis (eds.): Proceedings of 4th the Speech Prosody 2008 Conference, Campinas, Editora RG/CNPq, pp. 217-222.
HIETANEN, J.; J. LEPPÄNEN; M. ILLI y V. SURAKKA. (2004): «Evidence for the integration of audiovisual emotional information at the perceptual level of processing», European Journal of Cognitive Psychology, 16, pp. 769-790.
JIMÉNEZ, F. (1987): Marcadores emocionales en la conducta vocal, Madrid, Universidad Autónoma de Madrid.
JIMÉNEZ, F. y M. MALLO. (1988): «El reconocimiento de emociones a través de la voz», Estudios de psicología, 33-34, pp. 31-52.
JOHNSTONE, T. y K. SCHERER. (2000): «Vocal communication of emotion», en M. Lewis y J. Haviland (eds.): Handbook of emotions, Nueva York, Guilford Press, pp. 220-235.
KRAUSS R.; Y. CHEN y P. CHAWLA. (1996): «Nonverbal behavior and nonverbal communication: What do conversational hand gestures tell us?», en M. Zanna (ed.): Advances in experimental social psychology, San Diego, CA, Academic Press, pp. 389-450.
MALTIN, M. y H. FOLEY. (1996): Sensación y percepción, México, Prentice Hall.
MARRERO, V. (2001): Fonética perceptiva, Madrid, Universidad Nacional de Educación a Distancia.
MARTÍNEZ, H. (2009): Evaluación de la percepción y producción del habla, tesis doctoral, Universidad de Los Andes (Mérida, Venezuela).
MARTÍNEZ, H. y D. ROJAS. (2011): «Prosodia y emociones: datos acústicos, velocidad de habla y percepción de un corpus actuado», Lengua y Habla, 15, pp. 59-72. http://erevistas.saber.ula.ve/index.php/lenguayhabla/article/viewFile/3356/3257 [15/11/2011].
MASSARO, D. (1987): Speech perception by ear and by eye, Hillsdale, NJ, Erlbaum.
MASSARO D. y M. COHEN (1983): «Integration of visual and auditory information in speech perception», Journal of Experimental Psychology: Human Perception and Performance, 9, pp. 753-771.
MASSARO, D. y P. EGAN. (1996): «Perceiving affect from the voice and the face», Psychonomic Bulletin and Review, 3, pp. 215-221.
MACDONALD, I. y H. MCGURK. (1978): «Visual influences on speech perception process», Perception & Psychophysics, 24, pp. 253-257.
ROBINSON, C. y V. SLOUTSKY. (2004): «Auditory dominance and its change in the course of development», Child Development, 75, 5, pp. 1387–1401.
ROTHMAN, A. y S. NOWICKI. (2004): «A measure of the ability to identify emotion in children's tone of voice», Journal of Nonverbal Behavior, 28, pp. 67-92.
SCHERER, K. (1982): «Emotion as a process: function, origin and regulation», Social Science Information, 21, pp. 550-570.
SCHERER, K. (2003): «Vocal communication of emotion: A review of research paradigms», Speech Communication, 40, pp. 227-256.
SCHERER, K.; T. JOHNSTONE y G. KLASMEYER. (2003): «Vocal expression of emotion», en R. Davidson, K. Scherer y H. Goldsmith (eds.): Handbook of the affective sciences, Nueva York y Oxford, Oxford University Press, pp. 433-456.
SCHERER, K. (2005): «What are emotions? And how can they be measured?», Social Science Information, 44, 4, pp. 695–729.
SOTO, M. (2000): Influencia de la percepción visual del rostro del hablante en la credibilidad de su voz, tesis doctoral, Universitat Autònoma de Barcelona.
SWERTS, M. y E. KRAHMER. (2005): «Audiovisual prosody and feeling of knowing», Journal of Memory and Language, 53(1), pp. 81–94.
VIOLA, I y S. MADUREIRA. (2007): «Voice quality and speech expressiveness», Proccedings of the Paralinguistic Speech07-between models and data, Saarbrucken, DFKI Publication, pp. 65–70.
YENI-KOMSHIAN, G. (1999): «Percepción del habla», en J. Berko Gleason y N. Bernstein (eds): Psicolingüística, Madrid, Mc Graw Hill.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
All articles published online by Estudios de Fonética Experimental are licensed under Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International (CC BY-NC-ND 4.0 DEED), unless otherwise noted. Estudios de Fonética Experimental is an open access journal. Estudios de Fonética Experimental is hosted by RCUB (Revistes Científiques de la Universitat de Barcelona), powered by Open Journal Systems (OJS) software. The copyright is not transferred to the journal: authors hold the copyright and publishing rights without restrictions. The author is free to use and distribute pre and post-prints versions of his/her article. However, preprint versions are regarded as a work-in-progress version used as internal communication with the authors, and we prefer to share postprint versions.
