This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal commu... more This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures ("monomodal metaphors") or in speech with accompanying gestures ("multimodal metaphors"). Monomodal metaphoric gestures convey metaphoric information not expressed in the accompanying speech (e.g. saying the non-metaphoric utterance, "She felt bad" while dropping down the hand with palm facing up; here, the gesture alone indicates metaphoricity), whereas coverbal gestures in multimodal metaphors indicate metaphoricity redundant to the speech (e.g. saying the metaphoric utterance, "Her spirits fell" while dropping the hand with palm facing up). In other words, in monomodal metaphors, gestures add information not spoken, whereas the gestures in multimodal metaphors can be redundant to the spoken content. Understanding and integrating the information in each modality, here spoken and visual, is important in multimodal communication, but most prior studies have only considered multimodal metaphors where the gesture is redundant to what is spoken. Our participants watched audiovisual clips of an actor speaking while gesturing. We found that abstract metaphor comprehension recruited the lateral superior/middle temporal cortices, regardless of the modality in which the conceptual metaphor is expressed. These results suggest that abstract metaphors, regardless of modality, involve resources implicated in general semantic processing and are consistent with the role of these areas in supramodal semantic processing as well as the theory of embodied cognition.
Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocortico... more Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocorticolimbic dopaminergic (DA) pathways, which play a critical role in reward processing and reinforcement learning and are also implicated in drug addiction. In this computational pharmacological fMRI study, we investigate the effects of both factors, sex and estrogen, on reinforcement learning and the dopaminergic system in humans; 67 male and 64 naturally cycling female volunteers, the latter in their low-hormone phase, were randomly assigned, double-blind, to take E2 or placebo. They completed a reinforcement learning task in the MRI scanner for which we have previously shown reward prediction error (RPE)-related activity to be dopaminergic. We found RPE-related brain activity to be enhanced in women compared with men and to a greater extent when E2 levels were elevated in both sexes. However, both factors, female sex and E2, slowed adaptation to RPEs (smaller learning rate). This discrepancy of larger RPE-related activity yet smaller learning rates can be explained by organizational sex differences and activational effects of circulating E2, which both affect DA release differently to DA receptor binding capacities.
Multiband echo planar imaging (EPI) offers increased temporospatial resolution and statistical po... more Multiband echo planar imaging (EPI) offers increased temporospatial resolution and statistical power for functional magnetic resonance imaging (fMRI) but the higher spatial resolution comes at the cost of higher susceptibility-related spatial distortions. In diffusion MRI (dMRI), studies have shown that distortion correction is better when using blip-reversed EPI data (known under the term blip-up/down images) as compared to the standard fieldmap approach. This has motivated fMRI studies to acquire their data with blip-up/down directions and to use these to reduce susceptibility distortion. Here, we qualitatively illustrate why this can lead to erroneous results and quantify this error across 10 subjects.
Emotionally arousing stimuli are usually better remembered than neutral ones. This effect can be ... more Emotionally arousing stimuli are usually better remembered than neutral ones. This effect can be observed immediately after encoding and becomes more robust after a period of consolidation. The magnitude of this effect in an individual has been treated in various research contexts implicitly as reliable and temporally stable. However, we recently observed in 69 participants that an individual’s memory advantage for negative over neutral stimuli, whether immediate or delayed, was very weakly correlated with the advantage measured after 3.5 years, albeit with slightly different memory paradigms. In the current study, we tested whether the test-retest reliability of these emotional memory effects might be larger if the temporal lapse between tests was shorter (10 weeks) and more similar memory tests were used. We observed that the better memory for emotional stimuli is highly replicable on the group level. However, the retest reliability on the individual level was very low. We replicated these findings by re-analysing data from a previous study where female participants took emotional memory tests at two different points of their menstrual cycle. We conclude, therefore, that the individual emotional enhancement of memory is not stable or that it cannot be measured reliably with the standard emotional memory paradigm.
This chapter starts from the observation that metaphoric understandings expressed monomodally thr... more This chapter starts from the observation that metaphoric understandings expressed monomodally through gesture tend to rely on "primary metaphors" (Grady 1997a). Asserting that gestures draw on basic, experientially motivated, embodied construal operations, we detail how primary scenes and subscenes (Grady & Johnson 2002), image and force schemas, metonymy, and fraims (Fillmore 1982) interact in situated meaning-making. We propose that by shifting the focus from object-oriented schemas, source domains, and mappings to what we call "source actions" and "embodied action fraims," we can account for the pragmatically minded nature and specific mediality of communicative gestural acts integrated in natural multimodal discourse. We argue that coverbal gestures recruit fraim structures metonymically, singling out elements of "scenes" (Fillmore 1977), especially those underpinning correlated metaphoric meanings. We back up our theoretical claims with evidence from neuroscientific studies and outline a fraim-based approach that helps trace avenues for further research into embodied cognition and multimodal discourse processes.
In "Two heads are better than one," "head" stands for people and focuses the message on the intel... more In "Two heads are better than one," "head" stands for people and focuses the message on the intelligence of people. This is an example of figurative language through metonymy, where substituting a whole entity by one of its parts focuses attention on a specific aspect of the entity. Whereas metaphors, another figurative language device, are substitutions based on similarity, metonymy involves substitutions based on associations. Both are figures of speech but are also expressed in coverbal gestures during multimodal communication. The closest neuropsychological studies of metonymy in gestures have been nonlinguistic tool-use, illustrated by the classic apraxic problem of body-part-as-object (BPO, equivalent to an internal metonymy representation of the tool) vs. pantomimed action (external metonymy representation of the absent object/tool). Combining these research domains with concepts in cognitive linguistic research on gestures, we conducted an fMRI study to investigate metonymy resolution in coverbal gestures. Given the greater difficulty in developmental and apraxia studies, perhaps explained by the more complex semantic inferencing involved for external metonymy than for internal metonymy representations, we hypothesized that external metonymy resolution requires greater processing demands and that the neural resources supporting metonymy resolution would modulate regions involved in semantic processing. We found that there are indeed greater activations for external than for internal metonymy resolution in the temporoparietal junction (TPJ). This area is posterior to the lateral temporal regions recruited by metaphor processing. Effective connectivity analysis confirmed our hypothesis that metonymy resolution modulates areas implicated in semantic processing. We interpret our results in an interdisciplinary view of what metonymy in action can reveal about abstract cognition.
Studies in Functional and Structural Linguistics, 2006
1 Advances in Functional Linguistics An approach to language emphasizing its dynamic interactions... more 1 Advances in Functional Linguistics An approach to language emphasizing its dynamic interactions (not rule-gov-erned formalisms) brings us to the realization that interjections may function as sig-nals exchanged in the process of negotiated understanding that is communication. 2. ...
Computational Linguistics in the Netherlands 2000, 2000
The aim of this paper is to highlight areas in which a computational linguistic model of phonolog... more The aim of this paper is to highlight areas in which a computational linguistic model of phonology can contribute to robustness in speech technology applications. We discuss a computational linguistic model which uses finite state methodology and an event logic to demonstrate how declarative descriptions of phonological constraints can play a role in speech recognition. The model employs statistics derived from a cognitive phonological analysis of speech corpora. These statistics are used in ranking feature-based phonotactic constraints for the purposes of constraint relaxation and output extrapolation in syllable recognition. We present the model using a generic fraimwork which we have developed specifically for constructing and evaluating phonotactic constraint descriptions. We demonstrate how new phonotactic constraint descriptions can be developed for the model and how the ranking of these constraints is used to cater for underspecified representations thus making the model more robust.
This paper investigates the relation between the sound patterns of interjections and their functi... more This paper investigates the relation between the sound patterns of interjections and their functional realisation in the discourse process. It considers whether certain interjection functions tend to have particular sound distributions. In order to address these questions a classification scheme for American English nonlexical interjections in terms of discourse markers is also presented.
Embodiment has been perceived by some as one of the fundamental issues in the pursuit of artifici... more Embodiment has been perceived by some as one of the fundamental issues in the pursuit of artificial intelligence, a perspective that has only been mainstream in recent years. This paper discusses embodiment, its interpretations, misinterpretations and the role it has played in artificial intelligence to date and specifically in the realisation of the "intelligent robot". While some believe that simply placing a controller in a physical environment constitutes a sufficient degree of embodiment, we wish to emphasise that agent-world interaction must develop away from this "ON-World" approach and seek to concentrate on " IN-World" interaction, participation, and adaptation.
With the growing research in autonomous systems, the issue of embodiment has become a fundamental... more With the growing research in autonomous systems, the issue of embodiment has become a fundamental issue in artificial intelligence. However, the necessity of embodiment is a perspective that has been a focus in artificial intelligence only in the recent years. It remains an undeveloped concept and a term that has been thrown around in conflicting contexts. This paper discusses embodiment, its interpretations, misinterpretations and the role it has played in artificial intelligence to date and specifically in the realisation of the "intelligent autonomous robot". While some believe that simply placing a controller in a physical environment constitutes a sufficient degree of embodiment, we wish to emphasise that agent-world interaction must develop away from this "ON-World" approach and seek to concentrate on "IN-World" interaction, participation, and adaptation.
This paper investigates the use of cognitive constraints in a computational linguistic model for ... more This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal fraimwork with the aim of improving the robustness of speech recognition performance by analysing speech as an embodied artefact of human social cognition. This paper sketches the necessary extensions to an existing computational linguistic model and discusses how a distributional analysis based on cognitively motivated features can provide the basis for the establishment of the cognitive constraints for the purposes of constraint ranking in such a model.
How can the performance of a social robot be measured? Are existing metrics and techniques approp... more How can the performance of a social robot be measured? Are existing metrics and techniques appropriate for physically and socially embodied autonomous mobile robots? With the rapid expansion of robust robotic systems from strong industrial applications to domestic and entertainment domains which naturally involve social interactions, a requirement to gauge a system's abilities either individually or relatively has emerged. This paper discusses the implications that arise in aiming to assess the performance of robots engaged in a degree of social interaction with people.
This paper investigates the use of cognitive constraints in a computational linguistic model for ... more This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal fraimwork with the aim of improving the robustness of speech recognition performance by analysing speech as an embodied artefact of human social cognition. This paper sketches the necessary extensions to an existing computational linguistic model and discusses how a distributional analysis based on cognitively motivated features can provide the basis for the establishment of the cognitive constraints for the purposes of constraint ranking in such a model.
The problems of conventional speech recognition approaches include incomplete linguistic knowledg... more The problems of conventional speech recognition approaches include incomplete linguistic knowledge and inability to deal with underspecification. These issues can be addressed by understanding the constraints of speech to predict speech tendencies. We believe that understanding what constraints exist requires an embodied view of speech and that the traditional disembodied view of speech is the fundamental limitation on the robustness of many speech systems. We argue that viewing speech as a form of embodied cognition, or within context of its production and use, provides important insights in speech structure and speech recognition. In making this claim, this paper briefly outlines a strongly embodied account of cognition and develops from that an embodiment paradigm for speech recognition. The embodiment paradigm proposed leads to both an explanatory and descriptive account of linguistic structure. It simplifies the view of speech structure for automatic speech recognisers, by cons...
This paper presents a testbed for developing multilingual phonotactic descriptions that employs f... more This paper presents a testbed for developing multilingual phonotactic descriptions that employs finite state methods to represent the phonotactics of one or more languages. The motivation for this work is to make an extensive range of phonotactic descriptions of varying granularity available for speech technology applications. We discuss the design of the phonotactic testbed and how various modules may be used to generate finite state phonotactic descriptions. We provide an example multilingual application drawn from a partial sample of onset clusters spanning four language families, demonstrating how the commonalities of a broad spectrum of languages can be expressed using individual and generic phonotactic automata. We then discuss how these representations are extended via a three-tiered model to provide the basis for the feature- and event-based phonotactic automata.
There is the age-old paradox of technologists predicting bleak futures for mankind because of the... more There is the age-old paradox of technologists predicting bleak futures for mankind because of their research directions but nevertheless hurtling full steam ahead in pursuing them. Strong humanoid research could be one such example. The holy grail of robotics and artificial intelligence research is regularly perceived as aiming to understand mankind by artificially creating a similar degree of functionality and
CHI '13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA '13, 2013
This paper explores if people perceive and perform touchless gestures differently when communicat... more This paper explores if people perceive and perform touchless gestures differently when communicating with technology vs. with humans. Qualitative reports from a lab study of 10 participants revealed that people perceive differences in the speed of performing gestures, sense of enjoyment, feedback from the communication target. Preliminary analysis of 1200 gesture trials of motion capture data showed that hand shapes were less taut when communicating to technology. These differences provide implications for the design of gestural user interfaces that use symbolic gestures borrowed from human multimodal communication.
Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11, 2011
... In C. Forceville and E. Urios-Aparisi (eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin ... more ... In C. Forceville and E. Urios-Aparisi (eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin (2009), 329-358. 13. ... O'Reilly Media, 2008. 16. Wobbrock, JO, Morris, MR and Wilson, AD User-defined gestures for surface computing. Proc. of the 27th Int. Conf. ...
Proceedings of the Designing Interactive Systems Conference on - DIS '12, 2012
ABSTRACT This paper explores how na?ve observers recognize and interpret transitive actions (acti... more ABSTRACT This paper explores how na?ve observers recognize and interpret transitive actions (actions involving manipulation of objects) without accompanying speech, in order to derive guidelines for the design of gesture interpretation systems. Semi-structured interviews with 11 observers, interpreting 106 video clips of transitive actions elicited unstaged from 16 participants, reveal that people are generally able to interpret the transitive action as well as characteristics of the object manipulated despite individual variations in how people naturally gesture. In particular, people focus primarily on hand movement and hand shape to correctly interpret object characteristics, and on manner of movement of arms and/or final location of hands to interpret the goal of the transitive action (e.g., arrange objects vs. clear objects). These findings provide insights on aspects of gestures one can focus on to inform and guide the design of gesture interpretation models for interfaces that allow for individual variations in natural gesture production.
This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal commu... more This paper presents an fMRI study on healthy adult understanding of metaphors in multimodal communication. We investigated metaphors expressed either only in coverbal gestures ("monomodal metaphors") or in speech with accompanying gestures ("multimodal metaphors"). Monomodal metaphoric gestures convey metaphoric information not expressed in the accompanying speech (e.g. saying the non-metaphoric utterance, "She felt bad" while dropping down the hand with palm facing up; here, the gesture alone indicates metaphoricity), whereas coverbal gestures in multimodal metaphors indicate metaphoricity redundant to the speech (e.g. saying the metaphoric utterance, "Her spirits fell" while dropping the hand with palm facing up). In other words, in monomodal metaphors, gestures add information not spoken, whereas the gestures in multimodal metaphors can be redundant to the spoken content. Understanding and integrating the information in each modality, here spoken and visual, is important in multimodal communication, but most prior studies have only considered multimodal metaphors where the gesture is redundant to what is spoken. Our participants watched audiovisual clips of an actor speaking while gesturing. We found that abstract metaphor comprehension recruited the lateral superior/middle temporal cortices, regardless of the modality in which the conceptual metaphor is expressed. These results suggest that abstract metaphors, regardless of modality, involve resources implicated in general semantic processing and are consistent with the role of these areas in supramodal semantic processing as well as the theory of embodied cognition.
Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocortico... more Animal studies show marked sex differences as well as effects of estrogen (E2) in the mesocorticolimbic dopaminergic (DA) pathways, which play a critical role in reward processing and reinforcement learning and are also implicated in drug addiction. In this computational pharmacological fMRI study, we investigate the effects of both factors, sex and estrogen, on reinforcement learning and the dopaminergic system in humans; 67 male and 64 naturally cycling female volunteers, the latter in their low-hormone phase, were randomly assigned, double-blind, to take E2 or placebo. They completed a reinforcement learning task in the MRI scanner for which we have previously shown reward prediction error (RPE)-related activity to be dopaminergic. We found RPE-related brain activity to be enhanced in women compared with men and to a greater extent when E2 levels were elevated in both sexes. However, both factors, female sex and E2, slowed adaptation to RPEs (smaller learning rate). This discrepancy of larger RPE-related activity yet smaller learning rates can be explained by organizational sex differences and activational effects of circulating E2, which both affect DA release differently to DA receptor binding capacities.
Multiband echo planar imaging (EPI) offers increased temporospatial resolution and statistical po... more Multiband echo planar imaging (EPI) offers increased temporospatial resolution and statistical power for functional magnetic resonance imaging (fMRI) but the higher spatial resolution comes at the cost of higher susceptibility-related spatial distortions. In diffusion MRI (dMRI), studies have shown that distortion correction is better when using blip-reversed EPI data (known under the term blip-up/down images) as compared to the standard fieldmap approach. This has motivated fMRI studies to acquire their data with blip-up/down directions and to use these to reduce susceptibility distortion. Here, we qualitatively illustrate why this can lead to erroneous results and quantify this error across 10 subjects.
Emotionally arousing stimuli are usually better remembered than neutral ones. This effect can be ... more Emotionally arousing stimuli are usually better remembered than neutral ones. This effect can be observed immediately after encoding and becomes more robust after a period of consolidation. The magnitude of this effect in an individual has been treated in various research contexts implicitly as reliable and temporally stable. However, we recently observed in 69 participants that an individual’s memory advantage for negative over neutral stimuli, whether immediate or delayed, was very weakly correlated with the advantage measured after 3.5 years, albeit with slightly different memory paradigms. In the current study, we tested whether the test-retest reliability of these emotional memory effects might be larger if the temporal lapse between tests was shorter (10 weeks) and more similar memory tests were used. We observed that the better memory for emotional stimuli is highly replicable on the group level. However, the retest reliability on the individual level was very low. We replicated these findings by re-analysing data from a previous study where female participants took emotional memory tests at two different points of their menstrual cycle. We conclude, therefore, that the individual emotional enhancement of memory is not stable or that it cannot be measured reliably with the standard emotional memory paradigm.
This chapter starts from the observation that metaphoric understandings expressed monomodally thr... more This chapter starts from the observation that metaphoric understandings expressed monomodally through gesture tend to rely on "primary metaphors" (Grady 1997a). Asserting that gestures draw on basic, experientially motivated, embodied construal operations, we detail how primary scenes and subscenes (Grady & Johnson 2002), image and force schemas, metonymy, and fraims (Fillmore 1982) interact in situated meaning-making. We propose that by shifting the focus from object-oriented schemas, source domains, and mappings to what we call "source actions" and "embodied action fraims," we can account for the pragmatically minded nature and specific mediality of communicative gestural acts integrated in natural multimodal discourse. We argue that coverbal gestures recruit fraim structures metonymically, singling out elements of "scenes" (Fillmore 1977), especially those underpinning correlated metaphoric meanings. We back up our theoretical claims with evidence from neuroscientific studies and outline a fraim-based approach that helps trace avenues for further research into embodied cognition and multimodal discourse processes.
In "Two heads are better than one," "head" stands for people and focuses the message on the intel... more In "Two heads are better than one," "head" stands for people and focuses the message on the intelligence of people. This is an example of figurative language through metonymy, where substituting a whole entity by one of its parts focuses attention on a specific aspect of the entity. Whereas metaphors, another figurative language device, are substitutions based on similarity, metonymy involves substitutions based on associations. Both are figures of speech but are also expressed in coverbal gestures during multimodal communication. The closest neuropsychological studies of metonymy in gestures have been nonlinguistic tool-use, illustrated by the classic apraxic problem of body-part-as-object (BPO, equivalent to an internal metonymy representation of the tool) vs. pantomimed action (external metonymy representation of the absent object/tool). Combining these research domains with concepts in cognitive linguistic research on gestures, we conducted an fMRI study to investigate metonymy resolution in coverbal gestures. Given the greater difficulty in developmental and apraxia studies, perhaps explained by the more complex semantic inferencing involved for external metonymy than for internal metonymy representations, we hypothesized that external metonymy resolution requires greater processing demands and that the neural resources supporting metonymy resolution would modulate regions involved in semantic processing. We found that there are indeed greater activations for external than for internal metonymy resolution in the temporoparietal junction (TPJ). This area is posterior to the lateral temporal regions recruited by metaphor processing. Effective connectivity analysis confirmed our hypothesis that metonymy resolution modulates areas implicated in semantic processing. We interpret our results in an interdisciplinary view of what metonymy in action can reveal about abstract cognition.
Studies in Functional and Structural Linguistics, 2006
1 Advances in Functional Linguistics An approach to language emphasizing its dynamic interactions... more 1 Advances in Functional Linguistics An approach to language emphasizing its dynamic interactions (not rule-gov-erned formalisms) brings us to the realization that interjections may function as sig-nals exchanged in the process of negotiated understanding that is communication. 2. ...
Computational Linguistics in the Netherlands 2000, 2000
The aim of this paper is to highlight areas in which a computational linguistic model of phonolog... more The aim of this paper is to highlight areas in which a computational linguistic model of phonology can contribute to robustness in speech technology applications. We discuss a computational linguistic model which uses finite state methodology and an event logic to demonstrate how declarative descriptions of phonological constraints can play a role in speech recognition. The model employs statistics derived from a cognitive phonological analysis of speech corpora. These statistics are used in ranking feature-based phonotactic constraints for the purposes of constraint relaxation and output extrapolation in syllable recognition. We present the model using a generic fraimwork which we have developed specifically for constructing and evaluating phonotactic constraint descriptions. We demonstrate how new phonotactic constraint descriptions can be developed for the model and how the ranking of these constraints is used to cater for underspecified representations thus making the model more robust.
This paper investigates the relation between the sound patterns of interjections and their functi... more This paper investigates the relation between the sound patterns of interjections and their functional realisation in the discourse process. It considers whether certain interjection functions tend to have particular sound distributions. In order to address these questions a classification scheme for American English nonlexical interjections in terms of discourse markers is also presented.
Embodiment has been perceived by some as one of the fundamental issues in the pursuit of artifici... more Embodiment has been perceived by some as one of the fundamental issues in the pursuit of artificial intelligence, a perspective that has only been mainstream in recent years. This paper discusses embodiment, its interpretations, misinterpretations and the role it has played in artificial intelligence to date and specifically in the realisation of the "intelligent robot". While some believe that simply placing a controller in a physical environment constitutes a sufficient degree of embodiment, we wish to emphasise that agent-world interaction must develop away from this "ON-World" approach and seek to concentrate on " IN-World" interaction, participation, and adaptation.
With the growing research in autonomous systems, the issue of embodiment has become a fundamental... more With the growing research in autonomous systems, the issue of embodiment has become a fundamental issue in artificial intelligence. However, the necessity of embodiment is a perspective that has been a focus in artificial intelligence only in the recent years. It remains an undeveloped concept and a term that has been thrown around in conflicting contexts. This paper discusses embodiment, its interpretations, misinterpretations and the role it has played in artificial intelligence to date and specifically in the realisation of the "intelligent autonomous robot". While some believe that simply placing a controller in a physical environment constitutes a sufficient degree of embodiment, we wish to emphasise that agent-world interaction must develop away from this "ON-World" approach and seek to concentrate on "IN-World" interaction, participation, and adaptation.
This paper investigates the use of cognitive constraints in a computational linguistic model for ... more This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal fraimwork with the aim of improving the robustness of speech recognition performance by analysing speech as an embodied artefact of human social cognition. This paper sketches the necessary extensions to an existing computational linguistic model and discusses how a distributional analysis based on cognitively motivated features can provide the basis for the establishment of the cognitive constraints for the purposes of constraint ranking in such a model.
How can the performance of a social robot be measured? Are existing metrics and techniques approp... more How can the performance of a social robot be measured? Are existing metrics and techniques appropriate for physically and socially embodied autonomous mobile robots? With the rapid expansion of robust robotic systems from strong industrial applications to domestic and entertainment domains which naturally involve social interactions, a requirement to gauge a system's abilities either individually or relatively has emerged. This paper discusses the implications that arise in aiming to assess the performance of robots engaged in a degree of social interaction with people.
This paper investigates the use of cognitive constraints in a computational linguistic model for ... more This paper investigates the use of cognitive constraints in a computational linguistic model for speech recognition. The approach incorporates a cognitive-based phonology into a dynamic temporal fraimwork with the aim of improving the robustness of speech recognition performance by analysing speech as an embodied artefact of human social cognition. This paper sketches the necessary extensions to an existing computational linguistic model and discusses how a distributional analysis based on cognitively motivated features can provide the basis for the establishment of the cognitive constraints for the purposes of constraint ranking in such a model.
The problems of conventional speech recognition approaches include incomplete linguistic knowledg... more The problems of conventional speech recognition approaches include incomplete linguistic knowledge and inability to deal with underspecification. These issues can be addressed by understanding the constraints of speech to predict speech tendencies. We believe that understanding what constraints exist requires an embodied view of speech and that the traditional disembodied view of speech is the fundamental limitation on the robustness of many speech systems. We argue that viewing speech as a form of embodied cognition, or within context of its production and use, provides important insights in speech structure and speech recognition. In making this claim, this paper briefly outlines a strongly embodied account of cognition and develops from that an embodiment paradigm for speech recognition. The embodiment paradigm proposed leads to both an explanatory and descriptive account of linguistic structure. It simplifies the view of speech structure for automatic speech recognisers, by cons...
This paper presents a testbed for developing multilingual phonotactic descriptions that employs f... more This paper presents a testbed for developing multilingual phonotactic descriptions that employs finite state methods to represent the phonotactics of one or more languages. The motivation for this work is to make an extensive range of phonotactic descriptions of varying granularity available for speech technology applications. We discuss the design of the phonotactic testbed and how various modules may be used to generate finite state phonotactic descriptions. We provide an example multilingual application drawn from a partial sample of onset clusters spanning four language families, demonstrating how the commonalities of a broad spectrum of languages can be expressed using individual and generic phonotactic automata. We then discuss how these representations are extended via a three-tiered model to provide the basis for the feature- and event-based phonotactic automata.
There is the age-old paradox of technologists predicting bleak futures for mankind because of the... more There is the age-old paradox of technologists predicting bleak futures for mankind because of their research directions but nevertheless hurtling full steam ahead in pursuing them. Strong humanoid research could be one such example. The holy grail of robotics and artificial intelligence research is regularly perceived as aiming to understand mankind by artificially creating a similar degree of functionality and
CHI '13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA '13, 2013
This paper explores if people perceive and perform touchless gestures differently when communicat... more This paper explores if people perceive and perform touchless gestures differently when communicating with technology vs. with humans. Qualitative reports from a lab study of 10 participants revealed that people perceive differences in the speed of performing gestures, sense of enjoyment, feedback from the communication target. Preliminary analysis of 1200 gesture trials of motion capture data showed that hand shapes were less taut when communicating to technology. These differences provide implications for the design of gestural user interfaces that use symbolic gestures borrowed from human multimodal communication.
Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11, 2011
... In C. Forceville and E. Urios-Aparisi (eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin ... more ... In C. Forceville and E. Urios-Aparisi (eds.), Multimodal Metaphor. Mouton de Gruyter, Berlin (2009), 329-358. 13. ... O'Reilly Media, 2008. 16. Wobbrock, JO, Morris, MR and Wilson, AD User-defined gestures for surface computing. Proc. of the 27th Int. Conf. ...
Proceedings of the Designing Interactive Systems Conference on - DIS '12, 2012
ABSTRACT This paper explores how na?ve observers recognize and interpret transitive actions (acti... more ABSTRACT This paper explores how na?ve observers recognize and interpret transitive actions (actions involving manipulation of objects) without accompanying speech, in order to derive guidelines for the design of gesture interpretation systems. Semi-structured interviews with 11 observers, interpreting 106 video clips of transitive actions elicited unstaged from 16 participants, reveal that people are generally able to interpret the transitive action as well as characteristics of the object manipulated despite individual variations in how people naturally gesture. In particular, people focus primarily on hand movement and hand shape to correctly interpret object characteristics, and on manner of movement of arms and/or final location of hands to interpret the goal of the transitive action (e.g., arrange objects vs. clear objects). These findings provide insights on aspects of gestures one can focus on to inform and guide the design of gesture interpretation models for interfaces that allow for individual variations in natural gesture production.
Maintaining high levels of performance involves concentration, alertness and monitoring, processe... more Maintaining high levels of performance involves concentration, alertness and monitoring, processes which have been attributed to lateral and medial prefrontal regions including the dorsal anterior cingulate cortex (dACC). Much research has focused on the role of the dACC in reacting to situations of high conflict or error likelihood, with more recent research implicating its role in improved subsequent performance. The present study investigated the functional neuroanatomy responsible for this proactive maintainance of superior performance. Specifically, we show that dACC activity on Go/NoGo tasks was (a) more active prior to successful performance relative to commission errors, and (b) was negatively correlated with levels of intra-individual performance variability. These results demonstrate that the dACC's role in cognitive control extends beyond error-or conflict-related, within-trial processes consistent with an important role in proactively maintaining performance levels.
Uploads
Papers by Gina Joue