Skip to main content

Tilbe Goksun

Koç University, Psychology, Faculty Member

Temple University, Psychology, Graduate Student

Followers

206

Following

113

Co-authors

11

Public Views

Supervisors: Kathy Hirsh-Pasek and Anjan Chatterjee

less

University of Gothenburg

University of Ljubljana

Georgios Athanasopoulos

Consiglio Nazionale delle Ricerche (CNR)

Ivana Kruijff-Korbayova

German Research Center for Artificial Intelligence

Delft University of Technology

Matthijs Smakman

HU University of Applied Sciences

Offenburg University of Applied Sciences

InterestsView All (8)

Uploads

Papers by Tilbe Goksun

Social Robots for Early Language Learning: Current Evidence and Future Directions

by Tilbe Goksun and Junko Kanero

Child Development Perspectives, 2018

In this article, we review research on child–robot interaction (CRI) to discuss how social robots... more In this article, we review research on child–robot interaction (CRI) to discuss how social robots can be used to scaffold language learning in young children. First we provide reasons why robots can be useful for teaching first and second languages to children. Then we review studies on CRI that used robots to help children learn vocabulary and produce language. The studies vary in first and second languages and demographics of the learners (typically developing children and children with hearing and communication impairments). We conclude that, although social robots are useful for teaching language to children, evidence suggests that robots are not as effective as human teachers. However, this conclusion is not definitive because robots that tutor students in language have not been evaluated rigorously and technology is advancing rapidly. We suggest that CRI offers an opportunity for research and list possible directions for that work.

Guidelines for Designing Social Robots as Second Language Tutors

International Journal of Social Robotics, 2018

In recent years, it has been suggested that social robots have potential as tutors and educators ... more In recent years, it has been suggested that social robots have potential as tutors and educators for both children and adults. While robots have been shown to be effective in teaching knowledge and skill-based topics, we wish to explore how social robots can be used to tutor a second language to young children. As language learning relies on situated, grounded and social learning, in which interaction and repeated practice are central, social robots hold promise as educational tools for supporting second language learning. This paper surveys the developmental psychology of second language learning and suggests an agenda to study how core concepts of second language learning can be taught by a social robot. It suggests guidelines for designing robot tutors based on observations of second language learning in human–human scenarios, various technical aspects and early studies regarding the effectiveness of social robots as second language tutors.

The relationship between handedness and valence: A gesture study

Quarterly Journal of Experimental Psychology, 2018

People with different hand preferences assign positive and negative emotions to different sides o... more People with different hand preferences assign positive and negative emotions to different sides of their bodies and produce co-speech gestures with their dominant hand when the content is positive. In this study, we investigated this side preference by handedness in both gesture comprehension and production. Participants watched faceless gesture videos with negative and positive content on eye tracker and were asked to retell the stories after each video. Results indicated no difference in looking preferences regarding being right- or left-handed. Yet, an effect of emotional valence was observed. Participants spent more time looking to the right (actor’s left) when the information was positive and to the left (actor’s right) when the information was negative. Participants’ retelling of stories revealed a handedness effect only for different types of gestures (representational vs beat). Individuals used their dominant hands for beat gestures. For representational gestures, while the right-handers used their right hands more, the left-handers gestured using both hands equally. Overall, the lack of significant difference between handedness and emotional content in both comprehension and production levels suggests that body-specific mental representations may not extend to the conversational level.

A review on complementary natures of tangible user interfaces (TUIs) and early spatial learning

International Journal on Child-Computer Interaction, 2018

Spatial skills are essential for everyday tasks, and technology blends seamlessly into children’s... more Spatial skills are essential for everyday tasks, and technology blends seamlessly into children’s everyday environment. Since spatiality as a term is ubiquitous in experience this paper bridges literature in two fields: theories on early spatial learning in cognitive development and potential benefits of tangible user interfaces (TUIs) for supporting very young children’s spatial skills. Studies suggest that the period between 2 and 4 years of age is critical for training spatial skills (e.g., mental rotation), which relate to further success in STEAM (science, technology, engineering, arts, and math) disciplines. We first present a review of the empirical findings on spatial skills, early interventions, and tools (i.e., narrative and gesture input) recommended for training preschool children’s spatial skills. By situating the work within the use and benefits of manipulatives (e.g., building blocks, puzzles, shapes) combined with digital affordances in interaction design, we address the relevance of TUIs as complementary tools for spatial learning. We concentrate on the supporting properties of TUIs that enable playful learning, make storytelling more concrete, and provide embodiment effects through physicality. Through various products found in the market and literature that address the physical–digital convergence, we invite designers and researchers to consider design practices and applicable technology that build on present efforts and paradigms in this area. To contribute to this area, we conclude with a discussion of the gaps in design methods to develop technologies for children younger than 4 years old, and propose directions for future work to leverage new tools that serve very young children’s spatial learning and possible inquiries for dual payoff.

Integrity and function of gestures in aphasia

Aphasiology, 2018

Background: Gestures can provide an excellent natural alternative to verbal communication in peop... more Background: Gestures can provide an excellent natural alternative to verbal communication in people with aphasia (PWA). However, despite numerous studies focusing on gesture production in aphasia, it is still a matter of debate whether the gesture system remains intact after language impairment and how PWA use gestures to improve communication. A likely source for the contradicting results is that many studies were conducted on individual cases or in heterogeneous groups of individuals with additional cognitive deficits such as conceptual impairment and comorbid conditions such as limb apraxia.

Aims: The goal of the current study was to evaluate the integrity and function of gestures in PWA in light of cognitive theories of language–gesture relationship. Since all such theories presuppose the integrity of the conceptual system, and the absence of comorbid conditions that selectively impair gesturing (i.e., limb apraxia), our sample was selected to fulfill these assumptions.

Methods & Procedures: We examined gesture production in eight PWA with preserved auditory comprehension, no comorbidities, and various degrees of expressive deficit, as well as 11 age- and education-matched controls, while they described events in 20 normed video clips. Both speech and gesture data were coded for quantitative measures of informativeness, and gestures were grouped into several functional categories (matching, complementary, compensatory, social cueing, and facilitating lexical retrieval) based on correspondence to the accompanying speech. Using rigorous group analyses, individual-case analyses, and analyses of individual differences, we provide converging evidence for the integrity and type of function(s) served by gesturing in PWA.

Outcomes & Results: Our results indicate that the gesture system can remain functional even when language production is severely impaired. Our PWA heavily relied on iconic gestures to compensate for their language impairment, and the degree of such compensation was correlated with the extent of language impairment. In addition, we found evidence that producing iconic gestures was related to higher success rates in resolving lexical retrieval difficulties.

Conclusions: When comprehension and comorbidities are controlled for, impairment of language and gesture systems is dissociable. In PWA with good comprehension, gesturing can provide an excellent means to both compensate for the impaired language and act as a retrieval cue. Implications for cognitive theories of language–gesture relationship and therapy are discussed.

Narratives of focal brain injured individuals: A macro-level analysis

Neuropsychologia, 2017

Focal brain injury can have detrimental effects on the pragmatics of communication. This study e... more Focal brain injury can have detrimental effects on the pragmatics of communication. This study examined narrative production by unilateral brain damaged people (n=36) and healthy controls and focused on the complexity (content and coherence) and the evaluative aspect of their narratives to test the general hypothesis that the left hemisphere is biased to process microlinguistic information and the right hemisphere is biased to process macrolinguistic information. We found that people with left hemisphere damage's (LHD) narratives were less likely to maintain the overall theme of the story and produced fewer evaluative comments in their narratives. These deficits correlated with their performances on microlinguistic linguistic tasks. People with the right hemisphere damage (RHD) seemed to be preserved in expressing narrative complexity and evaluations as a group. Yet, single case analyses revealed that particular regions in the right hemisphere such as damage to the dorsolateral prefrontal cortex (DLPFC), the anterior and superior temporal gyrus, the middle temporal gyrus, and the supramarginal gyrus lead to problems in creating narratives. Our findings demonstrate that both hemispheres are necessary to produce competent narrative production. LHD people's poor production is related to their microlinguistic language problems whereas RHD people's impaired abilities can be associated with planning and working memory abilities required to relate events in a narrative.

Expression of motion events in Farsi

The current study examines how gestural representations of motion events arise from linguistic ex... more The current study examines how gestural representations of motion events arise from linguistic expressions in Farsi, as this language offers many unique characteristics; exhibiting characteristics of both Talmy's satellite-and verb-framed languages. We examined native Farsi speakers' speech and gestures in describing 20 motion events. We focused on two motion event components: path (trajectory of motion like up) and manner (how the action is performed like jumping). Analyses of syntactic packaging and clause-level correspondence between speech and gesture, as well as parallel ordering of speech and gesture sequences were, for the most part, in support of models that posit a close correspondence between speech-gesture production. However, while Farsi speakers described both path and manner in their speech, gesture was markedly impoverished for manner, suggesting constraints on the one-to-one mapping between linguistic and gestural expressions.

Phonological similarity affects production of gestures, even in the absence of overt speech

Frontiers in Psychology, 2015

Highlights Does phonological similarity affect gesture production in the absence of speech?Partic... more Highlights Does phonological similarity affect gesture production in the absence of speech?Participants produced gestures from pictures with no words presented or spoken.Same pictures and gestures but different training labels were used.Phonologically similar labels led to more errors in subsequent gestures.Thus, phonological similarity affects gesture production in the absence of speech. Are manual gestures affected by inner speech? This study tested the hypothesis that phonological form influences gesture by investigating whether phonological similarity between words that describe motion gestures creates interference for production of those gestures in the absence of overt speech. Participants learned to respond to a picture of a bottle by gesturing to open the bottle&amp;amp;amp;amp;amp;amp;amp;amp;#39;s cap, and to a picture of long hair by gesturing to twirl the hair. In one condition, the gestures were introduced with phonologically-similar labels &amp;amp;amp;amp;amp;amp;amp;amp;quot;twist&amp;amp;amp;amp;amp;amp;amp;amp;quot; and &amp;amp;amp;amp;amp;amp;amp;amp;quot;twirl&amp;amp;amp;amp;amp;amp;amp;amp;quot; (similar condition), while in the other condition, they were introduced with phonologically-dissimilar labels &amp;amp;amp;amp;amp;amp;amp;amp;quot;unscrew&amp;amp;amp;amp;amp;amp;amp;amp;quot; and &amp;amp;amp;amp;amp;amp;amp;amp;quot;twirl&amp;amp;amp;amp;amp;amp;amp;amp;quot; (dissimilar condition). During the actual experiment, labels were not produced and participants only gestured by looking at pictures. In both conditions, participants also gestured to a control pair that was used as a baseline. Participants made significantly more errors on gestures in the similar than dissimilar condition after correction for baseline differences. This finding shows the influence of phonology on gesture production in the absence of overt speech and poses new constraints on the locus of the interaction between language and gesture systems.

Spatial gestures point the way: A broader understanding of the gestural referent

by Carol Ormand, Thomas Shipley, Tilbe Goksun, and Ilyse Resnick

We investigated the use of iconic and deictic gestures during the communication of spatial inform... more We investigated the use of iconic and deictic gestures during the communication of spatial information. Expert structural geologists were asked to explain one portion of a geologic map. Spatial gestures used in each expert’s response were coded as deictic (indicating an object in the conversational space), iconic (depicting an aspect of an object or event), or both deictic and iconic (indicating an object in the conversational space by depicting an aspect of that object). Speech paired with each gesture was coded for whether or not it referred to complex spatial properties (e.g. shape and orientation of an object). Results indicated that when communicating spatial information, people occasionally use gestures that are both deictic and iconic, and that these gestures tend to occur when complex spatial information is not provided in speech. These results suggest that existing classifications of gesture are not exclusive, especially for spatial discourse.

GESTURE COMPENSATES FOR IMPAIRED LEXICAL ACCESS FOR SPATIAL LANGUAGE

by Anjan Chatterjee and Tilbe Goksun

Spontaneous gesture and spatial language: Evidence from focal brain injury

by Tilbe Goksun and K. Malykhina

Brain and Language, 2015

People often use spontaneous gestures when communicating spatial information. We investigated foc... more People often use spontaneous gestures when communicating spatial information. We investigated focal brain-injured individuals to test the hypotheses that (1) naming motion event components of manner-path (represented by verbs–prepositions in English) are impaired selectively, (2) gestures compensate for impaired naming. Patients with left or right hemisphere damage (LHD or RHD) and elderly control participants were asked to describe motion events (e.g., running across) depicted in brief videos. Damage to the left posterior middle frontal gyrus, left inferior frontal gyrus, and left anterior superior temporal gyrus (aSTG) produced impairments in naming paths of motion; lesions to the left caudate and adjacent white matter produced impairments in naming manners of motion. While the frequency of spontaneous gestures were low, lesions to the left aSTG significantly correlated with greater production of path gestures. These suggest that producing prepositions–verbs can be separately impaired and gesture production compensates for naming impairments when damage involves left aSTG.

Developing Transmedia Puzzle Play to Facilitate Spatial Skills of Preschoolers ACM Classification Keywords

by Gökçe Elif Baykal, Ilgim Veryeri Alaca, asim evren yantac, and Tilbe Goksun

This proposed project aims to develop a research-driven interactive product that facilitates spat... more This proposed project aims to develop a research-driven interactive product that facilitates spatial skills of preschoolers. Here, we present the preliminary qualitative results of a user study with the paper prototype of Fungram, merging physical tangram pieces on the screen within a narrative context. A pilot study was conducted with eight children between the ages of 25 and 48 months, who were presented tangible tangram pieces along with two sets of papers with and without narrative context. Our preliminary qualitative data indicates that narrative context helps children's coherence of abstract figures and triggers rotation of geometric tangram pieces. This study provide insights about children's user needs and action strategies within the proposed use scenario with graspable puzzle pieces related to screen interaction at this age range. We suggest that transmedia play expand opportunities for children to employ their spatial skills in different settings.

Carving the world for language: how neuroscientific research can enrich the study of first and second language learning

Developmental neuropsychology, 2014

Linguistics, psychology, and neuroscience all have rich histories in language research. Crosstalk... more Linguistics, psychology, and neuroscience all have rich histories in language research. Crosstalk among these disciplines, as realized in studies of phonology, is pivotal for understanding a fundamental challenge for first and second language learners (SLLs): learning verbs. Linguistic and behavioral research with monolinguals suggests that infants attend to foundational event components (e.g., path, manner). Language then heightens or dampens attention to these components as children map word to world in language-specific ways. Cross-linguistic differences in semantic organization also reveal sources of struggles for SLLs. We discuss how better integrating neuroscience into this literature can unlock additional mysteries of verb learning.

Forces and Motion: How Young Children Understand Causal Events

Child Development , Jan 1, 2013

How do children evaluate complex causal events? This study investigates preschoolers’ representat... more How do children evaluate complex causal events? This study investigates preschoolers’ representation of force dynamics in causal scenes, asking whether (a) children understand how single and dual forces impact an object’s movement and (b) this understanding varies across cause types (Cause, Enable, Prevent). Three-and-a half- to 5.5-year-olds (n = 60) played a board game in which they were asked to predict the endpoint of a ball being acted upon by one or two forces. Children mostly understood the interactions of forces underlying each type of cause; only 5.5-year-olds could integrate two contradictory forces. Children perceive force interactions underlying causal events, but some concepts might not be fully understood until later in childhood. This study provides a new way of thinking about causal relations.

Naming and gesturing spatial relations: evidence from focal brain-injured individuals

Neuropsychologia, 2013

Spatial language helps us to encode relations between objects and organize our thinking. Little i... more Spatial language helps us to encode relations between objects and organize our thinking. Little is known about the neural instantiations of spatial language. Using voxel-lesion symptom mapping (VLSM), we tested the hypothesis that focal brain injured patients who had damage to left frontal-parietal peri-Sylvian regions would have difficulty in naming spatial relations between objects. We also investigated the relationship between impaired verbalization of spatial relations and spontaneous gesture production. Patients with left or right hemisphere damage and elderly control participants were asked to name static (e.g., an apple on a book) and dynamic (e.g., a pen moves over a box) locative relations depicted in brief video clips. The correct use of prepositions in each task and gestures that represent the spatial relations were coded. Damage to the left posterior middle frontal gyrus, the left inferior frontal gyrus, and the left anterior superior temporal gyrus were related to impairment in naming spatial relations. Production of spatial gestures negatively correlated with naming accuracy, suggesting that gestures might help or compensate for difficulty with lexical access. Additional analyses suggested that left hemisphere patients who had damage to the left posterior middle frontal gyrus and the left inferior frontal gyrus gestured less than expected, if gestures are used to compensate for impairments in retrieving prepositions.

Carving Categories in a Continuous World: Preverbal Infants Discriminate Categorical Changes Before Distance Changes in Dynamic Events

Abstract To learn motion verbs and prepositions, children must categorize event components such t... more Abstract To learn motion verbs and prepositions, children must categorize event components such that jumping 1 meter over the puddle and jumping 3 meters over the puddle are both instances of jumping. Thus, children's categories must allow variability in coordinate properties, while preserving the relational component labeled by motion verbs and prepositions.

Infant categorization of path relations during dynamic events

Fundamental to amassing a lexicon of relational terms (ie, verbs, prepositions) is the ability to... more Fundamental to amassing a lexicon of relational terms (ie, verbs, prepositions) is the ability to abstract and categorize spatial relations such as a figure (eg, boy) moving along a path (eg, around the barn). Three studies examine how infants learn to categorize path over changes in manner, or how an action is performed (eg, running vs. crawling). Experiment 1 (n= 60) finds that 10-to 12-month-old English-learning infants categorize a figure's path.

Individual differences in mental rotation: what does gesture tell us?

Gestures are common when people convey spatial information, for example, when they give direction... more Gestures are common when people convey spatial information, for example, when they give directions or describe motion in space. Here, we examine the gestures speakers produce when they explain how they solved mental rotation problems (Shepard and Meltzer in Science 171:701–703, 1971). We asked whether speakers gesture differently while describing their problems as a function of their spatial abilities. We found that low-spatial individuals (as assessed by a standard paper-and-pencil measure) gestured more to explain their solutions than high-spatial individuals. While this finding may seem surprising, finer-grained analyses showed that low-spatial participants used gestures more often than high-spatial participants to convey “static only” information but less often than high-spatial participants to convey dynamic information. Furthermore, the groups differed in the types of gestures used to convey static information: high-spatial individuals were more likely than low-spatial individuals to use gestures that captured the internal structure of the block forms. Our gesture findings thus suggest that encoding block structure may be as important as rotating the blocks in mental spatial transformation.

Who is crossing where? Infants' discrimination of figures and grounds in events

Cognition, Jan 1, 2011

To learn relational terms such as verbs and prepositions, children must first dissect and process... more To learn relational terms such as verbs and prepositions, children must first dissect and process dynamic event components. This paper investigates the way in which 8-to 14month-old English-reared infants notice the event components, figure (i.e., the moving entity) and ground (i.e., stationary setting), in both dynamic (Experiment 1) and static representations of events (Experiment 2) for categorical ground distinctions expressed in Japanese, but not in English. We then compare both 14-and 19-month-old English-and Japanese-reared infants' processing of grounds to understand how language learning interacts with the conceptualization of these constructs (Experiment 3). Results suggest that (1) infants distinguish between figures and grounds in events; (2) they do so differently for static vs. dynamic displays; (3) early in the second year, children from diverse language environments form nonnative -perhaps universal -event categories; and (4) these event categories shift over time as children have more exposure to their native tongue.

Trading Spaces: Carving Up Events for Learning Language

Perspectives on Psychological Science, Jan 1, 2010

Relational terms (e.g., verbs and prepositions) are the cornerstone of language development, brin... more Relational terms (e.g., verbs and prepositions) are the cornerstone of language development, bringing together two distinct fields: linguistic theory and infants' event processing. To acquire relational terms such as run, walk, in, and on, infants must first perceive and conceptualize components of dynamic events such as containment-support, path-manner, source-goal, and figure-ground. Infants must then uncover how the particular language they are learning encodes these constructs. This review addresses the interaction of language learning with infants' conceptualization of these nonlinguistic spatial event components. We present the thesis that infants start with language-general nonlinguistic constructs that are gradually refined and tuned to the requirements of their native language. In effect, infants are trading spaces, maintaining their sensitivity to some relational distinctions while dampening other distinctions, depending on how their native language expresses these constructs.

Social Robots for Early Language Learning: Current Evidence and Future Directions

by Tilbe Goksun and Junko Kanero

Child Development Perspectives, 2018

In this article, we review research on child–robot interaction (CRI) to discuss how social robots... more In this article, we review research on child–robot interaction (CRI) to discuss how social robots can be used to scaffold language learning in young children. First we provide reasons why robots can be useful for teaching first and second languages to children. Then we review studies on CRI that used robots to help children learn vocabulary and produce language. The studies vary in first and second languages and demographics of the learners (typically developing children and children with hearing and communication impairments). We conclude that, although social robots are useful for teaching language to children, evidence suggests that robots are not as effective as human teachers. However, this conclusion is not definitive because robots that tutor students in language have not been evaluated rigorously and technology is advancing rapidly. We suggest that CRI offers an opportunity for research and list possible directions for that work.

Guidelines for Designing Social Robots as Second Language Tutors

International Journal of Social Robotics, 2018

In recent years, it has been suggested that social robots have potential as tutors and educators ... more In recent years, it has been suggested that social robots have potential as tutors and educators for both children and adults. While robots have been shown to be effective in teaching knowledge and skill-based topics, we wish to explore how social robots can be used to tutor a second language to young children. As language learning relies on situated, grounded and social learning, in which interaction and repeated practice are central, social robots hold promise as educational tools for supporting second language learning. This paper surveys the developmental psychology of second language learning and suggests an agenda to study how core concepts of second language learning can be taught by a social robot. It suggests guidelines for designing robot tutors based on observations of second language learning in human–human scenarios, various technical aspects and early studies regarding the effectiveness of social robots as second language tutors.

The relationship between handedness and valence: A gesture study

Quarterly Journal of Experimental Psychology, 2018

People with different hand preferences assign positive and negative emotions to different sides o... more People with different hand preferences assign positive and negative emotions to different sides of their bodies and produce co-speech gestures with their dominant hand when the content is positive. In this study, we investigated this side preference by handedness in both gesture comprehension and production. Participants watched faceless gesture videos with negative and positive content on eye tracker and were asked to retell the stories after each video. Results indicated no difference in looking preferences regarding being right- or left-handed. Yet, an effect of emotional valence was observed. Participants spent more time looking to the right (actor’s left) when the information was positive and to the left (actor’s right) when the information was negative. Participants’ retelling of stories revealed a handedness effect only for different types of gestures (representational vs beat). Individuals used their dominant hands for beat gestures. For representational gestures, while the right-handers used their right hands more, the left-handers gestured using both hands equally. Overall, the lack of significant difference between handedness and emotional content in both comprehension and production levels suggests that body-specific mental representations may not extend to the conversational level.

A review on complementary natures of tangible user interfaces (TUIs) and early spatial learning

International Journal on Child-Computer Interaction, 2018

Spatial skills are essential for everyday tasks, and technology blends seamlessly into children’s... more Spatial skills are essential for everyday tasks, and technology blends seamlessly into children’s everyday environment. Since spatiality as a term is ubiquitous in experience this paper bridges literature in two fields: theories on early spatial learning in cognitive development and potential benefits of tangible user interfaces (TUIs) for supporting very young children’s spatial skills. Studies suggest that the period between 2 and 4 years of age is critical for training spatial skills (e.g., mental rotation), which relate to further success in STEAM (science, technology, engineering, arts, and math) disciplines. We first present a review of the empirical findings on spatial skills, early interventions, and tools (i.e., narrative and gesture input) recommended for training preschool children’s spatial skills. By situating the work within the use and benefits of manipulatives (e.g., building blocks, puzzles, shapes) combined with digital affordances in interaction design, we address the relevance of TUIs as complementary tools for spatial learning. We concentrate on the supporting properties of TUIs that enable playful learning, make storytelling more concrete, and provide embodiment effects through physicality. Through various products found in the market and literature that address the physical–digital convergence, we invite designers and researchers to consider design practices and applicable technology that build on present efforts and paradigms in this area. To contribute to this area, we conclude with a discussion of the gaps in design methods to develop technologies for children younger than 4 years old, and propose directions for future work to leverage new tools that serve very young children’s spatial learning and possible inquiries for dual payoff.

Integrity and function of gestures in aphasia

Aphasiology, 2018

Background: Gestures can provide an excellent natural alternative to verbal communication in peop... more Background: Gestures can provide an excellent natural alternative to verbal communication in people with aphasia (PWA). However, despite numerous studies focusing on gesture production in aphasia, it is still a matter of debate whether the gesture system remains intact after language impairment and how PWA use gestures to improve communication. A likely source for the contradicting results is that many studies were conducted on individual cases or in heterogeneous groups of individuals with additional cognitive deficits such as conceptual impairment and comorbid conditions such as limb apraxia.

Aims: The goal of the current study was to evaluate the integrity and function of gestures in PWA in light of cognitive theories of language–gesture relationship. Since all such theories presuppose the integrity of the conceptual system, and the absence of comorbid conditions that selectively impair gesturing (i.e., limb apraxia), our sample was selected to fulfill these assumptions.

Methods & Procedures: We examined gesture production in eight PWA with preserved auditory comprehension, no comorbidities, and various degrees of expressive deficit, as well as 11 age- and education-matched controls, while they described events in 20 normed video clips. Both speech and gesture data were coded for quantitative measures of informativeness, and gestures were grouped into several functional categories (matching, complementary, compensatory, social cueing, and facilitating lexical retrieval) based on correspondence to the accompanying speech. Using rigorous group analyses, individual-case analyses, and analyses of individual differences, we provide converging evidence for the integrity and type of function(s) served by gesturing in PWA.

Outcomes & Results: Our results indicate that the gesture system can remain functional even when language production is severely impaired. Our PWA heavily relied on iconic gestures to compensate for their language impairment, and the degree of such compensation was correlated with the extent of language impairment. In addition, we found evidence that producing iconic gestures was related to higher success rates in resolving lexical retrieval difficulties.

Conclusions: When comprehension and comorbidities are controlled for, impairment of language and gesture systems is dissociable. In PWA with good comprehension, gesturing can provide an excellent means to both compensate for the impaired language and act as a retrieval cue. Implications for cognitive theories of language–gesture relationship and therapy are discussed.

Narratives of focal brain injured individuals: A macro-level analysis

Neuropsychologia, 2017

Focal brain injury can have detrimental effects on the pragmatics of communication. This study e... more Focal brain injury can have detrimental effects on the pragmatics of communication. This study examined narrative production by unilateral brain damaged people (n=36) and healthy controls and focused on the complexity (content and coherence) and the evaluative aspect of their narratives to test the general hypothesis that the left hemisphere is biased to process microlinguistic information and the right hemisphere is biased to process macrolinguistic information. We found that people with left hemisphere damage's (LHD) narratives were less likely to maintain the overall theme of the story and produced fewer evaluative comments in their narratives. These deficits correlated with their performances on microlinguistic linguistic tasks. People with the right hemisphere damage (RHD) seemed to be preserved in expressing narrative complexity and evaluations as a group. Yet, single case analyses revealed that particular regions in the right hemisphere such as damage to the dorsolateral prefrontal cortex (DLPFC), the anterior and superior temporal gyrus, the middle temporal gyrus, and the supramarginal gyrus lead to problems in creating narratives. Our findings demonstrate that both hemispheres are necessary to produce competent narrative production. LHD people's poor production is related to their microlinguistic language problems whereas RHD people's impaired abilities can be associated with planning and working memory abilities required to relate events in a narrative.

Expression of motion events in Farsi

The current study examines how gestural representations of motion events arise from linguistic ex... more The current study examines how gestural representations of motion events arise from linguistic expressions in Farsi, as this language offers many unique characteristics; exhibiting characteristics of both Talmy's satellite-and verb-framed languages. We examined native Farsi speakers' speech and gestures in describing 20 motion events. We focused on two motion event components: path (trajectory of motion like up) and manner (how the action is performed like jumping). Analyses of syntactic packaging and clause-level correspondence between speech and gesture, as well as parallel ordering of speech and gesture sequences were, for the most part, in support of models that posit a close correspondence between speech-gesture production. However, while Farsi speakers described both path and manner in their speech, gesture was markedly impoverished for manner, suggesting constraints on the one-to-one mapping between linguistic and gestural expressions.

Phonological similarity affects production of gestures, even in the absence of overt speech

Frontiers in Psychology, 2015

Highlights Does phonological similarity affect gesture production in the absence of speech?Partic... more Highlights Does phonological similarity affect gesture production in the absence of speech?Participants produced gestures from pictures with no words presented or spoken.Same pictures and gestures but different training labels were used.Phonologically similar labels led to more errors in subsequent gestures.Thus, phonological similarity affects gesture production in the absence of speech. Are manual gestures affected by inner speech? This study tested the hypothesis that phonological form influences gesture by investigating whether phonological similarity between words that describe motion gestures creates interference for production of those gestures in the absence of overt speech. Participants learned to respond to a picture of a bottle by gesturing to open the bottle&amp;amp;amp;amp;amp;amp;amp;amp;#39;s cap, and to a picture of long hair by gesturing to twirl the hair. In one condition, the gestures were introduced with phonologically-similar labels &amp;amp;amp;amp;amp;amp;amp;amp;quot;twist&amp;amp;amp;amp;amp;amp;amp;amp;quot; and &amp;amp;amp;amp;amp;amp;amp;amp;quot;twirl&amp;amp;amp;amp;amp;amp;amp;amp;quot; (similar condition), while in the other condition, they were introduced with phonologically-dissimilar labels &amp;amp;amp;amp;amp;amp;amp;amp;quot;unscrew&amp;amp;amp;amp;amp;amp;amp;amp;quot; and &amp;amp;amp;amp;amp;amp;amp;amp;quot;twirl&amp;amp;amp;amp;amp;amp;amp;amp;quot; (dissimilar condition). During the actual experiment, labels were not produced and participants only gestured by looking at pictures. In both conditions, participants also gestured to a control pair that was used as a baseline. Participants made significantly more errors on gestures in the similar than dissimilar condition after correction for baseline differences. This finding shows the influence of phonology on gesture production in the absence of overt speech and poses new constraints on the locus of the interaction between language and gesture systems.

Spatial gestures point the way: A broader understanding of the gestural referent

by Carol Ormand, Thomas Shipley, Tilbe Goksun, and Ilyse Resnick

We investigated the use of iconic and deictic gestures during the communication of spatial inform... more We investigated the use of iconic and deictic gestures during the communication of spatial information. Expert structural geologists were asked to explain one portion of a geologic map. Spatial gestures used in each expert’s response were coded as deictic (indicating an object in the conversational space), iconic (depicting an aspect of an object or event), or both deictic and iconic (indicating an object in the conversational space by depicting an aspect of that object). Speech paired with each gesture was coded for whether or not it referred to complex spatial properties (e.g. shape and orientation of an object). Results indicated that when communicating spatial information, people occasionally use gestures that are both deictic and iconic, and that these gestures tend to occur when complex spatial information is not provided in speech. These results suggest that existing classifications of gesture are not exclusive, especially for spatial discourse.

GESTURE COMPENSATES FOR IMPAIRED LEXICAL ACCESS FOR SPATIAL LANGUAGE

by Anjan Chatterjee and Tilbe Goksun

Spontaneous gesture and spatial language: Evidence from focal brain injury

by Tilbe Goksun and K. Malykhina

Brain and Language, 2015

People often use spontaneous gestures when communicating spatial information. We investigated foc... more People often use spontaneous gestures when communicating spatial information. We investigated focal brain-injured individuals to test the hypotheses that (1) naming motion event components of manner-path (represented by verbs–prepositions in English) are impaired selectively, (2) gestures compensate for impaired naming. Patients with left or right hemisphere damage (LHD or RHD) and elderly control participants were asked to describe motion events (e.g., running across) depicted in brief videos. Damage to the left posterior middle frontal gyrus, left inferior frontal gyrus, and left anterior superior temporal gyrus (aSTG) produced impairments in naming paths of motion; lesions to the left caudate and adjacent white matter produced impairments in naming manners of motion. While the frequency of spontaneous gestures were low, lesions to the left aSTG significantly correlated with greater production of path gestures. These suggest that producing prepositions–verbs can be separately impaired and gesture production compensates for naming impairments when damage involves left aSTG.

Developing Transmedia Puzzle Play to Facilitate Spatial Skills of Preschoolers ACM Classification Keywords

by Gökçe Elif Baykal, Ilgim Veryeri Alaca, asim evren yantac, and Tilbe Goksun

This proposed project aims to develop a research-driven interactive product that facilitates spat... more This proposed project aims to develop a research-driven interactive product that facilitates spatial skills of preschoolers. Here, we present the preliminary qualitative results of a user study with the paper prototype of Fungram, merging physical tangram pieces on the screen within a narrative context. A pilot study was conducted with eight children between the ages of 25 and 48 months, who were presented tangible tangram pieces along with two sets of papers with and without narrative context. Our preliminary qualitative data indicates that narrative context helps children's coherence of abstract figures and triggers rotation of geometric tangram pieces. This study provide insights about children's user needs and action strategies within the proposed use scenario with graspable puzzle pieces related to screen interaction at this age range. We suggest that transmedia play expand opportunities for children to employ their spatial skills in different settings.

Carving the world for language: how neuroscientific research can enrich the study of first and second language learning

Developmental neuropsychology, 2014

Linguistics, psychology, and neuroscience all have rich histories in language research. Crosstalk... more Linguistics, psychology, and neuroscience all have rich histories in language research. Crosstalk among these disciplines, as realized in studies of phonology, is pivotal for understanding a fundamental challenge for first and second language learners (SLLs): learning verbs. Linguistic and behavioral research with monolinguals suggests that infants attend to foundational event components (e.g., path, manner). Language then heightens or dampens attention to these components as children map word to world in language-specific ways. Cross-linguistic differences in semantic organization also reveal sources of struggles for SLLs. We discuss how better integrating neuroscience into this literature can unlock additional mysteries of verb learning.

Forces and Motion: How Young Children Understand Causal Events

Child Development , Jan 1, 2013

How do children evaluate complex causal events? This study investigates preschoolers’ representat... more How do children evaluate complex causal events? This study investigates preschoolers’ representation of force dynamics in causal scenes, asking whether (a) children understand how single and dual forces impact an object’s movement and (b) this understanding varies across cause types (Cause, Enable, Prevent). Three-and-a half- to 5.5-year-olds (n = 60) played a board game in which they were asked to predict the endpoint of a ball being acted upon by one or two forces. Children mostly understood the interactions of forces underlying each type of cause; only 5.5-year-olds could integrate two contradictory forces. Children perceive force interactions underlying causal events, but some concepts might not be fully understood until later in childhood. This study provides a new way of thinking about causal relations.

Naming and gesturing spatial relations: evidence from focal brain-injured individuals

Neuropsychologia, 2013

Spatial language helps us to encode relations between objects and organize our thinking. Little i... more Spatial language helps us to encode relations between objects and organize our thinking. Little is known about the neural instantiations of spatial language. Using voxel-lesion symptom mapping (VLSM), we tested the hypothesis that focal brain injured patients who had damage to left frontal-parietal peri-Sylvian regions would have difficulty in naming spatial relations between objects. We also investigated the relationship between impaired verbalization of spatial relations and spontaneous gesture production. Patients with left or right hemisphere damage and elderly control participants were asked to name static (e.g., an apple on a book) and dynamic (e.g., a pen moves over a box) locative relations depicted in brief video clips. The correct use of prepositions in each task and gestures that represent the spatial relations were coded. Damage to the left posterior middle frontal gyrus, the left inferior frontal gyrus, and the left anterior superior temporal gyrus were related to impairment in naming spatial relations. Production of spatial gestures negatively correlated with naming accuracy, suggesting that gestures might help or compensate for difficulty with lexical access. Additional analyses suggested that left hemisphere patients who had damage to the left posterior middle frontal gyrus and the left inferior frontal gyrus gestured less than expected, if gestures are used to compensate for impairments in retrieving prepositions.

Carving Categories in a Continuous World: Preverbal Infants Discriminate Categorical Changes Before Distance Changes in Dynamic Events

Abstract To learn motion verbs and prepositions, children must categorize event components such t... more Abstract To learn motion verbs and prepositions, children must categorize event components such that jumping 1 meter over the puddle and jumping 3 meters over the puddle are both instances of jumping. Thus, children's categories must allow variability in coordinate properties, while preserving the relational component labeled by motion verbs and prepositions.

Infant categorization of path relations during dynamic events

Fundamental to amassing a lexicon of relational terms (ie, verbs, prepositions) is the ability to... more Fundamental to amassing a lexicon of relational terms (ie, verbs, prepositions) is the ability to abstract and categorize spatial relations such as a figure (eg, boy) moving along a path (eg, around the barn). Three studies examine how infants learn to categorize path over changes in manner, or how an action is performed (eg, running vs. crawling). Experiment 1 (n= 60) finds that 10-to 12-month-old English-learning infants categorize a figure's path.

Individual differences in mental rotation: what does gesture tell us?

Gestures are common when people convey spatial information, for example, when they give direction... more Gestures are common when people convey spatial information, for example, when they give directions or describe motion in space. Here, we examine the gestures speakers produce when they explain how they solved mental rotation problems (Shepard and Meltzer in Science 171:701–703, 1971). We asked whether speakers gesture differently while describing their problems as a function of their spatial abilities. We found that low-spatial individuals (as assessed by a standard paper-and-pencil measure) gestured more to explain their solutions than high-spatial individuals. While this finding may seem surprising, finer-grained analyses showed that low-spatial participants used gestures more often than high-spatial participants to convey “static only” information but less often than high-spatial participants to convey dynamic information. Furthermore, the groups differed in the types of gestures used to convey static information: high-spatial individuals were more likely than low-spatial individuals to use gestures that captured the internal structure of the block forms. Our gesture findings thus suggest that encoding block structure may be as important as rotating the blocks in mental spatial transformation.

Who is crossing where? Infants' discrimination of figures and grounds in events

Cognition, Jan 1, 2011

To learn relational terms such as verbs and prepositions, children must first dissect and process... more To learn relational terms such as verbs and prepositions, children must first dissect and process dynamic event components. This paper investigates the way in which 8-to 14month-old English-reared infants notice the event components, figure (i.e., the moving entity) and ground (i.e., stationary setting), in both dynamic (Experiment 1) and static representations of events (Experiment 2) for categorical ground distinctions expressed in Japanese, but not in English. We then compare both 14-and 19-month-old English-and Japanese-reared infants' processing of grounds to understand how language learning interacts with the conceptualization of these constructs (Experiment 3). Results suggest that (1) infants distinguish between figures and grounds in events; (2) they do so differently for static vs. dynamic displays; (3) early in the second year, children from diverse language environments form nonnative -perhaps universal -event categories; and (4) these event categories shift over time as children have more exposure to their native tongue.

Trading Spaces: Carving Up Events for Learning Language

Perspectives on Psychological Science, Jan 1, 2010

Relational terms (e.g., verbs and prepositions) are the cornerstone of language development, brin... more Relational terms (e.g., verbs and prepositions) are the cornerstone of language development, bringing together two distinct fields: linguistic theory and infants' event processing. To acquire relational terms such as run, walk, in, and on, infants must first perceive and conceptualize components of dynamic events such as containment-support, path-manner, source-goal, and figure-ground. Infants must then uncover how the particular language they are learning encodes these constructs. This review addresses the interaction of language learning with infants' conceptualization of these nonlinguistic spatial event components. We present the thesis that infants start with language-general nonlinguistic constructs that are gradually refined and tuned to the requirements of their native language. In effect, infants are trading spaces, maintaining their sensitivity to some relational distinctions while dampening other distinctions, depending on how their native language expresses these constructs.

Temporal Expressions in Speech and Gesture

by Tilbe Goksun and Ahmet Borutecene

People use spatial metaphors to talk about temporal concepts. They also gesture frequently during... more People use spatial metaphors to talk about temporal concepts. They also gesture frequently during speech. The characteristics of these gestures give information regarding the mental timelines people form to experience time. The present study investigates the expression of temporal concepts on a natural setting with Turkish speakers. We found that Turkish speakers used more metaphoric temporal phrases (e.g., short period, time flies quickly) than words referring to time without spatial content (e.g., today, nowadays) in a session where they talked about people's fortune. Spontaneous gestures were mainly classified as metaphoric and beat gestures and were mostly produced on the sagittal axis, which contradicts with the previous findings. Yet, we also found that people used vertical axis to represent current and future events. These findings suggest that lateral axis may not always be the most common direction for co-speech temporal gesture use, and the pragmatic constraints of the environment may influence the spatial conceptualization of time.

Disfluency production in speech and gesture

The cognitive architecture and function of co-speech gesture has been the subject of a large body... more The cognitive architecture and function of co-speech gesture has been the subject of a large body of research. We investigate two main questions in this field, namely, whether language and gesture are the same or two interrelated systems, and whether gestures help resolve speech problems, by examining the relationship between gesture and disfluency in neurotypical speakers. Our results support the view of separate, but interrelated systems by showing that speech problems do not necessarily cause gesture problems, and on many occasions, gestures signal an upcoming speech problem even before it surfaces in overt speech. We also show that while gestures are more common on fluent trials, speakers use both iconic and beat gestures on disfluent trials to facilitate communication, although the two gesture types support communication in different ways.

Motion event expressions in language and gesture: Evidence from Persian

by NILOOFAR AKHAVAN and Tilbe Goksun

How do people conceptualize motion events and talk about them? The current study examines how ges... more How do people conceptualize motion events and talk about them? The current study examines how gestural representations of motion events arise from linguistic expressions in Persian, which has characteristics of both Talmy's satellite-and verb-framed languages. We examined native Persian speakers' speech and gestures in describing 20 motion events. We focused on two motion event components: path (trajectory of motion like up) and manner (how the action is performed like jumping). Results indicated that when expressing motion, Persian speakers produced path in both speech and gesture, whereas manner was conveyed only through speech (mostly as adverbs). Additionally, dynamic gestures tended to occur in the same order they were uttered. The difference between path and manner findings asks for further research to examine language-gesture interaction in detail among different languages. Results also suggest refinement in gesture theories that argue for one-to-one correspondence between speech and gesture.

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.

Alternative Proxies:

Alternative Proxy