As part of the Language Production Lab at UCSD, I was in charge of an independent research project exploring how semantic cues can influence speakers' sentence production under uncertain contexts—an extremely understudied topic. To account for limited groundwork, I designed a new image-based experimental paradigm that manipulates the semantic relatedness between a cue and a target object in a image. Participants will be asked to describe the images, and their pattern of production will provide insight on how predictions can influence fluency and reflect a mechanism of incremental planning.
I was deeply involved with every step of this process, from ideation to running participants, the latter being where the project is at presently. Since language production happens frequently in dynamic contexts where a speaker's environment may be changing unpredictably, I believe the results of our study hold promise in helping psycholinguists develop more realistic natural language models, communication strategies, and clinical applications.
For the typical person, language is a highly learned skill. In psychology, it's often even described as an overlearned skill. Yet, very little is actually known about the psychology of language production. I'm incredibly lucky to contribute to this understudied field.
As part of the Language Production Lab at UCSD, headed by principle investigator Victor Ferreira, I assist my graduate advisor, Thomas Morton, with his research as well as run my own independent project, which I will be introducing below.
My role: I had an active and leading role in every step, and collaborate with my graduate advisor on steps that require niche technical skills such as coding the experimental software and data analysis. After graduating and leaving the lab, I am no longer an active researcher, but my lab is continuing in my work.
I've presented (or plan to present) this project at multiple conferences.
In language production, speakers do not always know exactly what they will say before they begin speaking (Brown-Schmidt & Konopka, 2015). This uncertainty over their intended message can be resolved during production, however, as a function of incremental planning (Ferreira & Swets, 2002). In such situations, speakers utilize top-down information and probability estimations in their utterance plan in order to begin speaking sooner (Gussow, 2023). Gussow, using semantic and visual cues, found evidence that probable message components were prioritized in planning by speakers over improbable ones.
This study aims to understand how top-down semantic cues may play a part in speakers’ utterance planning when faced with message uncertainty. We seek to investigate this phenomena by measuring word onset times during the course of sentence production. Previous work shows that these measures can provide insight into the processes of uncertain production. For instance, Arnold et. al (2003) shows that newly presented discourse can manifest in speech disfluencies, including pauses and increased speech latencies. We often see these disfluencies specifically when a speaker adjusts their utterance plan in response to new information (Brown-Schmidt & Tanenhaus, 2006). Few studies, however, have investigated how speech disfluencies may reflect sentence planning under ambiguous contexts and semantic cues.
To investigate sentence planning under uncertainty, we have designed a novel sentence-elicitation paradigm. In this study, UCSD undergraduates will be asked to perform an in-person production task, describing images by typing sentences. Participants are shown an image for each trial which has an active agent, a neutral background, a target object, and, on some trials, a cue object. Each trial differs along three variables: whether the cue is congruent (semantically valid and thematically probable with a target) incongruent, or absent. The purpose of the cue is to encourage speakers to plan their message in a way that’s more probable given the cue, as per Gussow’s findings.
During each trial, the target object starts off obscured with a black box and the cue, if present, is unobscured. The black box serves as a tool for enforcing message uncertainty and the cue encourages predictions for utterance planning. Participants are instructed to begin describing the scene in a specific format within the text box, such as “The [agent] grabbed the [target object].” After the experimental software detects the verb having been typed, the black box is deleted and the target object is revealed. The participant will then end their sentence and the trial by typing the target object. We will measure the latency between the verb and the next word typed. See Figure 1 for an example trial procedure.
In a 1x3 within-subjects design, we will compare the measured latency when a cue is congruent, incongruent, or absent. See Table 1 for example object pairs. We predict that when a cue is congruent to its target, speakers will use it to make a valid prediction that they will incorporate into their utterance plan, thus decreasing the latency to produce the correct response when the expected target is revealed. Conversely, when a cue is incongruent, speakers will make an incorrect prediction, creating an utterance plan that needs to be rejected or adjusted when the unexpected target is revealed, increasing latency. Lastly, when no cue is presented, no prediction is made, potentially resulting in a latency time between that of a congruent cue and that of an incongruent cue. Using fillers, a majority of trials will allow for a correct prediction, encouraging speakers’ continued prediction. The latency data will be analyzed using a linear mixed effects model. Fixed effects would include cue congruency and random effects would include target item variation, subject groups, and individual differences like typing speed.
Preliminary data (28 participants and counting) indicates a statistically significant difference in latency between the congruent and incongruent conditions, with speakers showing less latency between the verb and the object under the congruent condition, p = 0.036.
Moving on to implications, if our results support our hypothesis, we would have a better understanding of how our environments and our expectations can affect sentence planning and fluency; our experiences shape how well we can communicate a message and when that’s likely. Our study could also contribute to natural language models that simulate how speakers produce sentences in dynamic, realistic environments. In a clinical setting, understanding how people utilize semantic cues can have influence language learning, speech therapy, and different communication strategies.
This project is still currently ongoing, and has no conclusion or final deliverables to present yet. I am no longer involved in this project after graduating, but my lab is continuing my work. However, if you are interested in more details, please don't hesitate to email me at amberjiang2002@gmail.com. I would love to chat about specifics!