Semantic Processing in Deaf and Hard-of-Hearing ... - Semantic Scholar

1 downloads 0 Views 2MB Size Report
Aug 10, 2016 - Lund, Sweden, 3 Department of Psychology, Stockholm University, Stockholm, Sweden, 4 Department of Behavioral ...... 117, 1772–1782. doi:.

ORIGINAL RESEARCH published: 10 August 2016 doi: 10.3389/fpsyg.2016.01146

Semantic Processing in Deaf and Hard-of-Hearing Children: Large N400 Mismatch Effects in Brain Responses, Despite Poor Semantic Ability Petter Kallioinen 1, 2*, Jonas Olofsson 3 , Cecilia Nakeva von Mentzer 4 , Magnus Lindgren 5, 6 , Marianne Ors 7, 8 , Birgitta S. Sahlén 5, 9 , Björn Lyxell 4 , Elisabet Engström 10 and Inger Uhlén 10 1

Department of Linguistics, Stockholm University, Stockholm, Sweden, 2 Lund University Cognitive Science, Lund University, Lund, Sweden, 3 Department of Psychology, Stockholm University, Stockholm, Sweden, 4 Department of Behavioral Sciences and Learning, Swedish Institute for Disability Research, Linkoping University, Linkoping, Sweden, 5 Linneaus Centre, Cognition, Communication and Learning, Lund University, Lund, Sweden, 6 Department of Psychology, Lund University, Lund, Sweden, 7 Division of Clinical Neurophysiology, Department of Clinical Neuroscience, Lund University, Lund, Sweden, 8 Division of Clinical Neurophysiology, Department of Clinical Neuroscience, Skåne University Hospital, Lund, Sweden, 9 Logopedics, Phoniatrics and Audiology, Department of Clinical Sciences Lund University, Lund, Sweden, 10 Department of Hearing and Balance, Karolinska University Hospital and Karolinska Institutet (CLINTEC), Stockholm, Sweden

Edited by: Guillaume Thierry, Bangor University, UK Reviewed by: Christopher Conway, Georgia State University, USA Jon Andoni Dunabeitia, Basque Center on Cognition, Brain and Language, Spain *Correspondence: Petter Kallioinen [email protected] Specialty section: This article was submitted to Language Sciences, a section of the journal Frontiers in Psychology Received: 28 February 2016 Accepted: 18 July 2016 Published: 10 August 2016 Citation: Kallioinen P, Olofsson J, Nakeva von Mentzer C, Lindgren M, Ors M, Sahlén BS, Lyxell B, Engström E and Uhlén I (2016) Semantic Processing in Deaf and Hard-of-Hearing Children: Large N400 Mismatch Effects in Brain Responses, Despite Poor Semantic Ability. Front. Psychol. 7:1146. doi: 10.3389/fpsyg.2016.01146

Frontiers in Psychology |

Difficulties in auditory and phonological processing affect semantic processing in speech comprehension for deaf and hard-of-hearing (DHH) children. However, little is known about brain responses related to semantic processing in this group. We investigated event-related potentials (ERPs) in DHH children with cochlear implants (CIs) and/or hearing aids (HAs), and in normally hearing controls (NH). We used a semantic priming task with spoken word primes followed by picture targets. In both DHH children and controls, cortical response differences between matching and mismatching targets revealed a typical N400 effect associated with semantic processing. Children with CI had the largest mismatch response despite poor semantic abilities overall; Children with CI also had the largest ERP differentiation between mismatch types, with small effects in within-category mismatch trials (target from same category as prime) and large effects in between-category mismatch trials (where target is from a different category than prime), compared to matching trials. Children with NH and HA had similar responses to both mismatch types. While the large and differentiated ERP responses in the CI group were unexpected and should be interpreted with caution, the results could reflect less precision in semantic processing among children with CI, or a stronger reliance on predictive processing. Keywords: children, cochlear implants, hearing aids, semantics, N400 evoked potential

INTRODUCTION In a spoken language environment, impaired hearing can limit the development of words, concepts and ultimately language comprehension and communication in children. Indeed, deaf and hardof-hearing children (henceforth, DHH) children have, on average, a more limited vocabulary than their peers (e.g., Luckner and Cooke, 2010 for a review; Blamey et al., 2001; Geers et al., 2003;


August 2016 | Volume 7 | Article 1146

Kallioinen et al.

DHH Children’s Semantic Brain-Responses

Semantic processing can be investigated using the event related potential (ERP) component N400, and the N400 is arguably the most studied brain response in language processing research (Kutas and Federmeier, 2011). The typical N400 component is a negative peak at centro-parietal electrodes around 400 ms after event onset, elicited by meaningful stimuli such as spoken or written words (Kutas and Federmeier, 2011), but also pictures (West and Holcomb, 2002; Franklin et al., 2007; Proverbio and Riva, 2009). Semantically improbable or incongruent stimuli elicit large negative N400 responses compared to probable or congruent stimuli. The N400 is modulated by semantic structure; when primes are semantically related, but mismatching, to targets, the N400 amplitude is reduced (Kutas and Federmeier, 2011). This relatedness effect might be due to an increased use of predictive processing (Franklin et al., 2007; Kutas and Federmeier, 2011). In study designs using picture targets, the N400 is typically preceded by the N300, a more frontal negative component that responds to very distinct semantic deviations such as unrelated or betweencategory mismatches (Barrett and Rugg, 1990; McPherson and Holcomb, 1999; Hamm et al., 2002). Research on CI routinely uses ERP assessment, but the focus is often on processing of auditory stimuli in cortical auditory evoked potentials (CAPS) and auditory oddball paradigms (Groenen et al., 2001; Martin et al., 2008; Peterson et al., 2010). Traditional ERP components such as the P1-N1-P2 complex, acoustic change complex (ACC), mismatch negativity (MMN), and P3 have been used to assess auditory discrimination, maturation and intervention effects in persons with CI (Kraus et al., 1993; Okusa et al., 1999; Eggermont and Ponton, 2003; Beynon and Snik, 2004; Kral and Sharma, 2012; Näätänen et al., 2012; Timm et al., 2012; Vavatzanidis et al., 2016) and with HA (Thai-Van et al., 2010). Studies of N400 responses are scarce, in particular among DHH-children, leaving their semantic processing changes relatively unexplored on a biological level (Johnson, 2009). We are aware of only one study of N400 conducted on a child with CI. In this study (Key et al., 2010), N400 responses were recorded from a 6-year-old girl with unilateral CI from 2 years of age. Assessment before and after activating the CI resulted in a dramatic increase of the N400. A few studies report N400 results among adult CI-users with post-lingual deafness. One study (Hahne et al., 2012) assessed 13 CI-users (mean age 51 years) and found N400 effects for both semantic violations and cloze probability manipulations in an auditory sentence comprehension test. The N400 effects consisted of later and more long-lasting peaks among CI-users than controls. Another study (Finke et al., 2016) with 13 CIusers (mean age 60 years) found an N400-like effect in an oddball task with word stimuli, although the authors described it as an N2 component. Here, ERP latencies were associated with listening effort and intelligibility in the CI group. A third study (Henkin et al., 2015) assessed 9 CI-users (mean age 66 years) in a voice gender discrimination task with auditory word stimuli. Results showed nominally longer N400 latencies among CI-users compared to controls, however, the difference was not tested for statistical difference.

Le Normand et al., 2003; Kenett et al., 2013; Walker and McGregor, 2013). With a small vocabulary, an underdeveloped semantic structure (i.e., the taxonomic, associative or similaritybased relations between words) could also be expected, but research indicates a large heterogeneity among DHH children (Peterson et al., 2010; Löfkvist et al., 2012; Kenett et al., 2013; Li et al., 2013; Nakeva von Mentzer, 2014). In fact, semantic and other cognitive cues may play a more important role in linguistic processing of DHH children, as a means to compensate for poor phonological skills (Lyxell et al., 2009; Nakeva von Mentzer et al., 2014a). What cannot be extracted from the speech signal bottom-up might be inferred using top-down processes (Wingfield and Tun, 2007). Thus, predicting semantic content might be of importance for DHH persons, due to their difficulties in extracting semantic content from speech input. Children who are DHH are characterized by widely varying etiologies and symptoms. The most common mitigation for hearing deficits in DHH children is hearing aids (HAs) and/or cochlear implants (CIs). Traditional HAs amplify sounds and optimize the auditory input for the children’s residual sensory function. In contrast, CIs convert sounds to coded electrical signals that are transmitted to the auditory nerve in the cochlea, enabling access to sound. Cochlear implants have shown to recover auditory function also in profoundly deaf individuals (Henkin et al., 2003; Sullivan, 2013). Listening through HAs or CIs is associated with specific limitations (Moore, 2008; Nittrouer et al., 2012). For example, the limited temporal and spectral resolution of the CI signal can lead to difficulties in perceiving segments in consonant clusters and other aspects of speech. Phonological skills involve the decoding of speech into linguistically relevant information such as phoneme combinations that are central for learning, storing and accessing words (Ramus and Szenkovits, 2008; Stoel-Gammon, 2010; Dillon et al., 2012). Poor phonological skills might explain DHH children’s poor performance in many cognitive and linguistic tests, such as those assessing lexical access and lexical variation (Lyxell et al., 2009; Asker-Árnason et al., 2010). The present study investigated cortical processing of semantics before and after a computer-assisted reading intervention with a phonics approach (Nakeva von Mentzer et al., 2013, 2014b). The intervention focus is on strengthening the connection between graphemes and phonemes, which was hypothesized to boost phonological awareness skills, which in turn could enhance lexical access and vocabulary development. The intervention did have effects on phonological processing, in particular for DHH children starting with low phonological skills (Nakeva von Mentzer et al., 2013). There were also effects on reading skills (Nakeva von Mentzer et al., 2014b), however semantic tasks such as lexical prediction was not affected. Children with CI performed worse than controls on auditory lexical prediction tasks (Nakeva von Mentzer et al., 2013). This is in apparent contrast to recent results using picture naming that show semantic performance et al. with controls (Löfkvist et al., 2014; Wechsler-Kashi et al., 2014), but this difference might be explained by varying difficulty in processing speech stimuli.

Frontiers in Psychology |


August 2016 | Volume 7 | Article 1146

Kallioinen et al.

DHH Children’s Semantic Brain-Responses

to their spoken language. All children performed within normal limits on nonverbal intelligence as assessed by Ravens colored matrices, and there was no significant difference between the groups regarding nonverbal intelligence (see Table 1 and Nakeva von Mentzer et al., 2013). Four more children participated in the study but were excluded from the present analysis (one control did not meet inclusion criteria, one control did not participate in the training intervention, and two children with CI were excluded due to ERP-recording issues). The DHH children were found through clinical records of the participating hospitals. All children were invited who fulfilled the criteria; 5–7 years old with bilateral hearing aids and/or CIs, speaking Swedish in their educational setting, and with no known disability affecting language development. Invitations were sent to 90 families and approximately one third of those accepted to participate. The controls were recruited from preschools and schools in the Stockholm area. Written informed parental consent was obtained for all the participants. The study was approved by the Regional Committee of Medical Research Ethics in Stockholm.

In the present study, we investigated semantic processing in DHH children using an N400 paradigm with spoken primes and picture targets. The spoken primes were either fully congruent with targets (matching), unrelated to the target (betweencategory mismatch), or a mismatching prime that was related to the target by category membership (within-category mismatch). Participating children were asked whether the picture target matched the word prime or not. The task challenged semantic processing, and allowed us to compare DHH children to matched controls with normal hearing (NH). We compared results from the two mismatch types to investigate effects related to semantic structure. We hypothesized differences in brain responses between normal hearing children (NH), children with HA and children with CI, reflecting increased semantic difficulties related to the severity of hearing impairment. Presumably this would be reflected in smaller mismatch effects overall (NH > HA > CI), or smaller response to within-category mismatches relative to between-category mismatches, due to a less fine-grained semantic structure in DHH children. We also investigated effects of a reading intervention with the phonics approach directed at beginning readers (Lovio et al., 2012). This intervention was hypothesized to strengthen phonological awareness by training grapheme-phoneme correspondence. We hypothesized that better phonological awareness among DHH children would make words more distinct and thereby easier to process semantically, resulting in larger N400 mismatch effects after training.

Language Testing, Intervention, and ERP Recording Participation started 1 month before the first ERP-recording, with a set of assessments of language and cognitive skills, conducted in a quiet room in the children’s homes or in their educational setting. The same tests were repeated on the day of the first ERP recording, and on the day of the second ERP recording following a month of intervention training. Tests, scores and behavioral effects of the intervention were described previously, i.e., the phonological composite variable was described in Nakeva von Mentzer et al. (2013), and a reading composite variable and lexical expectation test were described in Nakeva von Mentzer et al. (2014b). Scores on key tests are presented in Table 1. The N400 procedure was identical across the two ERP sessions. Participants sat in front of a monitor at a distance of approximately 1 m. Each trial started with a fixation cross

MATERIALS AND METHODS Participants This study was based on data from 42 children (21 girls) aged 5–7 years. Thirty of them were deaf or hard-of-hearing (DHH) and 12 were normal hearing controls (NH; 3 girls). Of the DHH children, 15 had bilateral hearing aids (HAs) and 15 had at least one cochlear implant (CI). In each of these groups 9 were girls. Nine children (7 girls) had bilateral cochlear implants, and six children (2 girls) had CI in one ear and hearing aid in the other. Participants were grouped based on their type of hearing amplification: NH, HA, and CI (at least one implant). Seventeen children (9 girls) had a severe/profound hearing impairment with a pure tone average (PTA) at > 70 dB Hearing Level unaided. Eleven children (7 girls) with hearing aids had a moderate HI (PTA 40-60 dB) and two children (girls) had a mild HI (PTA < 40 dB). The mean age at diagnosis was 1 year and 2 months, ranging from 0 weeks to 5 years. Seven children were diagnosed with a progressive hearing impairment, where one child was born with unilateral deafness and later developed progressive hearing impairment on the other ear. The mean age for receiving HA was 2 years and 8 months (ranging from 3 months to 6 years) and the mean age for first CI-operation was 1 year and 7 months (ranging from 11 months to 5 years). Aided thresholds with CI or HA were at 20–40 dB, with higher values in the high frequencies for children with hearing aids. Three children had another spoken language besides Swedish, two children used sign language as their first mode of communication at home and used spoken Swedish in school and two children used sign support

Frontiers in Psychology |

TABLE 1 | Selected test results (from first ERP session) presented as means and standard deviations for each hearing amplification group. Hearing amplification groups, selected test results Hearing amplification group

Age (months)




hearing (NH)

aid (HA)

implant (CI)

Mean (Std)

Mean (Std)

Mean (Std)

81 (12.0)

76 (11.9)

76 (11.0)

85.8 (24.0)

76.3 (18.7)

75.3 (24.5)

Phonological composite

86.7 (7.9)

68.6 (10.8)

59.3 (17.4)

Lexical access

14.8 (1.8)

13.6 (4.3)

9.3 (6.6)

Raven colored matrices (%)

Reading skill composite

0.111 (0.126)

Auditory ERP response

2.66 (0.66)

1.69 (0.76)

2.19 (1.32)





0.044 (0.07) 0.083 (0.131)

The auditory ERP response is the average amplitude at 6 fronto-central electrodes at a latency of 80–220 ms in response to both standards and 4 deviant types in a MMN paradigm (see Uhlén et al., in preparation).


August 2016 | Volume 7 | Article 1146

Kallioinen et al.

DHH Children’s Semantic Brain-Responses

amplifier, electrode nets of the hydrocel 129 channel type (EOG channels were removed leaving 125 channels), using Cz as a reference channel and a ground channel positioned between CPz and Pz. In this system recordings are sampled at 20,000 Hz, low pass filtered online with a cut off at 4000 Hz and resampled to 250 Hz. The impedance of the channels was kept below 50 Ohm as recommended by the manufacturer. Hearing aids were refitted after the net was applied. Recordings were filtered offline with a 1–40 Hz band pass FIR filter, resampled to 125 Hz and epoched. Only responses to picture targets were considered for the present study. Epochs with extreme amplitudes (exceeding ±500 µV) were rejected. Epoched data was subjected to preprocessing procedures in EP toolkit (Dien, 2010). Blink artifacts were removed with an automatic procedure, where independent component topographies are matched to a blink template. Movement artifacts were isolated using PCA and an amplitude criterion (i.e., principal components of single trial data with more than 200 µV amplitude change were removed). Channels with poor signal quality were identified globally (by means of correlation: correlation with neighboring channels should be above 0.4 but not perfect) and per epoch (amplitude differences within the epoch should be below 200 µV). Data from these channels were interpolated. A negligible number of artifacts could be attributed to CIs (Gilley et al., 2006; Debener et al., 2008; Viola et al., 2012) and did not demand special treatment. In total, 6% of all trials were rejected, and each subject retained on average 220 trials (SD = 28) of 240.

followed by a spoken word presented after 1 s. Word primes consisted of recorded spoken words (in Swedish) naming baselevel common objects like foods, animals, clothes, body parts, vehicles, furniture, baby supplies, kitchen utensils and outdoor objects. Word primes were delivered at 75 dB (SPL). Picture targets were presented 2.3 s after word onset. After picture presentation participants indicated if the picture matched the word by pressing buttons on a response box corresponding to “yes” or “no.” This procedure was repeated for 120 trials. The procedure was introduced by a short training session including trials similar to those of the experimental paradigm, but without time limits for the response. When these trials were successfully completed, further trials included time limits for the response. Each stimulus pair consisted of a spoken prime followed by a picture target. The pairs were of three types, constituting the semantic conditions of the experiment: matches, where the target is a typical illustration of the prime word (e.g., “wolf ” followed by a picture of a wolf), within-category mismatches, where the target is an illustration of another object than the prime, but from the same category or domain (e.g., “wolf ” followed by a picture of a bear) or between-category mismatches where there is no apparent semantic link between prime and target (e.g., “wolf ” followed by a picture of a car). There were 40 stimulus pairs in each condition, in total 120 pairs that were presented in mixed and random order. The pictures consisted of simple color drawings, depicting familiar objects in a cartoonlike or realistic manner. Pictures were presented on the screen against white background (width 12–18 cm and height 12–20 cm). Presentation and randomization of stimuli was handled by E-prime 2.0 software (Psychology 370 Software Tools Inc., 20121 ; Pittsburgh, PA). Note that while the targets were pictures, the mismatch effects depend entirely on perceiving and deriving meaning from the spoken primes. A speech pathologist with experience working with DHH children prepared the words and pictures used as stimuli, the prime-target pairings, and recorded the spoken word stimuli. There was no quantitative matching of lexical, auditory or visual features of stimuli between conditions, as they were all very familiar base-level nouns and objects, chosen with intelligibility in mind. Stimulus pairs are described further in the Supplementary Materials. The behavioral procedure was slightly revised after 10 of the included participants were tested (4 CI, 6 HA, 0 NC), after concerns that that use of the response box was confusing for some participants. Visual feedback was added in each trial, and a visual prompt for responses was omitted. All participants are included in the present analysis, but a complementary analysis excluding the first 10 participants is provided in the Supplementary Materials. This analysis reveals highly similar results and suggests that this subtle methodological change had no effect on outcomes.

RESULTS First, we established the presence of semantic mismatch effects in the EEG data. Thus, data (collapsed across groups and intervention conditions) were visually inspected, and showed a large negative fronto-central peak in responses to pictures in all semantic conditions. Difference waves, produced by subtracting responses to matching pictures from responses to mismatching pictures, showed a broad negative deflection and polarity shift at lateral sites. A broad electrode window was used to capture these effects (all electrodes except edges and lateral sites, see Figure 1A). Only one electrode region was used, in order to reduce number of factors in the ANOVA (Luck, 2014). To obtain information about the time course of this semantic differential, average amplitudes of these electrodes were tested for semantic mismatch effects in series of T-tests, using 50 ms bins, from 0 to 650 ms after stimulus onset. Between 350 and 500 ms, both mismatch types differed significantly from congruent responses (see Table 2B), constituting an overall semantic mismatch effect in line with typical N400 descriptions (Picton et al., 2000; Kutas and Federmeier, 2011). T-values over time bins are presented in Figure 1B with separate lines for within-category and betweencategory mismatches, each compared to the congruent responses. While significance levels presented in Table 2B are uncorrected, the N400 difference results remain significant at the 0.05 level (one-tailed) also with Bonferroni-correction (corrected p-value 0.05/26 = 0.002 and critical t-value at 3.05).

EEG Recordings and Processing We recorded EEG at Department of Linguistics at Stockholm University, and at Humlab, Lund University using identical equipment from EGI (Electrical Geodesics Inc.), net amp 300 1 Psychology Software Tools

Inc. (2012). Retrieved from

Frontiers in Psychology |


August 2016 | Volume 7 | Article 1146

Kallioinen et al.

DHH Children’s Semantic Brain-Responses

FIGURE 1 | (A) Overview of ERP responses: average responses at FCz and Oz, and topographic maps for the N400 time window 350–500 ms. Averages include all participants, collapsed across pre- and post-intervention. (B). Within- and between-category mismatch effects (mismatches compared to matches) over time. T-tests based on amplitude averages in 50 ms time windows in the Region of Interest (ROI). Both effects were found in the time window 350–500 ms, which was used for the main analysis. Topographic maps show the mismatch effect amplitude differences, in 50 ms time windows and in the critical 350–500 ms interval.

the intervention factor interacting with semantic condition. A three-way interaction between group, semantic condition and intervention reached significance (Figure 2C). In the pretraining session, responses were similar among groups except for between-category responses (where the response was largest for the CI group and smallest for NH children). In the posttraining session, groups were more different, with both mismatch responses larger than before for NH, almost no differences among children with HA and essentially the same response to congruent and within-category incongruent for CI-users (still with a large between effect). This pattern does not fit predictions of improved semantic processing for DHH children due to the intervention. Rather, it indicates that group differences in the first session were somewhat enhanced in the second. In sum, the analysis confirmed typical N400 incongruence effects, and show differences in semantic processing among groups. Smaller mismatch effects overall for HA-users, and little within-category

The 350–500 ms time window was used for the main analyses to assess group differences in the N400 response in relation to semantic difficulties among DHH children. To this end, three factors were analyzed in a repeated-measures ANOVA: semantic incongruence (within-category, betweencategory, and congruent), intervention (before and after) and hearing amplification group (CI, HA, and NH). There was a main effect of semantic condition (Figure 2A, Table 2A), again confirming an overall semantic N400 effect among the participants, where the congruent condition differed from both mismatch types. There was also an interaction between semantic condition and group: Whereas between-category vs. congruent trials displayed the largest mismatch effect for the CI group, within- and between-category mismatch types were similar for the NH and HA groups. The HA group had less pronounced mismatch effects overall (Figure 2B). There was no meaningful main effect of intervention, or of

Frontiers in Psychology |


August 2016 | Volume 7 | Article 1146

Kallioinen et al.

DHH Children’s Semantic Brain-Responses

TABLE 2 | (A) Main effects and interactions of semantic conditions, training and group. Significant effects, and the nonsignificant effect of training is included. (B) Mismatch effects were assessed in 50 ms time windows across all participants, to establish the time-window of the effects (dotted square). Group-specific tests explored the semantic condition × group interaction over time. (C) Explorative correlation of peak mismatch effects and language test variables. Results summary (A) Repeated measurements ANOVA (N = 42)


(B) T-values of within and between category mismatch ERP effects in time bins

Partial eta squared

Semantic conditions

2, 78


Suggest Documents

The Equalizer 2 (2018) [BluRay] [720p] | Simple Sandbox v1.5.0 APK | PelisPedia