1. Introduction

During social interactions we routinely process emotional signals, whether from facial (Tracy & Robins, 2008) and bodily (de Gelder, 2016; de Gelder et al., 2004) expressions or from voices (Lima et al., 2019). However, evidence suggests that certain populations such as violent offenders show a different pattern in response to perceiving emotional expressions. Violent behaviour has been associated with deficits in recognising negative emotions and differences in processing stimuli with ambiguous emotional expressions (Blair et al., 2001; Kret & de Gelder, 2013; Wegrzyn et al., 2017). This raises the question of whether deficits in emotion recognition might be addressed by targeted interventions (Baron-Cohen et al., 2009; Schönenberg et al., 2014). Expected outcomes of such interventions might include changes at the perceptual level; for instance, differences in how a stimulus attribute, such as fear in a facial expression, is perceived, as well as changes at the cognitive processing level related to emotion perception (i.e., perspective-taking and empathy).

Virtual reality (VR) offers a new type of targeted intervention that aims to achieve behavioural and subjective changes in the participant (Slater, 2009; Slater & Sanchez-Vives, 2016). Furthermore, unlike traditional purely verbal and cognitive interventions, VR-based therapeutic interventions may be significantly enhanced with embodied perspective-taking. Through embodiment of the VR character, participants are able to experience events from the first-person perspective (1PP) of a virtual character (Gonzalez-Liencres et al., 2020). Virtual embodiment is implemented using a head-mounted display and an end-effector tracking device, where the participants see a life-sized virtual body from a first-person perspective moving synchronously with their actual body movements. This typically gives rise to a strong illusion of ownership of the virtual body and contributes to a sense of agency (Banakou & Slater, 2014; Nierula et al., 2019; Slater et al., 2010). Current studies leveraging VR for therapeutic and social purposes have related these VR-based illusions to sensory perception modification as well as to cognitive effects (Maister et al., 2014). For example, the experience of being virtually embodied in a child avatar results in the subsequent overestimation of the size of objects compared to embodiment in an adult body of the same height as the child (Banakou et al., 2013). Similarly, being embodied in a virtual body of a different race is related to a reduction of implicit racial bias (Maister, Sebanz, et al., 2013; Peck et al., 2013), an effect that may last for at least a week (Banakou et al., 2016). Moreover, in a behavioural study directly related to the present fMRI research, domestic violent offenders were able to overcome their deficit in recognizing fearful female facial expressions after experiencing a female victim’s perspective in VR (Seinfeld et al., 2018).

At present the underlying brain mechanisms sustaining the behavioural and neuropsychological modifications of VR experiences remain largely unknown and unexplored (Blanke et al., 2015; Maister et al., 2014). In relation to the Seinfeld et al. (2018) behavioural study, where results indicated the existence of increased fear recognition after experiencing the perspective of a victim, it is plausible that changes in emotion perception are associated with enhanced processing of specific visual features of the emotional stimuli per se. A possible explanation may be that the VR intervention specifically influenced the perception of fearful facial expressions by enhancing the processing of emotional content in face- and emotion-coding areas including the fusiform face area and the amygdala (Méndez-Bértolo et al., 2016; Vuilleumier et al., 2001). At the same time such activity pattern changes might be linked to social cognition processes. The latter involves the Default-Mode Network (DMN) areas such as the temporo-parietal junction (TPJ), the posterior cingulate cortex (PCC), the anterior temporal lobe (aT), and the medial prefrontal cortices (mPFC) (Arzy et al., 2006; Bernhardt & Singer, 2012; Bzdok et al., 2012; Saxe & Kanwisher, 2003; Tusche et al., 2016; Zaki & Ochsner, 2012). These areas take part in multiple internally directed processes and autobiographical memories, as well as in focused attention and information encoding (Andrews-Hanna, 2012; Buckner et al., 2008; Spreng et al., 2009).

To examine the underlying brain mechanisms of VR (Seinfeld et al., 2018), we designed a study in which male participants passively viewed facial and bodily emotion expressions (i.e., 9-step morph continua between extremely fearful and happy expressions) during two fMRI scanning sessions. Between the two fMRI scanning sessions, participants experienced an immersive VR domestic violence scene from the perspective of the female victim. In the VR scene, participants entered a virtual environment where their bodies were substituted with that of a life-sized virtual female body moving synchronously with their own real movements. Having embodied the virtual female, participants saw a virtual male approaching them, delivering abusive speech and gestures at the participant. Based on a previous study that reported an impact of this same virtual experience on the processing of facial expressions (Seinfeld et al., 2018), we hypothesized that the VR intervention would trigger changes in activity in the processing of the emotional morph stimulus as well as in social cognition network areas. To test this hypothesis, we used whole-brain and regions of interest (ROIs) analysis with repeated-measures ANOVA, and fMRI data were analysed by comparing both fMRI scanning sessions before and after the VR intervention. Specifically, we ran a contrast analysis comparing the fMRI scanning sessions before (pre-VR fMRI) and after (post-VR fMRI) the VR session; we also ran an ANOVA including as factors the fMRI scanning Session (i.e., pre and post VR), the Gender of the emotional morph stimuli (i.e., male and female) and the Morph step (i.e., more or less ambiguous). The pre- and post-VR scanning sessions each consisted of two runs, allowing us to assess within-run habituation effects (Plichta et al., 2014; Wright et al., 2001) due to stimuli repetition, and also to evaluate possible short-term effects of the VR experience in the post-VR session itself. Thus we ran an additional ANOVA including Run, Gender, and Morph step as independent variables to examine these effects. The analysis particularly focused on brain areas previously linked to emotional stimuli perception (i.e., amygdala and fusiform cortex) and social information processing (i.e., DMN areas), examining the interactions between the Run and Session factors and either the Gender or Morph factors. Furthermore, we examined changes in multivariate patterns in these ROIs in the runs across the two sessions (i.e., ANOVA by Run for pattern similarity).

2. Methods and materials

2.2. Experimental design

We used a within-subject design and assessed the impact of the VR intervention by comparing pre-to post-VR measures. For the behavioural responses, we compared pre-VR to post-VR sensitivity to fear in emotional morphs (i.e., see sections 2.3 Emotional stimuli and 2.4 Behavioural tests and questionnaires for details). For the brain responses level, we used a similar methodology comparing pre-to post-VR brain processing (i.e., see sections 2.3. Emotional stimuli and 2.5. fMRI design for details). Each fMRI scanning session consisted of two runs with the same set of stimuli. Given the evidence that brain activity for emotional stimuli undergoes habituation (i.e., decreased activity for repeated stimuli presentation) (Fischer et al., 2003), especially in the amygdala (Plichta et al., 2014; Wright et al., 2001), we also analysed the data by run to account for habituation effects and to evaluate possible shorter-term changes within the session after the VR intervention. Fig. 1 summarizes the different stages of our experimental design.

1. Download : Download high-res image (283KB)
2. Download : Download full-size image

Fig. 1. Stages of our experimental design based on pre- and post-VR intervention.

2.3. Emotional stimuli

For the behavioural tests and fMRI scans, we created 9 step morph continua between 100% fearful and happy emotional expressions. Male and female facial expressions were selected from the Karolinska Directed Emotional Faces inventory (Goeleven et al., 2008; Lundqvist et al., 1998). Bodily stimuli were avatars expressing happy and fearful body postures based on the Bodily Expressive Action Stimulus Test (de Gelder & Van den Stock, 2011). All steps of the emotional morph stimuli were used in the behavioural tests. We specifically used the second, fourth and sixth steps of morph continua stimuli for our fMRI experiment, in order to capture possible changes in fearful-expression processing, as previously shown in Seinfeld et al. (2018). Fig. 2 shows examples of the stimuli used. See Supplementary Materials for details of stimuli construction.

1. Download : Download high-res image (304KB)
2. Download : Download full-size image

Fig. 2. Examples of face and body stimuli presented in the fMRI experiment. The morph steps 2, 4, and 6 were selected from 9-step morph continua from fearful to happy. The IDs of the KDEF faces here were AM14 and AF14.

2.6. VR intervention

Between the pre and post-VR fMRI Sessions, participants went to a room adjacent to the scanner and experienced the same VR scenario as in Seinfeld et al. (2018) with Dutch dubbing. Through a head mounted display (Oculus Rift DK2 HMD, California, United States) participants saw a life-size virtual female body both when they looked down towards their own body and also reflected in a mirror moving in accordance with the participants’ real-time body movements, producing visuomotor synchrony (Fig. 3A, Kinect V2 for Xbox One, Microsoft, Washington, United States). After this embodiment phase, a virtual male entered the scene that approached and verbally abused the participant embodied as the female victim (Fig. 3B and C). Towards the end of the scene, the male avatar was standing face to face with the participant (i.e., female avatar), effectively intruding in their personal space (de Borst et al., 2018) (Fig. 3D). The technical implementation and equipment used for the VR experience are described in Supplementary Materials.

1. Download : Download high-res image (444KB)
2. Download : Download full-size image

Fig. 3. A) Participants were embodied in the 1PP of a female avatar; thus they were able to see the female virtual body when looking down towards their own body or at a virtual mirror. B) A male avatar entered the room and started to verbally assault the female avatar. C) The male avatar approached the female avatar and threw a telephone to the floor. D) At the end, the male avatar was very close to the female avatar (i.e., participants) and invaded their personal space.

3. Results

3.1. Behavioural analysis

When comparing the averaged fear-rating proportion for pre-test stimuli (i.e., four identities) with those for all stimuli in post-test (i.e., four pre-test identities + four novel ones), we found a significant Session × Gender interaction effect (F (1,13) = 4.856, p = .046, η_p² = .272). This interaction indicated that a higher proportion of female faces compared to male faces were rated as expressing fear in the post-test (Fig. 4A). However, this interaction effect disappeared when we compared exactly the same four identities between pre-test and post-test (F (1,13) = .151, p = .704, η_p² = .011), excluding the four other novel stimuli in the post-test (Fig. 4B). This indicated that the female-specific effect could be driven by the novel stimuli identities (i.e., differences in individual face expressivity) in the post-test. We tested and validated this possibility in a behavioural validation study in a separate group without VR intervention (see Supplementary Materials). In this additional study we found that stimulus identity played a significant role in driving the perceived degree of fear in each face. Therefore, it is difficult to make conclusive inferences about the behavioural effects of the VR intervention, even though we found a positive effect of the VR intervention on female fear perception between the pre- and post-tests.

1. Download : Download high-res image (185KB)
2. Download : Download full-size image

Fig. 4. Boxplots with data point showing the difference in the proportion of fear responses between the post-test minus the pre-test (i.e., ΔFearResponses). (A) ΔFearResponses when all stimulus identities are included, thus when four novel identities plus four identities used in pre-test are included. (B) ΔFearResponses when only the same identities are included, thus only the four original identities that comprised the pre-test.

Since we observed a clear identity effect with facial stimuli, and the number of body stimuli identities was lower (two in pre-test and four in post-test) in the behavioural body tests, we decided to not analyse the behavioural data for body stimuli. Note that in the fMRI study this was not an issue as more stimulus identities and an equal number of stimuli identities were included for face and body stimuli.

3.2. Whole brain analysis

We report the results of contrasts and interaction effects, which would reflect the VR influence. See results of the main effects in Supplementary Materials.

3.2.1. Session × Gender × Morph ANOVA

We performed the contrast of post-VR fMRI Session > pre-VR fMRI Session and the ANOVA for face and body stimuli, respectively. For faces, the contrast analysis indicated higher activity in post-VR fMRI Session compared to pre-VR fMRI, in clusters including the ventral medial prefrontal cortex (vmPFC), PCC, ACC, early visual cortex, thalamus, and supramarginal gyrus (Fig. 5A, Table S4 and S5). No interaction between Session and Gender/Morph factors was found using the ANOVA. For body stimuli, the contrast of the post-VR fMRI Session > pre-VR fMRI only showed a cluster in the early visual cortex (Fig. 5B). In the ANOVA, a VR-related interaction of Session × Gender was found in the right superior frontal gyrus, where the activity in the post-VR fMRI Session for male but not for female bodies was lower than in the pre-VR fMRI Session.

1. Download : Download high-res image (506KB)
2. Download : Download full-size image

Fig. 5. Clusters with higher activity in post-VR fMRI Session, in the contrast post-VR fMRI Session > pre-VR fMRI, for face (A) and body (B) stimuli (cluster size corrected for each map by Monte-Carlo simulation, thresholded at alpha = .05, initial p = .001, number of simulations = 5000). Clusters were plotted on the averaged anatomical image across participants (Talairach space, n = 14). The brain images are shown in the radiological convention.

3.2.2. Run × Gender × Morph ANOVA

We performed an ANOVA by Run to capture possible shorter-term influence of VR within the post-VR fMRI Session (effects in run 3 or run 4) that can be reflected by a main effect of Run or by an interaction Run × Gender or Run × Morph. For face stimuli, we found a significant interaction of Run × Morph in the right posterior superior temporal sulcus (pSTS) and left vmPFC, showing that activity patterns for the processing of morph step 2 was different compared to other morphs across runs. In the left vmPFC cluster, activity for morph step 2 was significantly higher than step 4 in run 2, but was significantly lower in run 3 (Fig. 6B,D). No other significant interaction was present (see Table S6 for cluster details). For body stimuli, we found a significant interaction of Run × Morph in the right posterior intraparietal sulcus IPS (see Fig. 6C,E).

1. Download : Download high-res image (741KB)
2. Download : Download full-size image

Fig. 6. Whole-brain ANOVA of Run × Gender × Morph, for face and body stimuli (cluster size corrected for each map by Monte-Carlo simulation, thresholded at alpha = .05, initial p = .001, number of simulations = 5000), clusters plotted on the averaged anatomical image across participants (Talairach space, n = 14). The brain images are shown in the radiological convention. (A) The main effect of run, with extensive overlapping between face and body stimuli, which generally shows lower activity for the subsequent runs after run 1. F: faces; B: bodies. (B) and (C) The interaction of Run × Morph for face (B) and body (C) stimuli. (D) and (E) % BOLD signal change in the clusters of (B) and (C), for each morph step, for faces (D) and bodies (E). Abbreviations: SFS: superior frontal sulcus; mSFG: medial superior frontal gyrus; pIPS: posterior intraparietal sulcus; pSTS: posterior superior temporal sulcus; EBA: extrastriate body area; vmPFC: ventral medial prefrontal cortex.

3.2.3. Habituation effect in both by-session and by-run ANOVAs

For both face and body stimuli, extensive decreases of activity in run 2, 3, and 4 were observed in several regions, likely due to stimulus repetition. This habituation-related effect was observed in bilateral visual areas including the fusiform face area (FFA) for face stimuli. For body stimuli, decreased activity in the post-VR fMRI Session was observed in the extrastriate body areas (EBA), fusiform body areas (FBA), IPS, and inferior frontal gyrus (IFG) (Fig. 5, Fig. 6A).

3.3. ROI analyses: effects in socio-affective related areas

3.3.1. Amygdala, DMN ROIs

We performed an ANOVA for Face/Body × Gender × Morph × Run in areas that have been consistently related to emotional and social-cognition processing, including the amygdala, PCC, TPJ, aT and mPFC. The possible VR influence could be captured by a significant interaction of Run, Gender, or Morph (Fig. 7A, Table 1).

1. Download : Download high-res image (1MB)
2. Download : Download full-size image

Fig. 7. Estimates of BOLD % signals in ROIs related to emotion and empathy processing, and in the fusiform ROI, defined in individual participants. The VR experience was carried out between run 2 and run 3. Error bars and bands denote standard errors. (A). % BOLD signal change for face and body stimuli, across runs. The factor of gender was not plotted, for lack of significant main effects/interactions (apart from in the PCC, see text), and for clarity. ∗: p < .05. See text for post-hoc pairwise comparison results (Sidak adjustment for multiple comparisons). (B). Event-related time courses for face stimuli only, from the same set of ROIs, averaged across participants. The x axes denote number of TRs (1330 ms per TR). The grey bars at the background indicate stimuli block presentation. Abbreviations: PCC: posterior cingulate cortex; TPJ: temporo-parietal junction; aT: anterior temporal lobe; mPFC: medial prefrontal cortex.

Table 1. Statistical details of the significant main effects and interaction effects found through the ROIs ANOVAs.

ROI	Main Effect or Interaction	Detailed Statistical Values
Amygdala	Face/Body	F (1,13) = 17.520, p = .001, ηp2 = .574
Amygdala	Morph	F (2,26) = 4.818, p = .017, ηp2 = .270
PCC	Face/Body × Morph × Run	F (6,78) = 3.860, p = .002, ηp2 = .229
TPJ	Face/Body × Morph × Run	F (6,78) = 2.939, p = .012, ηp2 = .184
aT	Face/Body × Morph × Run	F (6,78) = 2.802, p = .016, ηp2 = .177
PCC	Face/Body × Gender × Morph × Run	F (6,78) = 2.879, p = .014, ηp2 = .181
PCC	Morph × Run (effect only significant for faces)	F (6,78) = 2.209, p = .051, ηp2 = .145
TPJ	Morph × Run (effect only significant for faces)	F (6,78) = 2.611, p = .023, ηp2 = .167
aT	Morph × Run (effect only significant for faces)	F (6,78) = 2.240, p = .048, ηp2 = .147
Fusiform	Run	F (3,39) = 12.148, p = .000009
Fusiform	Morph (no significant effect)	F (2,26) = 1.187, p = .321, ηp2 = .084
Fusiform	Gender (no significant effect)	F (1,13) = .039, p = .846, ηp2 = .003
Fusiform	Gender × Morph (no significant effect)	F (1.315,17.092) = 1.466, p = .249, ηp2 = .101

In the amygdala ROI, we did not find any such interaction effect, but found a significant main effect of Morph (p = .017), and of Face/Body (p = .001), with higher activity for faces than bodies (% signal change (SC) = .115), a result consistent with the literature (Kret et al., 2011).

In the DMN ROIs however, we found a significant Face/Body × Morph × Run interaction in the PCC (p = .002), TPJ (p = .012), aT (p = .016), and a significant Face/Body × Gender × Morph × Run interaction in PCC (p = .014). Subsequent ANOVAs separately performed for faces and bodies showed significant interactions of Morph × Run for faces in TPJ (p = .023), aT (p = .048) and a trend in the PCC (p = .051), not for bodies (all p > .277). Post-hoc pairwise comparisons for faces further showed a specific effect for fearful faces in run 3 (i.e., after the VR intervention). In run 3, there was decreased activity for morph step 2 (i.e., fear) compared to morph step 4 and 6 (i.e., happy) in the PCC (p = .053, d = −.722; p = .028, d = −.812), in TPJ (p = .014, d = −.913; p = .043, d = −.753) and in aT (p = .058, d = −.711; p = .060, d = −.705) (Fig. 7).

To clarify the directionality of DMN activity and to better understand whether the stimuli and task induced activations or deactivations, we performed a one-sample t-test against baseline (fixation only) for each condition of the DMN ROIs (FDR q < .05). We did not find consistent above- or below–baseline activity for most facial expression stimuli conditions in each run. However, face female step 2 (i.e., fear) in run 1 was above baseline in the PCC (% SC = .281; p = .026, d = .672; FDR q = .291), while in run 3 it was significantly below baseline in aT (% SC = −.258; p = .030, d = −.652, FDR q = .0039). Assuming that the activity in DMN areas would have the same directionality for the same task without VR intervention (i.e., pre-VR fMRI), this above-baseline activity in run 1 indicates that the DMN was not consistently deactivated during passive viewing for externally-oriented emotional stimuli in our current study, and should be interpreted as task-induced activation instead.

The activity for face morph step 4 (i.e., ambiguous stimuli) was more consistently above baseline in DMN areas after the VR intervention, as is the case for the mPFC (% SC = .232, p = .046, d = .590, FDR q = .0039) in run 3 for male morph step 4 faces. This is also the case in the TPJ (% SC = .404; p = .006; d = .879, FDR q = .036), mPFC, and PCC (% SC = .397, .356; p = .021, .042; d = .702, .604; without FDR correction) for female morph step 4 faces in run 4. This above-baseline activity for face step 4 in DMN areas was consistent with the vmPFC and PCC clusters observed in the whole-brain post-VR fMRI Session > pre-VR fMRI contrast.

For body conditions, we did not observe consistent directionality for the activity, although male step 4 in run 2 in the PCC was significantly above baseline (% SC = .292, p = .003, d = .973, FDR q = .0259), and body female step 2 in Run 1 in aT was significantly below baseline (% SC = −.274, p = .004, d = −.950, FDR q = .039).

3.3.2. Face-processing ROI

To assess whether this fearful-face-specific DMN activity decrease in run 3 (i.e., run occurring immediately after the VR) also influenced the face-processing areas (i.e., bilateral fusiform ROI), we performed a Gender × Morph ANOVA for face stimuli in run 3, as a follow-up analysis. No significant effects related to the VR intervention were found. We found neither a significant main effect of Morph (p = .321), nor a difference between morph step 2 and 6 in Run 3 (mean difference = −.075, p = .446, d = −.380). We also performed a Run × Gender × Morph ANOVA across runs and found no significant interaction effects with the factor Run (all p > .302). Only a significant main effect of Run was observed (p = .000013), with higher activity for faces in Run 1 than Run 2 and 4 (p = .0000156, d = 2.109; p = .00128, d = 1.357), consistent with the habituation effect observed in the whole-brain analysis.

3.3.3. Averaged time-course in all ROIs

We further extracted the averaged time courses for each face-stimuli condition across runs, to see if there were within-epoch activity changes that may have been missed by the GLM analysis. The activity in DMN areas was more sluggish than in the fusiform ROIs (Fig. 7B), but the latency for the stimulus-locked signal drop in run 3 for fearful female faces (blue) was similar to that of the fusiform ROI (around 4–5 TRs after stimuli onset), while in run 1 this activity was stimulus-locked and above baseline. Also, there was a stimulus-locked elevation of signal for morph step 4 faces (green) mostly in run 4 (also in run 3, but peaked later). These patterns were consistent with the ROI activity versus baseline (with and without FDR correction), and the results of vmPFC and PCC clusters in the whole-brain contrast of the post-VR fMRI Session > pre-VR fMRI.

3.4. RSA analysis in ROIs

To further examine the fine-grained changes of multivariate patterns across runs for face stimuli, we separately correlated the neural representational dissimilarity matrices to the Gender and Morph model (Fig. 8A). Subsequently, we performed ANOVAs with the factor Run on the resulting correlation values.

1. Download : Download high-res image (804KB)
2. Download : Download full-size image

Fig. 8. RSA results in the six ROIs. (A) Example neural matrix of Pearson correlation distance (example taken from S01, in the mPFC ROI, in run 1), and RSA models. The relationship between face conditions were modelled by gender or morph, assuming high multivariate-pattern similarity within category. Blue: similar, red: dissimilar. The neural matrix was split into three compartments, according to the gender model matrix. (B) Spearman correlation of neural matrices to the gender model across runs. (C) The averaged correlation values within the three compartments for each ROI, plotted across runs. (D) The within-gender correlation minus between-gender correlation, plotted across Runs. All correlation values were Fisher’s Z-transformed in the current figure. ∗: p < .05; ∗∗: p < .01; ∗∗∗: p < .001. Apart from the asterisk in mPFC ROI in (C), which denotes a significant difference between within-male and within-female compartments (main effect of Compartment), all other asterisks denote significant linear increase of correlation to the model across Runs.

The neural patterns did not change their similarities to the Morph model across runs (main effects, all p > .193; linear contrasts: all p > .267). However, for the Gender model, we found that the activity patterns over consecutive Runs in TPJ and aT ROIs became increasingly similar to the model (TPJ: main effects p = .043, linear contrast p = .040; aT: main effects p = .060, linear contrast p = .025, Fig. 8B). This linear increase could be driven by (a) an increased similarity within one of the genders (across morph steps), or (b) a decreased similarity between genders, or (c) a net decrease of similarity between genders compared to within genders.

We examined these three possibilities by averaging the correlation of the gender matrix (i.e., using values below the diagonal) in three compartments: within-male, within-female, between-genders. The Compartment × Run ANOVA showed a general linear decrease of correlation values over consecutive runs in PCC and TPJ ROIs (i.e., linear contrast for the Run factor, p = .001, p = 3.791 × 10⁻⁶, Fig. 8C), thus ruling out the possibility of (a) while supporting (b). Since these correlation values correspond to across–morph correlations within each compartment, this result indicates an increase in the pattern variability across runs, probably reflecting more variable processing in these ROIs. Interestingly the fusiform ROI also showed significant linear decrease (i.e., linear contrast: p = .015), indicating that the processing of facial stimuli per se became more variable across runs. In the mPFC there was a main effect of compartment (p = .005), where the pattern for female faces was significantly less consistent than that for male face across runs (mean Z difference = −.057, p = .016). In the aT there was a Compartment × Run interaction (p = .010) and a linear contrast interaction (p = .00007), where the pattern became less consistent for female faces and between genders, but not for male faces. For the ANOVA of the net within/between-gender correlation difference, a significant linear increase across runs was observed again in the aT (p = .025), supporting possibility (c) (Fig. 8D). Thus, we found that across runs there were significant processing differences between stimuli belonging to different genders and a smaller processing difference within each gender (Table 2).

Table 2. Statistical details of the significant main effects and interaction effects found in the RSA analyses (ANOVA by Run, ANOVA of Compartment × Run, ANOVA of the net within/between-gender correlation difference across Runs).

ROI	Significant Main Effect or Interaction	Detailed Statistical Values
TPJ	Run (ANOVA by Run)	Linear contrast: F (1,13) = 5.235, p = .040, ηp2 = .287
		Main effect: F (3,39) = 2.979, p = .043, ηp2 = .186
aT	Run (ANOVA by Run)	Linear contrast: F (1,13) = 6.407, p = .025, ηp2 = .330
		Main effect: F (3,39) = 2.681, p = .060, ηp2 = .171
PCC	Run (ANOVA of Compartment × Run)	Linear contrast: F (1,13) = 17.397, p = .001, ηp2 = .572
		Main effect: F (3,39) = 9.093, p = .0001, ηp2 = .412
TPJ	Run (ANOVA of Compartment × Run)	Linear contrast: F (1,13) = 58.073, p = 3.791 × 10−6, ηp2 = .817
		Main effect: F (3,39) = 22.396, p = 1.348 × 10−8; ηp2 = .633
TPJ	Compartment × Run linear to linear contrast interaction	F (1,13) = 6.234, p = .027, ηp2 = .324
Fusiform	Run (ANOVA of Compartment × Run)	Linear contrast: F (1,13) = 7.794, p = .015, ηp2 = .375
		Main effect: F (3,39) = 6.331, p = .001, ηp2 = .328
aT	Compartment × Run (ANOVA of Compartment × Run)	F (3.237,42.080) = 4.189, p = .010, ηp2 = .244
		Linear to linear contrast interaction: F (13,1) = 32.762, p = .00007, ηp2 = .716
mPFC	Compartment (ANOVA of Compartment × Run)	F (2,26) = 6.693, p = .005, ηp2 = .340
aT	Run (ANOVA of the net within/between-gender correlation difference across Runs)	Linear contrast: F (1,13) = 6.456, p = .025, ηp2 = .332
		Main effect: F (3,39) = 2.746, p = .056, ηp2 = .174

For body stimuli none of the six ROIs showed a change of pattern correlation to either the gender or the morph model (main effects, all p > .205; linear contrasts: all p > .132), indicating that the linear changes of the neural patterns were specific for faces.

4. Discussion

We measured differences in brain activity associated with viewing facial and whole-body expressions before and after male participants experienced domestic violence in VR from the perspective of a female victim. Our study has three major results. First, embodiment in the victim specifically changes the neural processing of facial expressions. Second, post-VR brain activity changes related to processing fearful or ambiguous facial expressions were observed in several DMN regions and mainly consisted of enhanced DMN activity for ambiguous expression morphs compared with the baseline measure. For clearly fearful faces there was decreased DMN activity in run 3 directly following the VR intervention. Third, concerning gender-specific changes, we observed that patterns for processing stimuli within each gender and between the two genders became more variable across runs in several DMN regions (i.e., TPJ, aT) and in the fusiform area; the mPFC showed a more varied pattern for female than male faces throughout all runs. Taken together, these results clearly indicate that virtually experiencing a domestic violence situation from the perspective of the victim has an impact on brain areas related to socio-cognitive processing and to the perception of emotional expressions. The present findings provide a systematic effort to understand the brain mechanisms involved in VR interventions (Maister et al., 2014; Maister, Tsiakkas, & Tsakiris, 2013; van Loon et al., 2018) and specifically the brain mechanisms implicated in offenders’ increased sensitivity in recognizing fear in female expressions after the VR experience (Seinfeld et al., 2018). We discuss these findings now in more detail.

An important finding of the present study is that the VR intervention resulted in increased activity in the DMN when participants were processing ambiguous facial expressions. No other local univariate activity changes in specific face processing areas or the amygdala were observed. This VR-linked activity enhancement for the ambiguous faces may indicate that the participants were processing those emotional faces more intensely. Previous research has shown that the DMN contains crucial nodes supporting the recognition of discrete emotions through conceptualisation and abstraction (Satpute & Lindquist, 2019; Sreenivas et al., 2012). For this reason it has been proposed that there is less involvement of the DMN when less abstraction is needed to create an experience of emotion (see review by Satpute and Lindquist (2019)). Hence, it is reasonable to suggest that greater recruitment of DMN activity when cognitively processing emotional ambiguous stimuli might be related to difficulties in trying to recognize the emotions depicted by the faces. Likewise, since it was easier to recognize the discrete emotion with 100% fearful faces, this may have resulted in less activity of the DMN.

Moreover, decreased activity in the DMN has been consistently reported during the execution of externally-directed cognitively demanding tasks, and increased deactivations have been related to a higher degree of cognitive engagement in the task (Mason et al., 2007; Weissman et al., 2006). In contrast, internally-focused tasks that require participants to actively infer other people’s emotions and thoughts are frequently associated with a positive modulation of the DMN (Bzdok et al., 2012; Decety & Lamm, 2006; Saxe et al., 2006; Schurz et al., 2014). The above-baseline DMN activity in run 1 (i.e. without VR intervention) for fearful female faces suggested that our passive-viewing task did not induce consistent deactivation. Hence, the observed impact of the VR experience on the DMN as a function of stimulus ambiguity is likely related to the cognitive role played by these brain regions in emotion perception (Buckner et al., 2008; DiNicola et al., 2020). Furthermore, the passive-viewing task used in this study did not require the assignment of a definite verbal label to the facial expression, thus it is likely that participants were in a personal experiential mode rather than in a cognitive mode. In line with this, it has been found that the DMN is consistently involved in autobiographic memory retrieval and introspection (Philippi et al., 2015; Whitfield-Gabrieli & Ford, 2012). Our finding of the DMN deactivation for clear unambiguous fearful faces may indicate the DMN’s involvement in these processes. However, further research is required to better understand the differential DMN modulations when presented with faces varying on their emotional ambiguity after the VR intervention.

In the RSA analysis we found a lasting effect of the VR intervention on the DMN, as we observed that the voxel patterns became increasingly different between processing male and female faces across runs in the TPJ and aT. Moreover, for the different steps of the morph continua, activity patterns became more variable within each gender across runs in the PCC, TPJ and fusiform. This gender-based distinction was driven by the bigger decrease of correlation between genders, compared with the smaller decrease of correlation within genders. Moreover, the voxel patterns in the mPFC were more variable for female than male faces throughout the runs, indicating the involvement of the mPFC in gender-based perception. Based on past evidence, these observed patterns of increased variability might be potentially related to a more varied processing of the social-cognitive aspects for the face stimuli (Andrews-Hanna, 2012; Satpute & Lindquist, 2019), a topic which should be further investigated. Moreover, it is interesting to note that in this study male participants were embodied in a virtual female character. In Seinfeld et al.’s (2018) study this resulted in an increased sensitivity in detecting fear in female faces but not in male facial expressions, providing evidence for different behavioural correlates depending on the gender of the stimuli. The increased pattern variability might be related to other factors not specifically linked to the emotion or the gender of the stimuli (e.g., fatigue), but it is unlikely that these additional factors strongly drove the results found for the processing of facial expressions post-VR exposure (i.e., to the same immersive virtual scenario used in Seinfeld et al., 2018), since no significant changes in brain activity patterns across runs were observed for body stimuli. However, future studies might consider including additional control conditions based on the presentation of neutral stimuli such as object morphs.

VR-based clinical interventions, including body ownership illusions, have considerable potential in psychological and behavioural treatments (e.g., training empathy, pain, motor rehabilitation, obesity, etc.). Past studies have explored body ownership illusions using neuroimaging techniques (Blanke et al., 2015; Petkova et al., 2011). However, in this study we used a pre-post experimental design, paralleling as closely as possible the methodology used by Seinfeld et al. (2018), to understand the brain and behavioural impact of a VR intervention based on embodied perspective-taking. To the best of our knowledge, this study is among the first to measure the short-term impact on brain processes associated with affective stimuli processing after a VR intervention. Still, there are some limitations that will need to be addressed by future research. First, we used a within-subject design because of its methodological strengths and its control of inter-individual differences. Since this type of design required multiple repetitions of the stimuli, we observed habituation effects between the runs of the pre-VR scan, complicating the pre-and post VR analysis. However, as explained in the description of our analyses, we controlled for these habituation effects related to repetition by including two runs in both the pre-VR and post-VR scanning sessions. Second, we were interested in including participants with a very specific emotion recognition profile (i.e., difficulties in recognizing fear in female faces). This reduced our sample size (n = 14) and may have lowered power in the present analyses. Also, DMN showed heterogeneity across individual participants, yet showing robust individualized network parcellations, as has been shown by past studies including similar or smaller sample sizes when using large amount of data per participant (Braga & Buckner, 2017; Gordon et al., 2017). Future studies must look into individual variability of DMN to understand whether similar results are obtained with bigger sample sizes and with more data per participant. Third, since in the behavioural screening test fewer body identities than face identities were included, we were unable to perform statistical analysis on this behavioural data. In relation to this topic, it is important to note that the behavioural screening test had a stronger focus on faces (i.e., more identities included) since the primary goal of the behavioural screening was to try to match the emotional recognition profile (i.e., decreased recognition for fearful female faces) of Seinfeld et al. (2018) in a new sample of male participants that had undergone fMRI scans before and after the VR intervention. However, future studies should consider including the same number of facial and bodily identities, to better understand the behavioural impact of the VR intervention. Fourth, in the present study we found a strong identity effect in the behavioural pre- and post-tests, indicating that the behavioural results should be interpreted with caution. Note that the limitations encountered in the present study for the behavioural data analysis do not apply to the analysis of the brain data recorded through the fMRI scans, since here special care was taken to include more identities for face and body stimuli and compare exactly the same stimuli identities between pre- and post-VR brain scans, as well those as across runs.

Some possible explanations for the unexpected effect of personal identity are worth considering. First, actors appearing in emotional stimuli are not equally good across the whole range of emotional expressions. For instance, some actors are much better portraying expressions like fear and others potraying emotions like anger. We know from our extensive experience of constructing stimulus sets (e.g., de Gelder et al., 2015; de Gelder & Van den Stock, 2011) and the facial expression literature (Krumhuber et al., 2020) that some actors are more expressive than others. This type of individual face variability is well known from efforts at putting together a set of different expression images that is balanced for identity and expression. As a consequence, to keep identity stable, sometimes one has to settle for an identity with a less convincing expression. Secondly, although all the morphs were generated with the same methods and pixel-wise steps, the procedure may work out differently at the fine-grained level for different identities. This may be in part related to the fine morphology of the facial features. New methods may allow to morph facial expressions based on real-time motion capture of natural facial movements. However, based on the present experimental design we are unable to discern whether these factors played a role in the present findings. Prior to observing this identity effect, we assumed in line with the literature that only a few identities were needed for the behavioural tests, because the stimuli within the same emotion category condition are more or less equivalent tokens of that same category. However, researchers are increasingly aware of this issue in recent years, with some authors suggesting the stimuli being treated as random-effects, and advocating the use of statistical tools that incorporate mixed-effects analysis to cope with this problem (Westfall et al., 2017).

Another aspect that deserves further research is the role played by the visual perspective in VR, which in the present study was a first-person perspective experience. The specific role played by the visual perspective could be investigated by including a third-person perspective condition (Galvan Debarba et al., 2017; Gonzalez-Liencres et al., 2020) or by using asynchronous multisensory stimulation (Kokkinara & Slater, 2014). Also, we did not examine the effect of arousal, which may be induced by the current VR scenario. This could potentially be examined by behavioural control experiments with interventions elevating arousal level but without VR, or control experiments with different VR scenarios without the domestic violence content. Future studies might also benefit from using ecologically valid VR scenarios to better understand different clinical disorders and to gain knowledge about cognitive processes when participants are immersed in realistic real-life simulations, for example see Reggente et al. (2018). Finally, further research is needed to better understand the impact played by the gender or age of the virtual victim.

Authors contributions

Seinfeld, Sofia: Conceptualisation, Methodology, Validation, Formal analysis (behavioural), Investigation, Writing-Original Draft; Zhan, Minye: Conceptualization, Methodology, Validation, Formal analysis (fMRI), Investigation, Writing-Original Draft, Data Curation, Software; Poyo-Solanas, Marta: Investigation, Validation, Formal Analysis (behavioural), Writing – Review & Editing; Barsuola, Giulia: Investigation; Vaessen, Maarten: Writing – Review & Editing; Slater, Mel: Supervision, Writing – Review & Editing; Funding acquisition; Sanchez-Vives, Maria V.: Supervision, Writing – Review & Editing; Funding acquisition; de Gelder, Beatrice: Conceptualization, Methodology, Writing-Original Draft, Review & Editing, Project administration, Resources, Funding acquisition.

Data Files

We provide open access to the data files (i.e., behavioural, ROI, RSA), fMRI statistical maps, stimuli images, VR questionnaire, and fMRI experiment code used in the present study in the following link: https://osf.io/xmbn3/?view_only=cf6f708f17764f3e843594d36f024e2e.

The conditions of our ethics approval do not permit public archiving of the raw MRI data obtained in this study. Readers seeking access to the data should contact the first author Minye Zhan (zhanminye@gmail.com) or the lead author Beatrice de Gelder (b.degelder@maastrichtuniversity.nl). Access will be granted to named individuals in accordance with ethical procedures governing the reuse of sensitive data. Specifically, requestors must meet the following conditions to obtain the data: the data should be used solely for academic research and should not be publicly archived.

Legal copyright restrictions prevent public archiving of the Interpersonal Reactivity Index and Buss-Perry Aggressiveness Questionnaire, which can be obtained from the copyright holders in the stated references. The code of the virtual reality setup is currently owned by Virtual Bodyworks, interested researchers in obtaining access to the VR setup and specifications might contact the corresponding author of the present paper (b.degelder@maastrichtuniversity.nl).