CG versus human faces: Can we tell them apart, and do our first impressions differ?

​

Computer-generated (CG) beings are rapidly infiltrating our social world (e.g., in e-therapy and medical training, and virtual influencers such as Lil Miquela). CG faces are also increasingly being used in face research, predominantly to investigate questions about human face processing. Research directly comparing responses to CG versus human faces has produced mixed findings, making it difficult to conclude whether CG faces are appropriate substitutes in real life and research settings. Here, I will present meta-analytic results investigating two core questions about responses to CG versus human faces: (1) Can observers tell CG and human faces apart, and (2) Do first impressions for CG faces differ from human ones? Results suggest that while CG faces are clearly distinguishable from human ones, they are rated just as favourably as human faces on trait judgements (e.g., trustworthiness, attractiveness). Moderator analysis revealed CG faces were evaluated more favourably than human ones when stimuli were moving, suggesting movement may facilitate positive first impressions of CG faces and potentially downstream social connections (e.g., people may trust virtual influencers more when shown moving).

Testing Three Asymmetries in Scene Gist Perception

​

Scene gist extraction relies on rapid global processing. Are these global properties extracted
equally well across the visual field? Earlier findings indicate a right hemisphere advantage
for global processing, which suggests a left visual field advantage for scene gist processing.
Studies examining vertical perceptual asymmetries indicate a lower hemifield advantage over
the upper hemifield in visual perception. Furthermore, there are possible limits to
simultaneous monitoring of multiple scene categories, such as asymmetrical performance
with different cue positions. The current study directly tested these three asymmetries in a
single scene categorization task based on a modified divided visual field paradigm on right-
handed participants (n = 21). Results showed better performance when target scene categories
were cued before image presentation (mean d’ = 0.98) than after (mean d’ = 0.47) (p<.001,
partial eta-squared = 0.76), providing further evidence of limits in scene gist extraction. No significant
differences were found with horizontal or vertical visual field asymmetries when results were
pooled over response hand. However, the expected differences were seen significantly when
responses were made using the right hand only, with better performance when the target
scene appeared in the left visual field (mean d’ = 0.82) than the right visual field (mean d’ =
0.66) (p = .008, partial eta-squared = 0.32 ), and when the target appeared in the lower hemifield (mean d’ = 0.70) than in the upper hemifield (mean d’ = 0.57) (p = .046, partial eta-squared = 0.19). These findings suggest that, with an improved experimental design, this task could provide evidence that scene gist processing is right hemisphere lateralized like other global processes and could
also extend findings of lower hemifield advantages to scene stimuli.

Understanding emotion processing in Schizophrenia Spectrum Disorder (SSD): Specific deficit or difficulty confound?

It is well documented that face processing deficits occur among persons with schizophrenia
spectrum disorder (SSD). It is unclear whether specific deficits occur when judging emotional
expressions, or emotion processing deficits are accounted for by a general face processing
deficit (e.g., emotion, gender, identity). The literature provides conflicting evidence, which may
be explained by the general cognitive deficit in SSD interacting with task difficulty and
confounding results. That is, tasks made too easy or difficult dilute the general cognitive deficit
in SSD, which can be misinterpreted as preserved abilities. Here, we conducted a meta-analysis
to assess SSD-related performance across emotional versus non-emotional face processing
tasks. We consider: 1) the relative size of SSD-related deficits across tasks, and 2) whether the
pattern of deficits can be explained by systematic differences in task difficulty. We present
preliminary results on the first synthesis of face processing in SSD that considers the difficulty
confound.

The temporal and spatial resolution of a web-based eye tracking system

​

At the University of Victoria, we have developed a new web-based eye tracking
program called Gazer. In two experiments, we tested the temporal and spatial properties of
Gazer. In an exogenous cuing task (Experiment 1), we found that Gazer produced eye
movement results that were comparable to the results of the laboratory Eyelink 1000 system. In
Experiment 2, we used a Where’s Waldo visual search task where participants identified the
location of a Waldo target via a space bar response. Analysis revealed that participants
initiated an endogenous eye movement to the Waldo target approximately 3000ms before their space bar response. Together, the results indicate that the Gazer system has sufficient temporal
and spatial resolution to collect reliable eye tracking data via the internet.

People’s visual information use and internal prototypes of facial emotions - comparing across levels of eating disorder symptoms

​

Previous studies of emotion categorization abilities of people with eating disorders used accuracy and reaction time in emotion categorization tasks to identify performance deficits for these individuals. I build on this work with two advanced psychophysical methodologies - the bubbles task, and the reverse correlation procedure. In a well-powered student sample, I outlined regions of visual information which are critical to emotion categorization via the bubbles task. I was also able to generate a range of internal emotion prototypes which substantially differed across individuals, according to perceptual ratings of new participants. However, neither visual information use, the quality of internal prototypes, or the standard facial expression recognition accuracy were related to eating disorder severity in this sample. I provide potential future directions in this area by examining exploratory relationships between behavioral task performance and symptoms of comorbid conditions.

Forgetting a face: Attribute amnesia for familiar identities

​

People can recognize faces across variation in expression, viewpoint, and social contexts extremely quickly and with relative ease. The speed and ease at which faces can be identified suggests that identity processing is automatic, but does variation in expression, viewpoint, and social context affect how durably identity is encoded into memory? In five experiments, we tested the automaticity and durability of identity encoding using a task designed to elicit attribute amnesia, a phenomenon in which people fail to report task-relevant stimulus attributes despite having just focused their attention on the stimulus (Chen & Wyble, 2015). In each experiment, participants repeatedly viewed grids of four faces and indicated the location of the face with a specific target attribute (e.g., finding an unhappy face among happy faces). On the critical trial, participants were suddenly asked about the identity of the target face before indicating its location. Those participants who fail at this surprise identification task exhibit attribute amnesia for identity. We found that despite high accuracy when locating the target unhappy face, participants’ performance fell to chance (~25%) when identifying the target face – even when the faces were highly familiar to the participants (i.e., celebrities). However, in a separate experiment, when participants were asked to locate the celebrity face among unfamiliar faces (rather than the unhappy face, as in the previous experiment), significantly more participants successfully identified the target face in the critical surprise trial. Therefore, mere familiarity does not lead to automatic identity encoding; familiarity must be task-relevant to boost identification. Similarly, manipulating the expression valence and viewpoint of the target face did not lead to significant improvement in identification performance. Overall, we found widespread attribute amnesia for face identity, which suggests that identity is not encoded durably into memory and may not be as automatically processed as it seems.

Decoding the distributed neural code of super-categories and fine-grained categories of objects via deep learning-based fMRI analysis
 

Is it possible to read out the visual content of the mind in a data driven fashion from whole brain
fMRI activation using deep learning? If so, does this generalize to different individuals, i.e., can
we train with one person’s brain activation in response to a visual stimulus and read out another
person’s mind? In this study we examined the distributed neural code of super-categories and
fine-grained categories of visual objects via deep learning-based fMRI analysis. We used the
publicly available BOLD5000 dataset (Chang et al., 2019), which contains whole-brain fMRI
responses of four participants to over 5,000 images showing real-world scenes and objects. We
curated a custom dataset based on BOLD5000 by annotating the super-categories face,
person, animal, furniture, vehicle, sports, food, outdoor, and tool. We further annotated several
fine-grained categories for selected super-categories; e.g., orange, banana, broccoli, and pizza
for the ‘food’ super-category. This resulted in a custom dataset of 2864 trials per subject for
2681 unique images (except S4 who had fewer trials) labeled for nine super-categories and 16
fine-grained categories. We trained a deep neural network model for each super-category
composed of several 3D convolutional and 3D max-pooling layers to classify the presence or
absence of a super-category given the whole-brain fMRI response as the input to the network.
This 3D model preserves the volumetric cortical topography of neural activity. We used 8-fold
cross-validation and cross-entropy loss to train each model. Model performance metric was
specified as the area under the receiver operating characteristic curve (AUC). We used transfer-
learning to train models for the fine-grained categories by fine-tuning the models that were pre-
trained for the relevant super-category. Finally, we used a deep learning visualization tool called
Integrated Gradient (Sundararajan et al., 2017) to map out cortical regions responsible for these
categories of visual objects. Our implementation can be found at
https://github.com/khademi/Neural-Decoding. Our models achieved significantly above chance
classification performance for all nine super categories for S1, S2 and S3 (and two out of nine
for S4) with AUCs ranging between 0.50 and 0.83, compared to a randomized model where
AUCs hovered around the chance level (0.5). For fine-grained analysis, AUCs were generally
lower, often ranging between 0.45 and 0.69, where 12 (S1), 2 (S2), 5 (S3) and 0 (S4) out of 16
classifications were significantly above chance. Classification performance was lower, but still
significantly above chance for 8 out of 9 super-categories, when a model is trained on one
participant and is tested on the other three participants with AUCs ranging between 0.52 and
0.64, compared to a randomized model where AUCs hovered around 0.5. Visualization results
reveal the cortical distribution of regions most relevant to each super-category and allow us to
constrain broadly construed hypotheses regarding the topological organization of the higher-
order visual cortex. Our results show that it is possible to read out the visual content of the mind
in a data-driven fashion from whole-brain fMRI activation using deep learning. These results
generalize across individuals to some extent. This suggests a similar topological organization
may underlie representations of high-level visual objects.