Beyond Reliability in First Impressions Research: Considering Validity and the Need to “Mix It Up With Folks”

Open Stimulus Databases for Reproducibility

One way to test the replicability and improve the accessibility of first impressions research is through sharing open stimuli. This can address the challenges that individual laboratory groups might have with the demands of producing large databases of controlled stimuli. Open-access databases can enable and democratise research by providing opportunities for smaller research groups with fewer resources to conduct high-quality studies.

Examples of good practice in this area include large open databases of faces (i.e. the ‘Chicago Face Database’; Ma et al., 2015) and voices (i.e. the ‘Jena Speaker Set’; Zäske et al., 2020). Such databases are of broad use; the Radboud Face Database (Langner et al., 2010), published in 2010, has been cited 2329 times at the time of writing. Making such stimulus databases open allows for collaborative expansions on these stimuli, such as with the later inclusion of multiracial targets (Ma et al., 2015). Additionally, other researchers can collaboratively provide more data on the stimuli themselves for the benefit of all those who use the set. For example, facial landmark templates for all images were later added to the Chicago Face Database (Singh et al., 2022) and ratings of targets on 19 trait dimensions are available for the Radboud Face Database (Jaeger, 2020).

Large International Collaborations

One of the most notable consequences of the credibility crisis in psychology is the development of large collaborative initiatives like the Psychological Science Accelerator (PSA). The PSA is a ‘distributed network of laboratories’ dedicated to coordinating large-scale research projects (Moshontz et al., 2018). The development of the PSA was motivated by using crowdsourcing techniques “to accelerate the accumulation of reliable and generalizable evidence in psychological science” (Moshontz et al., 2018, p. 503). Inspired by other ‘Many Labs’ projects which coordinate large-scale, crowdsourced, replications, the PSA aims to collect data from a diverse range of countries and contexts. The first project tested by the PSA collaboration was a first impressions project. They investigated the generalizability of the valence-dominance model of first impressions of faces (developed by Oosterhof & Todorov, 2008) across cultures (Jones et al., 2021). The multinational collaboration used an ethnically diverse set of faces and sampled 11,570 participants from 41 countries in 11 world regions. This work was an impressive undertaking and a strong response to the concerns that psychological research, as a whole, faces about its credibility. This project is a model of how to consider the complexities of international work with substantial effort dedicated to translation, ethics, and stimulus diversity. Whilst other PSA projects focus on other areas of psychology, it is a good sign for the field of first impressions that a project on rating photographs of faces was the test case project.

Replication and Reanalysis

The availability of study materials, open data, and the willingness to facilitate others’ replications of published findings is an important part of developing a more credible science. In recent years, many first impression researchers have attempted to replicate and extend published findings to test the robustness of claims (e.g., Caton et al., 2022; Kramer & Gardner, 2020). For example, Ert and colleagues (2016) found that more trustworthy-looking Airbnb hosts in Stockholm are able to charge higher prices for qualitatively similar apartments, presumably because they are favoured by guests. Jaeger and colleagues (2019) conducted a conceptual replication by analysing a larger sample of Airbnb hosts from New York City while also controlling for additional characteristics of the apartment or the host that could confound the relationship between facial appearance and prices.

Replication efforts have also been facilitated by an increasing focus on data sharing. The first PSA project (Jones et al., 2021) produced a large and rich data set on first impressions of faces that has already been used to test unrelated hypotheses in follow-up investigations. For example, Batres and Shiramizu (2022) used the data to show that the attractiveness ‘halo effect’ (association of more positive traits with more attractive individuals) not only emerges reliably in Western samples, which were primarily used in previous studies, but also in other cultures. These examples highlight that shared data and stimuli can be enormously helpful for a field.

Inadequate Stimulus Sampling and Diversity

Conversations about adequate stimulus sampling have a long history in social psychology (Brunswik, 1955). As Hammond noted in 1948: “psychologists maintain a one-sided emphasis on the need for representativeness. They emphasize representativeness of populations, but not situations, tests, or objects - thereby implicitly limiting the generalization of results obtained to the population, or subject, side.” (p. 531). Whilst there is no simple, constant, or adequate number of target people needed for person perception research (much like samples of participants, this can be addressed in a simulation-based power analysis—for a tutorial see Kumle et al., 2021) it is important for researchers to acknowledge that each individual target person brings important variability to perception data.

To this end, researchers promote the use of linear mixed models (and their variations) for social psychology researchers to better understand how much of an influence the random factors of stimulus person variability has on the target effects being studied. A key consideration in effectively understanding these variance components is the need to use larger numbers and more diverse samples of stimulus people for first impressions studies. It should also be noted that within-person variability (how a person appears across photographs, contexts, times) has an important impact on how they are perceived as well, and naturally varying presentations of each target person (a driver’s licence photograph versus a staff profile picture versus a social media profile picture) should be considered in stimulus databases.

There have been clear efforts to improve the ethnic, age, and gender diversity of stimulus people in studies (see the stimulus databases mentioned above) to try and expand the generalisability of findings. Foo and colleagues (2022) meta-analysed research on face-based trustworthiness impressions and noted that the majority of studies did not describe the ethnicity of the target faces or raters in their method sections. They note that there were “potential biases, including a preponderance of Western studies, a lack of “cross-talk” between research groups, and clarity issues.” More specifically Cook and Over (2021) highlighted how most first impressions research is focused on White faces. They identified two parallel literatures on perceptions of faces: a considerable literature on investigating elements of face shape variance within White faces, and separately research with non-White faces—which are typically studied in contrast to White faces. Cook and Over observed that first impression papers “typically offer little or no justification for their use of ethnically homogeneous White face stimuli” (p. 5). It is important that first impressions researchers justify their stimulus choice and consider general trends in social psychology to focus more on diversity and inclusion. We should also consider different dimension of diversity that are important, such as neurodiversity, sexuality diversity, and social diversity.

It should also be noted that the way in which researchers instruct their target people can also restrict their appearance to perceivers. For example, when considering facial expressions of emotion, Barrett and colleagues (2019) reviewed the emotion recognition literature and found that most studies use prototypical facial expressions of core emotions. These stimuli involve extreme forms of target facial expression from a range of emotion categories chosen by the researchers. People posing as stimuli in these studies are encouraged to model these atypical presentations of emotion. As Durán and Fernández-Dols (2021) noted in their meta-analysis, in expression-generation studies, the prototypical expressions do not reliably occur with the intended emotional experience. Such strong forms of emotion expression are not typical in everyday experience. Further, in an analysis of spontaneous emotion footage (from 6.1 million open source videos from YouTube), variance in emotional expression was better explained by 16 dimensions rather than six (e.g., Cowen et al., 2021). Barrett and colleagues (2019) provide a thorough review of the methodological limitations in judgments of facial expressions of emotion, but in general, this area of research provides a salient example of how researchers may constrain their stimuli due to theoretical models. A field norm of six primary emotions leads to coached, exaggerated presentations of people for perceiving, which may poorly represent naturally occurring diversity in emotional expression. A useful example of how stimulus creation can have an impact on our investigation of everyday contexts.

Constrained Participant Responses Options (2AFC and Rating Scales)

First impression studies not only constrain the types of stimuli participants are exposed to, but also how participants can respond to the stimuli. For example, first impressions are often measured using a two-alternative forced-choice (2AFC) design. This involves showing two versions of the same stimulus, such as faces, bodies, or voice recordings, next to each other, and perceivers are instructed to directly compare which version scores higher on some dimension of interest (e.g., Alper et al., 2021). This design, which we refer to as the ‘evil twin paradigm’, is often used to study a specific characteristic in impression formation. For example, two versions of face images are created where the facial width-to-height ratio (fWHR) is reduced in one version and increased in the other version (Stirrat & Perrett, 2010). Participants view the two face versions and choose which one appears more trustworthy. If participants identify the face versions with lower fWHR as more trustworthy, then this is taken as evidence that people rely on fWHR when forming trustworthiness impressions. However, this inference is problematic because it is possible that perceivers are forced to rely on the manipulated facial characteristic when this is the only cue they have at their disposal (A. Jones & Jaeger, 2019). This is not necessarily what they would rely on in everyday life when they also have access to an array of other cues. The paradigm also highlights differences in facial appearance that may go completely unnoticed in everyday impression formation. For instance, fWHR may only be related to trustworthiness impressions when targets’ gender, age, ethnicity, facial expression, and other features are kept constant (as is the case in evil twin studies), but not when these dimensions vary (as is the case in everyday life; Jaeger & Jones, 2022). In this way, constrained response options can introduce false positive results (see Bovet et al., 2022).

Likert-style rating scales allow participants to give more nuanced responses to stimuli, but they still constrain how participants can react to stimuli. That is, ratings that are assessed (e.g., trustworthiness impressions) may not reflect people’s dominant reaction to a person in everyday life when their response options are not constrained (e.g., “this seems like a friendly person” such as in Sutherland et al., 2018).

Limited Consideration of Context

The most common experimental design in first impressions research asks a participant to pay close attention to a series of faces, bodies, or voices, and to form a judgement of them. But this focused observation is very different to the noisiness of everyday life when observations of others might be incidental, in passing, or occur in specific contexts. It could well be the case that particular parts of a whole person (such as their eyes or height), which we might consider to be important from a theoretical standpoint, might turn out to be less important when studied in a complex context. Recent work demonstrated that during the observation of a social interaction in naturalistic scenes, just 14% of participant eye gaze patterns were to the face of the social interactors (Varela et al., 2023). These results highlight how researchers’ assumptions about aspects of people that we might consider to be highly relevant—such as faces, which receive significant dedicated study in first impressions—might not be highly relevant in a noisy everyday environment.

Researchers rarely sample situations to the same extent that they sample participants or stimuli. Representative design approaches have long noted that different environments should also be considered for the impact they have on study findings (Brunswik, 1955). More than treating the laboratory environment as a fixed neutral experience, it is an environment of specific constraints. One controlled environment might tell us less about first impressions compared to sampling across multiple natural contexts. These settings elicit unique, specific behaviours. For example, there might be more to learn from seeing which faces consistently draw the attention of a participant in a restaurant, in a lecture hall, and on a street compared to how they might form a first impression of a face on a computer in a quiet, neutral room. Sampling across contexts and quantifying their variability is of significant theoretical value when the alternative is passive observation in a fixed environment.

Relatedly, there is limited investigation of classic first impressions questions in contexts with sufficient stakes to understand how important these findings are in practice. Consider a job interview, where the applicants might present considerable variability in job experience, expertise, and qualifications—as well as appearance, gender, and ethnicity. Despite research suggesting that perceptions of competency and related factors are influenced by facial appearance, limited research has explored how these factors explain concrete decisions, such as hiring, in complex settings (c.f. papers such as Carlsson & Eriksson, 2019). Understanding the variance explained by first impressions aspects in job applications or interviews is an important step in making our research more relevant to target contexts.

Reductionism and Atomisation of Target People

There are two main reasons to critique reductionist or atomised stimuli in first impressions research. The first is from a statistical covariance perspective, where it is important to consider that parts of a person are likely correlated. The second is from a theory perspective where we recognise classic Gestalt perception theory and how wholes are not equal to the sum of their parts.

Over/Underestimation of Everyday Experiences

The issue with methodologies that do not effectively emulate everyday experiences is that they might both over- and under-estimate the importance of key social features in everyday interaction. We would encourage readers to think beyond experimental control and replication, to the validity of the task in hand. For example, there may be parts of a person that are difficult to perceive or rarely attended to in noisy everyday situations, but these rise to prominence in experimental designs where they do affect first impressions when they are the only thing a participant sees. When participants only see a face or an eye, the information available to them to make a judgment is not a valid representation of the process of forming a first impression with a whole person before them. Eye colour might be important in a limited presentation of a person (where only the eye region is visible to the participant), but a person’s eye colour may have little influence on how we might form a first impression when that person is wearing an absurd hat or doing an unusual dance.

The relevance for atomised presentations of people in understanding noisy, information-rich, context-laden everyday situations, like a job interview, first date, or legal process, is extremely limited. However, the concern that laboratory research does not represent the effects in the wild is not just to claim that the effects might be artificially increased in experiments. It could well be the case that, within the range of all things going on in a high-stakes interaction, eye contact or face qualities or voice tone might play an oversized role in shaping first impressions. Perhaps an atypical voice outweighs the presence of an unusual hat in everyday life, but we can only know this through studying this in the presence of competing sources of variance. In order to build stronger, functional, theory, we should consider the value of holistic experiences of other people.

Misunderstanding Social Differences in Autism and Mental Health

Excluding interpersonal dynamics from first impressions research can also alter our research on diagnoses characterised by social behaviour differences. When the materials and methods used to build models of social differences are not based on social interactions, we might misunderstand differential responses as ‘social deficits’. Many of these lines of research are motivated by noticeable everyday challenges that some people experience, yet the selection of methodologies used to understand these differences might exclude or be misleading about the nature of these difficulties.

For example, in the case of autism, evidence for the most influential theory—that autism is resultant from the theory of mind deficits—largely rests on a widely used test that requires participants to match researcher-chosen words to photographs of magazine models’ eyes presenting an emotion, the ‘mind in the eyes test’ (Baron-Cohen et al., 1997). Similarly, the more recent ‘reduced social motivation’ theory of autism (Chevallier et al., 2012) also rests on evidence showing reduced social attention in autism to faces and eye regions of photographs (e.g., Chita-Tegmark, 2016). Yet, there are questions about how observations of photographs are comparable to being social with others. We should note that the ‘mind in the eyes’ test suffers from replication and measurement issues (see Gernsbacher & Yergeau, 2019), but more importantly, there are questions about the extent to which atomised images of actors presenting an emotion represent an everyday ‘mind reading’ activity. Therefore, we should question how ‘low scores’ on this task correspond to a social deficit. Critiques of these methods are supported by more recent studies including interacting social partners, which consistently fail to find differences between autistic and non-autistic participants in social attention patterns (see Kikuchi et al., 2022). In fact, many of these studies show that the differences present in autism are the result of subtle interpersonal dynamics such as the topic of the conversation (e.g., Hutchins & Brien, 2016), rather than a general lack of social motivation.

This leads to broader questions about methods that use photographs or digital renders of faces to gain insight into social differences across human neurodiversity. A variety of meta-analyses and systematic reviews highlight how a common approach to understanding social challenges starts with presenting images of emotional states—for example, in research on antisocial/dissocial personality disorder and psychopathy (Marsden et al., 2019), social phobia and Post Traumatic Stress Disorder (Plana et al., 2014), and in attention, eating, and anxiety disorders in child psychiatry (Collin et al., 2013). Many of these reviews do find differences between diagnoses in responses to faces, and these differences are of note in trying to understand these topics. However, more work is needed to understand how face processing relates to the complexity of everyday social encounters—especially when many of these diagnoses are related to a variety of dynamic interpersonal behaviours, not just facial image processing. It is important for future research to consider how much variance these aspects of first impressions matter for our studies of neurodiversity.

Codifying, Not Controlling, Interactions

As mentioned above, the atomised stimulus person, presented without context or dynamics, may not represent what many might experience as a ‘first impression’. It is important to consider how these first encounters with others—especially in settings of high stakes like job interviews, first dates, and legal interactions—are shaped by the complex environment and context of the interaction. There are frameworks that enable science in this way. Egon Brunswik famously proposed ‘lens model’ approaches to perception activities (Brunswik, 1955) and this has been applied by some researchers of first impressions (i.e., Nestler & Back, 2013). In these frameworks, participants are invited to judge another person in a context. Then aspects of the ecology and perceptual targets are coded. The naturally occurring variability between targets in the ecology reflects the different ways the targets provide information for the perceiver. In a general example, one might consider trying to judge whether someone in a store is an employee or a fellow customer. The other people in the store will vary on many aspects, for example, wearing a uniform, who they are with, or their age. Judgments of whether someone is an employee will be a consequence of how a perceiver considers all these elements and more. Yet amongst the ecology, a uniform may be the only genuine cue as to whether someone is an employee or not (in Brunswik’s terminology, an ecologically valid cue). However, other features such as being alone or being young might bias a perceiver into making an inaccurate judgment. Thus we can test the relationship between participant perceptions (perceived as employee) and the stimulus aspects (uniform, company, age) and the relationship between stimulus aspects and the stimulus qualities (is an employee) which allows us to identify: 1) useful cues (relating to both perceptions and target qualities), 2) irrelevant cues (neither relating to perceptions nor target qualities), 3) missed cues (ecologically valid cues which do not relate to perceptions) and 4) biases (cues which influence perceptions but are not ecologically valid). This framework allows the study of accuracy and bias in everyday contexts, based on the natural variation between people and stimuli in places.

In the context of first impressions, when asking participants to consider their judgments of new people, we might consider coding many different aspects of our targets and investigate how the different cues from a person might hold more or less attention from our participants. Moreover, in cases where we test judgment ‘accuracy’ (i.e., self-other agreement or task-other agreement) we can evaluate what cues might afford accurate perceptions, missed cues, and biases in interactions. This framework of coding and testing cues outside our primary cues of interest is important as it will identify how relevant our cues of interest are in a noisier, wider environment. Perhaps in everyday experience, the cacophony of other information washes out the explanatory variance of our cues of interest, as participants are biased towards other sources of information. Such methodologies can be standardised enough to be compared across contexts and countries for generalisability (similar spaces, same coding schemes). The coding books and final analysis code can be shared openly and transparently. With participant consent, the recordings of interactions could be stored and used in wider research activities for cross-laboratory coding and reliability checking. Whilst there has been great progress in terms of controlled experimental reliability in the field of first impressions, there are other ways of doing reliability checks on less controlled experiments.

Previously, researchers have used lens models and less controlled (‘zero acquaintance') interaction settings in their first impressions work, but these examples are the exception rather than the rule in the literature. As Nestler and Back (2013) highlight, deconstructing the lens of first impressions enables us to understand how different types of interpersonal judgments might be more or less accurate by identifying relevant cues. They note how, for example, first impressions of extraversion are more accurate than agreeableness, perhaps because the former traits have more behavioural manifestations identified in lens modelling. Many studies using round robin or dyadic interactions stem from influential work by Albright and colleagues (1988). They noted, almost 35 years ago, much as we do here, that “Using photographs increases experimental control over the information available to the perceiver (i.e., behavioral cues), but may have limited generalizability. Actual physical presence, conversely, decreases control over available information, but is certainly representative of the process of making judgments of strangers” (p. 388). Dynamic studies of first impressions are possible and have been conducted with some impressively large sample sizes—such as Tissera et al. (2021), who studied nearly 8000 dyads’ first interactions—however, such methods are not yet common throughout most of the literature on ‘first impressions’. In general, we advocate for this approach to be adopted more widely in light of trying to improve validity in a field which has made considerable efforts to improve reliability.

Inductive, Participant-Led, Approaches to First Impressions

As a way of developing more inductive approaches to understanding first impressions, some recent work has shifted to participant- or data-led methods. These move from researcher-defined key aspects of a person to study to using participants’ responses to guide identification of key aspects of a target person in data. For example, recent work that showed that the traits of symmetry, averageness, and dimorphism—all mostly studied with atomised presentations—showed little predictive capability of attractiveness perceptions (Jones & Jaeger, 2019). Holzleitner and colleagues (2019) used a data-driven approach to explaining variance in facial attractiveness judgments by using principal components analysis to map sources of variability from key marker points (i.e. corners of eyes, tops of lips, sides of nose, etc) across photographs of faces. They uncovered new parameters that explained greater variance in attractiveness than typical deductive aspects of interest mentioned above.

There are more opportunities for being participant-led when collecting judgment data from participants as well. To collect participants’ responses to stimulus people, researchers usually give a forced choice or forced spectrum of responses against key anchors of interest, for example, participants are asked to consider how ‘attractive’ or ‘threatening’ the stimulus person is. However, it may not be the case that participants’ first thoughts on seeing a person are related to these chosen adjectives. As such, research is losing out on key judgments that are important in everyday life. For example, fundamental models of facial first impressions, like the valence-dominance model (Oosterhof & Todorov, 2008), are derived from ratings provided by participants on a series of 14 core traits (e.g., trustworthiness, dominance, unhappy, confident) that are used across many studies. However, C. Lin and colleagues (2021) showed that by using a wider range of trait ratings, significant differences in the key “structure” of impressions emerged, such that four correlated components appeared, indexing traits like youth, age, warmth, and femininity. With advances in text processing techniques, it is also becoming increasingly possible to reproducibly manage data generated from unconstrained responses from participants. This may help develop more theory around an underlying “topic” structure of first impressions that may yield further insights into the underlying psychology.

Moreover, there are opportunities to use more inductive approaches to designing methodological settings and frameworks for first impressions tasks. This would involve rethinking method development by asking focus groups to reflect on what settings, judgments, and dynamics are important for them. This can allow us to move away from a researcher-led approach to designing research that is more data-rich. Whereas a researcher might think judgments of certain adjectives made on observing static faces are important, participants might express particular behaviours or movements as important. Stakeholder- or patient-informed designs are increasingly popular in applied research like forensic and health psychology (see dosReis et al., 2020), and these tools may help further develop first impressions research.

Theory Development and ‘Can’ vs. ‘Do’ Questions

We can broadly conceptualize research as asking two classes of questions—‘Can’ and ‘Do’ questions. This dichotomy is more useful than a general consideration of what is ‘lab’ or ‘field’ work as it focuses on the nature of questions we are trying to answer. ‘Can’ questions ask if it is possible for an effect to occur. Can people make first impressions of others in < 100ms (if that is what we provide them with)? Can people agree on ratings of attractiveness if only given a voice clip (if that is what we provide them with)? Can people use photographs of faces to accurately identify aggression in a target person (if provided only with a face)? These questions increase our knowledge base, but in a specific way. These controlled studies do all they can to maximise the chance of an effect occurring by limiting statistical error (experimental control, large sample size, high target number, etc.). This tells us, all things being equal these effects can occur.

In contrast, we can ask ‘do’ questions. Do people make first impressions of others in < 100ms (in everyday experience)? Do people use voices to decide how attractive a person is (in everyday experience)? Do people use faces to accurately detect how aggressive a person is (in everyday experience)? These are questions for which we do not have much empirical evidence. Given that these questions are often those that lay people and practitioners ask of us when we are applying our research—Does this bias our job selection process? Does this affect how my first date goes? Does this lead to a miscarriage of justice?—it should follow that researchers would start by asking ‘do’ questions before leading to ‘can’ questions. Rather than starting from the assumptions of ‘can’ questions and iteratively chipping away at larger wholes, it could improve the theoretical gains of our research to conduct reproducible ‘do’ studies first, to pivot to ‘can’ questions. This matters for sharpening up the resource use of the field. Higher quality, reliable research is more resource intensive. Effective selection of ‘can’ questions through asking broader ‘do’ questions could greatly improve the efficiency of our research questions and theory building.