Why do we mix up faces? Game of Thrones might help us find the answer

In the Game of Thrones universe, confusing a photograph of actor Jack Gleeson, who played the popular HBO TV show’s despised sadist Joffrey Baratheon, for one of Maisie Williams, the beloved Arya Stark, is an egregious case of mistaken identity.

Admittedly, Gleeson is sporting dark hair rather than Joffrey’s customary blond, but even so. If nothing else, Gleeson is male and Williams female.

This is just one of the failures we have seen when we used characters from Game of Thrones to develop a new test of human face recognition. Thanks to the series, our research shows that becoming familiar with a face and reliably recognising a person are complex processes.

Facial recognition feels deceptively easy

Humans are a social species and recognising people’s faces is a crucial skill. We seem to do it effortlessly, but there are immense individual differences. Some people are practically “face-blind” and can’t identify people in their own family, while others, dubbed “super-recognisers”, claim they never forget a face.

The latter seem so infallible that security and law enforcement agencies are now seeking to employ them. But it is critical to understand their limits, too.

Read more: Super-recognisers accurately pick out a face in a crowd – but can this skill be taught?

As researchers, we noticed significant limitations in existing tests of face recognition. Some tests rely on people’s recognition of famous faces – politicians, athletes, actors, musicians. But people vary widely in how often they might encounter these people so it is difficult to know if mistaken identities are the result of poor face recognition ability, limited exposure or a mere lack of interest in one of these areas.

Other tests use strictly controlled faces of strangers – stripped of hair, glasses and other adornments – that people study in the lab. But this seems very unlike the way we become familiar with faces in the real world, where we encounter people in many different situations and without any real intent to study their facial features.

Which brings us to Game of Thrones. Fans of the show have been exposed to hundreds of characters over a period of years, while they aged, changed hairstyles (think Cersei Lannister) or became disfigured. One of us (Christel Devue) is supposedly good with faces, at least according to the standard lab-based tests, but noticed she kept mixing up similar-looking characters while watching the show.

Game of Thrones as a research tool

When we conducted our test, Game of Thrones had been running for six seasons and had featured many hundreds of actors. Because the show has such a dedicated following, the internet provided us with a wealth of information to develop a very well-controlled experiment.

For example, some fans had calculated how long each actor was visible on screen. As well as being able to gauge actors’ exposure levels, we also knew how much time had elapsed since characters were last seen (before facing a gruesome death).

We tested 32 participants who had watched all six seasons only once, as each one was released. This way, we were sure everyone had had the same exposure to all the actors and at about the same time. Participants had not read George RR Martin’s original novels, so would not know the characters from a different source.

We showed participants 90 headshots of actors (not in character) who had four different levels of exposure in the show (as main heroes, lead characters, support characters and bit parts), mixed with 90 strangers. They judged whether each face was familiar, rated their confidence in that judgement, and tried to identify and name the character or the actor.

Half of the participants were shown pictures in which actors’ headshots were similar to their character in the show, while the others saw pictures in which the actors’ appearance differed (for example, different hairstyle, facial hair, make-up, glasses).

Surprisingly, we discovered that simple changes in hairstyle or the mere passage of time throw off recognition of actors in all exposure levels, even those that participants saw repeatedly for years.

Read more: Combining the facial recognition decisions of humans and computers can prevent costly mistakes

Limitations in human face recognition

No one, including the best recognisers, recognised all faces. The best hit rate was about 80% of the actors, but from someone who also falsely recognised about 50% of strangers. Good recognisers were distinguished not because they recognised more faces than others, but by their ability to reject novel faces as unfamiliar.

Some participants were extremely confident while they were in fact reporting very inaccurate information. Others were uncertain about their responses but were spot-on most of the time. Some could not remember many names and gave really convoluted descriptions of characters.

Joffrey and Arya were not the only cross-gender mix-up. A photograph of a short-haired Sibel Kekilli (Tyrion Lannister’s mistress Shae) was identified by several participants as Isaac Hempstead Wright (Bran Stark). Such misidentifications and others were often based on superficial features like hair colour or style, facial hair or the shape of a head, showing how important these are.

Although recognition rates increased with greater levels of exposure, performance was far from what you’d expect of fans of a show. Identification errors occurred at all exposure levels. Only the most prominent actors were correctly identified and named more often than they were just recognised. As for bit parts, they were never correctly identified and only rarely recognised, even if their brief appearance was sometimes shattering (one of them was the cause of the Red Wedding). Familiarity judgements for these actors were barely higher than for strangers.

This research has important practical implications. Criminals know simple disguises make it possible to escape prosecution. Many innocent people are convicted based on errors in eyewitness testimony.

Our research confirms recognition based on brief encounters is likely to be wrong. Confidence of a witness in these conditions is not a good indicator of whom they have actually seen. Moreover, while super-recognisers have made useful contributions to law enforcement agencies, they are not infallible.

As for people planning to watch the final season of Game of Thrones, perhaps a refresher viewing is in order to avoid any confusion and ensure you really do know who’s who.