Can ideology-detecting algorithms catch online extremism before it takes hold?

Rohit Ram receives funding from the Defence Science and Technology Group (DSTG) and was supported by an Australian Government Research Training Program (RTP) Scholarship. Ideology has always been a critical element in understanding how we view the world, form opinions and make political decisions.

However, the internet has revolutionised the way opinions and ideologies spread, leading to new forms of online radicalisation. Far-right ideologies, which advocate for ultra-nationalism, racism and opposition to immigration and multiculturalism, have proliferated on social platforms.

These ideologies have strong links with violence and terrorism. In recent years, as much as 40% of the caseload of the Australian Security Intelligence Organisation (ASIO) was related to far-right extremism. This has declined, though, with the easing of COVID restrictions.

Detecting online radicalisation early could help prevent far-right ideology-motivated (and potentially violent) activity. To this end, we have developed a completely automatic system that can determine the ideology of social media users based on what they do online.

How it works

Our proposed pipeline is based on detecting the signals of ideology from people’s online behaviour.

There is no way to directly observe a person’s ideology. However, researchers can observe “ideological proxies” such as the use of political hashtags, retweeting politicians and following political parties.

Read more: How far-right online spaces use mainstream media to spread their ideology

But using ideological proxies requires a lot of work: you need experts to understand and label the relationships between proxies and ideology. This can be expensive and time-consuming.

What’s more, online behaviour and contexts change between countries and social platforms. They also shift rapidly over time. This means even more work to keep your ideological proxies up to date and relevant.

You are what you post

Our pipeline simplifies this process and makes it automatic. It has two main components: a “media proxy”, which determines ideology via links to media, and an “inference architecture”, which helps us determine the ideology of people who don’t post links to media.

The media proxy measures the ideological leaning of an account by tracking which media sites it posts links to. Posting links to Fox News would indicate someone is more likely to lean right, for example, while linking to the Guardian indicates a leftward tendency.

To categorise the media sites users link to, we took the left-right ratings for a wide range of news sites from two datasets (though many are available). One was based on a Reuters survey and the other curated by experts at Allsides.com.

This works well for people who post links to media sites. However, most people don’t do that very often. So what do we do about them?

Read more: COVID wasn't a 'bumper campaign' for right-wing extremists. But the threat from terror remains

That’s where the inference architecture comes in. In our pipeline, we determine how ideologically similar people are to one another with three measures: the kind of language they use, the hashtags they use, and the other users whose content they reshare.

Measuring similarity in hashtags and resharing is relatively straightforward, but such signals are not always available. Language use is the key: it is always present, and a known indicator of people’s latent psychological states.

Using machine-learning techniques we found that people with different ideologies use different kinds of language.

Right-leaning individuals tend to use moral language relating to vice (for example, harm, cheating, betrayal, subversion and degradation), as opposed to virtue (care, fairness, loyalty, authority and sanctity), more than left-leaning individuals. Far-right individuals use grievance language (involving violence, hate and paranoia) significantly more than moderates.

By detecting these signals of ideology, our pipeline can identify and understand the psychological and social characteristics of extreme individuals and communities.

What’s next?

The ideology detection pipeline could be a crucial tool for understanding the spread of far-right ideologies and preventing violence and terrorism. By detecting signals of ideology from user behaviour online, the pipeline serves as an early warning systems for extreme ideology-motivated activity. It can provide law enforcement with methods to flag users for investigation and intervene before radicalisation takes hold.