AI is being used in social services – but we must make sure it doesn’t traumatise clients

Late last year, ChatGPT was used by a Victorian child protection worker to draft documents. In a glaring error, ChatGPT referred to a “doll” used for sexual purposes as an “age-appropriate toy”. Following this, the Victorian information commissioner banned the use of generative artificial intelligence (AI) in child protection.

Unfortunately, many harmful AI systems will not garner such public visibility. It’s crucial that people who use social services – such as employment, homelessness or domestic violence services – are aware they may be subject to AI. Additionally, service providers should be well informed about how to use AI safely.

Fortunately, emerging regulations and tools, such as our trauma-informed AI toolkit, can help to reduce AI harm.

How do social services use AI?

AI has captured global attention with promises of better service delivery. In a strained social services sector, AI promises to reduce backlogs, lower administrative burdens and allocate resources more effectively while enhancing services. It’s no surprise a range of social service providers are using AI in various ways.

Chatbots simulate human conversation with the use of voice, text or images. These programs are increasingly used for a range of tasks. For instance, they can provide mental health support or offer employment advice. They can also speed up data processing or help quickly create reports.

However, chatbots can easily produce harmful or inaccurate responses. For instance, the United States National Eating Disorders Association deployed the chatbot Tessa to support clients experiencing eating disorders. But it was quickly pulled offline when advocates flagged Tessa was providing harmful weight loss advice.

Recommender systems use AI to make personalised suggestions or options. These could include targeting job or rental ads, or educational material based on data available to service providers.

But recommender systems can be discriminatory, such as when LinkedIn showed more job ads to men than women. They can also reinforce existing anxieties. For instance, pregnant women have been recommended alarming pregnancy videos on social media.

Recognition systems classify data such as images or text to compare one dataset to another. These systems can complete many tasks, such as face matching to verify identity or transcribing voice to text.

Such systems can raise surveillance, privacy, inaccuracy and discrimination concerns. A homeless shelter in Canada stopped using facial recognition cameras because they risked privacy breaches – it’s difficult to obtain informed consent from mentally unwell or intoxicated people using the shelter.

Risk-assessment systems use AI to predict the likelihood of a specific outcome occurring. Many systems have been used to calculate the risk of child abuse, long-term unemployment, or tax and welfare fraud.

Often data used in these systems can recreate societal inequalities, causing harm to already-marginalised peoples. In one such case, a tool in the US used for identifying risk of child mistreatment unfairly targeted poor, black and biracial families and families with disabilities.

A Dutch risk assessment tool seeking to identify childcare benefits fraud was shut down for being racist, while an AI system in France faces similar accusations.

Read more: Algorithms that predict crime are watching – and judging us by the cards we've been dealt

The need for a trauma-informed approach

Concerningly, our research shows using AI in social services can cause or perpetuate trauma for the people who use the services.

The American Psychological Association defines trauma as an emotional response to a range of events, such as accidents, abuse or the death of a loved one. Broadly understood, trauma can be experienced at an individual or group level and be passed down through generations. Trauma experienced by First Nations people in Australia as a result of colonisation is an example of group trauma.

Between 57% and 75% of Australians experience at least one traumatic event in their lifetime.

Many social service providers have long adopted a trauma-informed approach. It prioritises trust, safety, choice, empowerment, transparency, and cultural, historical and gender-based considerations. A trauma-informed service provider understands the impact of trauma and recognises signs of trauma in users.

Service providers should be wary of abandoning these core principles despite the allure of the often hyped capabilities of AI.

Can social services use AI responsibly?

To reduce the risk of causing or perpetuating trauma, social service providers should carefully evaluate any AI system before using it.

For AI systems already in place, evaluation can help monitor their impact and ensure they are operating safely.

We have developed a trauma-informed AI assessment toolkit that helps service providers to assess the safety of their planned or current use of AI. The toolkit is based on the principles of trauma-informed care, case studies of AI harms, and design workshops with service providers. An online version of the toolkit is about to be piloted within organisations.

By posing a series of questions, the toolkit enables service providers to consider whether risks outweigh the benefits. For instance, is the AI system co-designed with users? Can users opt out of being subject to the AI system?

It guides service providers through a series of practical considerations to enhance the safe use of AI.

Social services do not have to avoid AI altogether. But social service providers and users should be aware of the risks of harm from AI – so they can intentionally shape AI for good.