Indigenous programs cost billions – but we know surprisingly little about what works

Billions of dollars are spent annually on Indigenous programs and services. Yet we know little about which programs are effective, and often struggle to understand their impact. That’s why evaluating programs is crucial.

We need to know what worked, what didn’t, and why.

The Productivity Commission has called for “more and better” evaluations of Indigenous programs, meaning evaluation processes that engage Indigenous communities, organisations and leaders.

So, what do best-practice program evaluations look like?

To find out, colleagues and I looked at how governments and non-government organisations commission evaluations of programs aimed at boosting Indigenous health and wellbeing. We wanted to know what kinds of evaluation commissioning practices would support Indigenous engagement and leadership.

Our paper, published this week in First Nations Health and Wellbeing – The Lowitja Journal, found it’s vital to explicitly embed Indigenous values from the moment an evaluation is commissioned.

Simply “consulting” Indigenous people later in the evaluation process after the evaluation has been designed – or having no meaningful Indigenous involvement at all – risks yielding evaluation results that don’t actually help, waste time and money, and may ultimately lead to more death and illness among First Nations people.

What we did and what we found

Our project included a comprehensive scoping review, where we analysed 39 peer reviewed and grey literature documents (meaning documents produced by government, academics, business and industry) from Australia and New Zealand, Canada and the US. The documents were mostly from Australia and New Zealand.

We identified five main ways these evaluations are commissioned:

1. Indigenous-led models

This is where the evaluation is commissioned by and for an Indigenous community-controlled organisation. All engagement in the evaluation is overseen by an Indigenous organisation.

2. Delegative models

This is where the person or organisation commissioning the review – the “commissioner” – requests an evaluation. However, the commissioner delegates responsibility and funding to an Indigenous-led service provider.

3. Co-designed models

This is where the person or organisation commissioning the evaluation works with the Indigenous service providers to design the evaluation, and each has equal power in the decision-making process.

4. Participatory models

This is where Indigenous people may be involved in the evaluation to varying degrees, from tokenistic participation to active engagement. However, the power to make decisions rests with the non-Indigenous person or organisation that commissioned the evaluation.

5. Top-down models

This is where the non-Indigenous person or organisation commissioning the review has all the power and places no emphasis on Indigenous people’s engagement.

An evaluation could fall into one or more of these categories at different points in the process.

3 good models

We identified three of these five models – Indigenous-led, delegative, co-design – as good practices.

They achieved outcomes that:

• were culturally safe (meaning they respected Indigenous people’s rights, cultures and traditions)
• met the the needs of service providers and commissioners
• provided insights that were actually useful for Indigenous communities.

Some service providers have extensive experience and can commission evaluations themselves.

Others, however, have limited capability and would value input from the person or organisation commissioning the evaluation. This could supplement their staff skills.

For evaluations involving multiple service providers, a co-design model may be used instead.

What does good practice look like?

One example we looked at showed how Indigenous health program evaluations can be done well.

The Healing Foundation, an Aboriginal and Torres Strait Islander organisation that supports members of the Stolen Generation and their families, funded three organisations to deliver services to their local communities.

The Healing Foundation contracted a non-Indigenous evaluation organisation, Social Compass, to evaluate the programs. They made sure local people were engaged in the evaluation design and conduct.

Guiding the evaluation was a “knowledge circle” that included Aboriginal men from the three communities in which the program was being developed.

Community and relevant government and non-government agencies in these three communities were also involved in the evaluation. However, the power was maintained by the Healing Foundation to ensure the program and evaluation were culturally appropriate.

Top-down doesn’t work

Not all evaluations are done so well, unfortunately.

The top-down approach, due to its lack of Indigenous engagement, emerged as the worst-performing model. It risks wasting time and money for little practical benefit.

Without Indigenous engagement, and ideally self-determination, in the evaluation process, evaluation findings would be of little value to organisations providing services to First Nations people.

More importantly, the evaluations would likely be conducted in a culturally unsafe way, causing potential harm.

For example, not engaging Indigenous people means the evaluation could focus on the wrong questions for key communities, rendering the findings useless.

And if the right Indigenous people are not engaged from the start, it might damage relationships between the service provider and commissioner. Indigenous service providers may choose not to engage with the project at all, making it hard or impossible to collect data needed for a good evaluation.

This doesn’t just waste time and money, including taxpayer dollars. It also means that, due to a lack of good information to inform policy, First Nations people will continue to be sicker and die younger than other Australians.

Evaluations matter

Our research comes as Indigenous leaders are calling for opportunities to influence the evaluation decision-making processes.

If we are to have any hope of closing the gap, our research suggest First Nations people should be meaningfully involved in evaluating what worked and what didn’t about Indigenous-focused programs.

The author would like to acknowledge the other authors on the paper: Jenni Judd (CQU), James A. Smith (Flinders), Helen Simpson (UOW), Bronwyn Fredericks (UQ), Amohia Boulton (Whakauae Research Services), Yvette Roe (CDU), Janaya Pender (Lowitja Institute), Sophie Kerrigan (UOW), Anna Temby (UOW), Melissa Opozda (Flinders) and Margaret Cargo (Flinders).