Australia’s 12th Closing the Gap report, published yesterday, shows by most socio-economic measures Indigenous Australians continue to lag behind the rest of the population.

Only two of seven targets – early education and Year 12 completion rates – are on track. On the five others – child mortality, school attendance, literacy and numeracy, employment and life expectancy – there has been little or no improvement.

Read more: 'Closing the Gap' process will better involve Indigenous Australians: Morrison

Federal Indigenous Affairs Minister Ken Wyatt says it’s another failure of Indigenous policy in Australia.

The Indigenous employment rate, for example, has improved by less than a percentage point in a decade – to 49%, compared with 75% for non-Indigenous Australians.

Cultural counterfactuals

One of the problems with government policies for Indigenous Australians is their lack of sensitivity to cultural differences. In particular, they fail to account for Indigenous notions of value.

As noted in a 2019 report from the Committee for Economic Development of Australia on how to break the cycle of entrenched disadvantage, there’s a lack of contact between Indigenous and non-Indigenous communities. This means public opinion and government policy evaluations are generally unaware there are different Indigenous notions of social value.

We call the adjustments that need to be made in policy and measurement to account for these different perceptions of value “cultural counterfactuals”.

Read more: Three reasons why the gaps between Indigenous and non-Indigenous Australians aren't closing

A good example of what this looks like can be found in Peak Hill, a small town between Parkes and Dubbo in New South Wales.

Here Maliyan Horizon, an Indigenous civil construction company, has been connecting people with jobs on country.

Creating sustainable careers for Aboriginal people in regional areas is a core business objective. The company does this through its commitment to training, upskilling and mentoring staff.

The company gives extra assistance, for example, to employees who have never before have had a full-time job. It also consults with local traditional owners to stay in touch with the needs of the community it works in.

Indigenous procurement policy

The federal government’s Indigenous procurement policy, on the other hand, is a good example of failing to account for cultural counterfactuals.

Since its introduction in 2015, this procurement policy has become one of the main government instruments to promote greater employment for Indigenous Australians. It requires all federal government agencies spend a percentage of their budgets with Indigenous enterprises.

According to the third-year evaluation of the policy by Deloitte Consulting, published in December 2019, the policy is a success by one key performance indicator – the number of Indigenous enterprises contracted to the Commonwealth government – and an “outstanding success” by the other – the number and value of contracts awarded to Indigenous enterprises. The report says:

Currently, Commonwealth procurement is exceeding the target by 4.1 percentage points (7.1% compared to 3.0%). In respect of providing genuine and sustainable economic opportunities for Indigenous businesses there is room for improvement as the above-mentioned 5.0% of total contracts issued equates to 1.5% of the value of those contracts.

But do Indigenous communities see it the same way?

IPP evaluation

Deloitte’s report has many good qualities. It makes useful recommendations. It doesn’t ignore problems like black cladding – where businesses employ Indigenous staff just to qualify for a government contract but have no commitment to providing stable long-term jobs.

But there is one fatal flaw.

Deloitte’s evaluation process involved focus groups with the owners and managers of Indigenous businesses. But the employees, the people supposed to be the beneficiaries of the policy, weren’t included.

This should be considered a major flaw in any policy evaluation. It assumes economic benefits from business will trickle down to communities. There is little evidence to prove it.

Read more: It's time we moved the goalposts on Indigenous policies, so they reflect Indigenous values

One of the evaluation’s recommendations is to use a measure called social return on investment (SROI) for contracts greater than A$4 million. This highly controversial idea attempts to reduce social policy impacts to a monetary value. It has been criticised for glossing over social and cultural complexities in its attempt to calculate a single financial value.

Towards more effective evaluations

Policies for Indigenous people should involve consulting Indigenous people to better reflect and prioritise Indigenous cultural knowledge and experiences. So should evaluations of those policies.

It’s crazy to not account for the perspectives of the people those policies are designed for.

Without more attention to cultural counterfactuals, Indigenous Australians will continue to be misrepresented. And Indigenous policies will continue to fail.

Martin Loosemore receives funding from the Australian Research Council

George Denny-Smith does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.