Around the world, the most informed and effective public servants are changing the way they develop policy and services. Australian public servants are eager to do the same, but their leaders are not supporting them to do so.

This is the key message from our new research on innovation in the public service, conducted for the Australian and New Zealand School of Government (ANZSOG).

With the final report of the independent review of the Australian Public Service by former Telstra CEO David Thodey expected to be published any day, Prime Minister Scott Morrison has signalled the need for a fundamental shake-up of the public service to improve the implementation of policies and strengthen accountability.

The prime minister is right to stress the performance of the public service. Concerns that it has lost capability and confidence in recent years are widespread, and important public outcomes are not being met.

Read more: Australia’s public servants: dedicated, highly trained ... and elitist

This struggle to deliver solutions to crucial public challenges, such as tackling greenhouse gas emissions or reducing Indigenous incarceration rates (the highest for any people in the world), helps explain why trust in Australian government is at an all-time low. Fewer than 41% of citizens are satisfied with the way democracy works, a decline from 86% in 2007.

To avoid making this worse, our leaders need to make sure they do not strip the public service of its ability to advise governments wisely and truthfully, and resist relying excessively on private sector consultants for solutions to public problems.

If Morrison wants a more effective public service, he instead needs to focus on investing in talent and letting the creative thinkers in the service advance innovative ideas that improve people’s lives.

Our governments also need to better understand public problems and how communities experience them – a challenge we can meet with better use of data-driven and participatory practices.

The findings of our research

Our report’s world-first survey of nearly 400 public servants shows that public servants are eager to become innovative public problem solvers, or what we call “public entrepreneurs”. Of the respondents who were not aware of specific innovation skills, for example, two out of three wanted to learn more about them.

The skills such public servants possess are specific and concrete. They know how to define actionable and specific problems. They consult data, and design effective solutions in partnership with the public. Above all, they recognise that we cannot be as smart alone as we are together, so they collaborate.

Our survey sets out nine specific skills - such as data anaytics, behaviour change and systems thinking - that make up a 21st century toolkit for public problem solving. We believe this toolkit is the strongest response yet to concerns about silo mentalities and risk aversion in the public service.

Read more: Grattan on Friday: Morrison can learn a lot from the public servants, but will he listen?

Such tools are powerful. For instance, the city of New Orleans used data analytics to reduce its murder rate by 20% in one year. However, we argue against over-hyping any one method, as they are most powerful when used in combination.

Along with this new mindset comes a need to rethink how these skills are taught. Our analysis of the best public sector training programs from around the world suggests a need for more online learning, flexible and self-paced formats, and hands-on coaching in problem-solving.

Another important change to implement here in Australia: learning how to work smaller and think bigger.

For instance, instead of researching and planning a project, policy or service from start to finish, some practitioners are breaking down big projects into smaller chunks. They develop projects incrementally and assess progress frequently, assessing the success of a small product before it is rolled out on a larger scale.

Public servants also need to make better use of digital technologies to amplify the impact of these new approaches. This enables conversations with the community about both problems and solutions, which leads to better-informed and more effective policy-making.

Innovations that are making a difference

To date, the adoption of these innovative ways of working in the public sector is not widespread in Australia.

The structures, incentives and cultures within the public sector are not set up to encourage innovation. In our interviews, middle management was pinpointed as a key blocker to innovation, while only 40% of respondents said that senior management was willing to take risks to support new ideas. The problem is pervasive.

But there are signs of hope, such as Australia’s early experiments with policy labs and mission-driven ways of working to tackle important problems.

For example, Victoria’s Transport Accident Commission and its many partners have employed a host of innovations to prevent injuries, save lives and optimise recovery for accident victims. These have included seat belt legislation, new road design, vehicle safety standards, case-managed rehabilitation, speed cameras and sophisticated advertising campaigns. Such initiatives have together reduced the road fatality rate in the state by 85% over the past 50 years.

Read more: Dealing with ministerial advisers: a practical guide for public servants

The scale of the changes needed is not small. Indeed, Australia may even require an innovation reset, through the creation of a new institution (like Nesta in the UK or Sitra in Finland) dedicated to advancing public problem-solving.

The consequences of inaction cannot be more serious. Although Australia’s public sector is generally well-functioning, there is a creeping crisis of effectiveness in the sector that must be addressed.

This radical re-imagining of the role of government and the public servant will be a defining challenge for the Thodey review and all Australian governments.

Professor Rod Glover has previously received funding from various governments for advice on economic development and innovation policy. He was on the staff of the Australian Prime Minister in 2008. He is a director of Save The Children Australia and the Centre for Evidence and Implementation, and Chairs the research committee of the think tank Per Capita. In his capacity as Interim Director of Monash Sustainable Development Institute at Monash University, he is also a Board member of ClimateWorks Australia and an Advisory Board Member of BehaviourWorks Australia.

Beth Noveck has received funding from ANZSOG. She is also director at Govlab and chief innovation officer for the state of New Jersey.