Anthony Albanese has moved to have John Setka, Victorian secretary of the Construction, Forestry, Maritime, Mining and Energy Union, expelled from the ALP after his derogatory remarks about anti-domestic violence campaigner Rosie Batty.

Setka was reported as telling a meeting of the CFMMEU national executive Batty’s work had led to men having fewer rights. He later claimed his comments had been taken out of context.

Albanese told a news conference: “John Setka does not belong in our party, because of the views that he holds”.

The opposition leader said at the next meeting of the party’s national executive he would move for Setka’s expulsion. Meanwhile, he had asked the national executive committee to suspend Setka immediately.

Setka attended last year’s ALP national conference, where he stormed out as former prime minister Kevin Rudd – who clashed with the union - was being awarded party life membership. Setka then called Rudd a “maggot”.

Former opposition leader Bill Shorten was close to the CFMMEU, sometimes relying on it for numbers.

Albanese spoke with Batty on Monday. She had indicated she was disappointed Setka’s comments distracted from the AO she was awarded in the Queen’s Birthday honours.

“Rosie Batty is a great campaigner against family violence and the idea that she should be denigrated by someone like John Setka is completely unacceptable to me as leader of the Australian Labor Party and I don’t want him in our party. It’s that simple,” Albanese said.

He said Setka had previously “made comments that I have said at the time are completely unacceptable”.

Attacking the ABCC and its head Stephen McBurney last year, Setka tweeted a photo of his son and daughter holding a sign saying “GO GET FU#KED”.

Albanese said Setka “undermines the credibility of the trade union movement through the position that he holds and the public views that he’s expressed”.

He said his action against Setka was not related to anything currently before the courts. Setka has said he would plead guilty to charges he faces of harassing a woman.

But, Albanese said, he did not want someone with views on family violence and the role of women that were so outside mainstream views “in the same party that I lead. It’s that simple.”

Albanese said he was confident he would get unanimous support from the national executive.

He avoided criticising the controversial CFMMEU more generally.

Michelle Grattan does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.