A poll commissioned by the building unions shows a dive in support for Labor in the Hunter Valley.

The polling has prompted a senior CFMEU official to call on Labor to dump Mr Albanese as leader ahead of the next election, which could be held in the second half of the year.

CFMEU national political organiser Elizabeth Doidge believes Tanya Plibersek would be a better chance to lead Labor to victory.

"We definitely want a change of leadership," she told The Australian on Monday.

"I'm not necessarily saying it needs to be Plibersek but it definitely needs to be someone other than Anthony Albanese.

"He can't unite the party, he can't unite the unions, he doesn't have the leadership the party needs to win a federal election."

There is no love lost between the Labor leader and Ms Doidge's Victorian division of the construction union.

Its members are furious with Mr Albanese for forcing union boss John Setka out of the party.

Mr Albanese said people could see through their calls for him to be dumped.

"I think it says more about them than it does about me," he told 3AW radio.

"I've never heard of the official, frankly, and I've never heard of the company that did the alleged poll. I just had a bit of a chuckle by myself frankly, and don't take it all that seriously."

The poll was conducted by Chorus Consulting with New Zealand research firm Community Engagement in November last year.

It shows Labor on track to lose the federal seats of Shortland and Paterson.

Labor's primary vote in the coal mining electorate of Shortland has dropped 13 points since the last election to 28 per cent, trailing the Liberals on 45 per cent.

In nearby Paterson, Labor's primary vote has fallen to 30 per cent, compared with 43 per cent for the Liberals.

Outspoken figures from the party's right faction have long been concerned Labor is drifting away from traditional voters in heartland seats.

Labor MP Joel Fitzgibbon, from the seat of Hunter, quit the shadow cabinet last year over concerns about climate and energy policies.

Mr Fitzgibbon played down the prospect of switching leaders but argued the party needed to change.

"I am far more concerned with the message than the messenger," he told the Seven Network.

"But it is apparent that unless the Labor Party recalibrates its language, policy and narrative, then it's going to continue to struggle in the electorate and in particular, in traditional seats like those in the Hunter Valley and in central Queensland.

"So it's up to the party - we start thinking more about our traditional base, start putting labour back into the Labor Party, or think about losing more elections. A simple equation."

Australian Associated Press