About 300 to 400 demonstrators returned to the city on Wednesday, despite the city's COVID-19 lockdown and a warning from Victorian authorities.

What started as a protest against mandatory COVID-19 vaccinations for the construction sector and a closure of building site tea rooms has since turned into wider unrest.

Chanting "every day" from the shrine, hundreds of mostly men without masks, some still wearing high-visibility clothing like in days earlier, marched through the city to the war memorial.

Heavily armed police surrounded the shrine, with officers slowly moving in on the mob making arrests.

The stand-off lasted more than three hours as police tried to negotiate with protesters to peacefully exit via St Kilda Road.

By 4:30pm, some of the crowd dispersed but dozens remained behind and became rowdy before police fired what appeared to be rubber bullets.

A flare was thrown in retaliation as the riot squad cleared the crowd and took control of the site, which was left strewn with broken bottles and rubbish.

Deputy Commissioner Ross Guenther said more than 200 arrests were made over the course of the day, with vision showing many fleeing protesters being chased down by police in nearby streets.

"It was completely disrespectful that the crowd ended up at the shrine, which is such hallowed ground in this great city," he told reporters.

RSL Victoria also condemned protesters for relocating to the shrine, saying it is sacred to all people who have served and died in the armed forces.

"If any individuals or groups choose to express their political views, positions or ideological theories in the grounds of the shrine at any time, they are completely disrespecting the sanctity of this time-honoured space," it said.

Shrine of Remembrance chair Stephen Bowater called it "disgraceful".

Victoria Police was granted temporary restricted airspace over the Melbourne CBD by Australia's Civil Aviation Safety Authority on Wednesday, in order to protect officers' safety on the ground.

News choppers were grounded under the four-day order but will be allowed to take to the air in future, subject to approval and publishing footage on a 60-minute delay or after an operation has finished.

In addition, police were permitted to use crowd control force against those trying to repeat the seven-hour "cat and mouse" game seen on Tuesday, when up to 2000 protesters led police across the city and shut down the West Gate Bridge.

"I thought what we did today was very effective," Mr Guenther said.

Deputy Premier James Merlino refused to call those in the city protesters, instead describing the scenes as "a mob acting criminally" who were putting the Victorian public at risk.

The state government has shut down the construction industry for two weeks in metropolitan Melbourne, City of Ballarat, City of Greater Geelong, Surf Coast Shire and Mitchell Shire.

The protests began last week when construction workers were told they could not have breaks in tea rooms because of the risk of spreading coronavirus.

Getting a COVID-19 vaccine was made mandatory for the industry, prompting another protest in front of the CFMEU office on Monday.

Authorities say while there are construction workers in the crowds, there have been other groups including anti-lockdown activists dressed in high-visibility clothing.

Victoria recorded 628 new coronavirus cases and three more deaths on Wednesday, the highest daily tally in the current outbreak.

The regional city of Ballarat will also emerge from a seven-day lockdown at midnight on Wednesday, but strict rules remain, including masks outdoors and indoors and a ban on home visits.

Australian Associated Press