A retired colonel says plans to sell "the ancestral home" of the Australian Army is "mind-boggling", joining a chorus of prominent voices criticising the federal government's decision.

Victoria Barracks in Sydney's east has been an active Australian Army base for over a century.

But a recent independent audit found Defence's estate portfolio was too difficult to maintain, with underused and ageing facilities placing an increasing burden on resources.

As a result, the barracks is one of 67 underused or vacant Defence properties earmarked for sale by the federal government.

A rich military history

Victoria Barracks Sydney was established in 1840, with the main complex completed by 1848.

The Regency-style barracks sits on 13 hectares of land and was mostly constructed using local sandstone quarried by convicts.

The barracks, located in Paddington, were originally designed to accommodate 40 officers and over 600 enlisted men, with early designs featuring soldiers' quarters, hospital, military prison and more.

Various British Army regiments occupied the site until 1870, followed by the NSW colonial government, and later the Australian Army following federation in 1901.

Graham Fleeton is a retired colonel and the chairman of The Army Museum of New South Wales Foundation, which maintains and manages the museum located at Victoria Barracks.

He described the site as one of the best known examples of military architecture in Australia.

"All of the buildings have stayed exactly as they were built," he said.

Mr Fleeton said soldiers had left Victoria Barracks to fight in a series of wars Australia was involved in, such as the Sudan War, Boer War and World War I.

"Many left from here, not to return. Victoria Barracks will always be a significant reminder of the sacrifices made by Australian men and women," he said.

The site had a musical history too, with the Australian Army Band Sydney tracing its lineage back to 1874 as the resident military band at Victoria Barracks.

'Ancestral home of the Army'

Today, the site's purpose has evolved from its original role.

About 500 people currently occupy the site, including members of numerous army commands and other support units.

There is also office space, live-in accommodation, a site for cadet training and the Army Museum of NSW.

The Sydney Waratahs and Wallabies have also previously used Victoria Barracks as a training ground.

But the federal government now plans to sell Victoria Barracks, putting the museum's future location in question.

Former prime ministers Paul Keating and John Howard, NSW Premier Chris Minns, and Greens leader David Shoebridge have been among the vocal critics of the decision.

Defence sources told the ABC on Monday there were currently "people packing boxes", with the goal of having everyone moved out by Christmas.

Memorabilia, such as trophies, cups and insignia from the officers' and sergeants' messes, were being stored elsewhere as there was no clear place to put them.

Mr Fleeton said the site was the "ancestral home of the Army".

"They're telling the people they have to start moving out from November 17 into the Defence Plaza. I was at the meeting … You should have seen the look on the faces of people,"he said.

"It's mind-boggling … People right across the political spectrum don't want this site sold."

Under the current relocation plan, the majority of those personnel will be shifted to the Randwick Barracks, while others will be temporarily moved to Defence Plaza Sydney, a commercial office tower located 2 kilometres from the barracks in the CBD.

The Commonwealth rents the foreign-owned office tower.

Defence Plaza Sydney is understood to be sitting 60 per cent empty, leaving plenty of room for additional personnel.

Federal government stands strong on decision

Defence Minister Richard Marles defended the federal government's stance, saying it would take $200 million over the next decade to maintain the barracks as a Defence site.

The sale of the barracks and the other underused or vacant Defence properties is expected to generate up to $3 billion in revenue.

"Given the strategic landscape we face, we simply cannot afford to be wasting money," Mr Marles said at Monday's question time.

Prime Minister Anthony Albanese added: "There would be very few people who ever have access to what is a very large site in need of substantial Commonwealth investment just to be maintained."

Mr Albanese recommended the site be "utilised appropriately with full respect for the heritage of the site".

A future owner of Victoria Barracks would need to take into account the 12 Commonwealth Heritage Listings for the site.

Properties on the Commonwealth Heritage List have protections, which means heritage management options have to be considered, such as retaining heritage elements and adaptively reusing buildings.

A City of Sydney survey found the main priority for community members was that the heritage of the barracks was maintained.

There was also strong support for using the site for affordable housing and cultural spaces such museums, galleries, performance areas and artist studios.

In a statement, a Defence Department spokesperson said Victoria Barracks was "a significant heritage site".

"Defence acknowledges Victoria Barracks Sydney has served Australia proudly and holds strong historical, cultural and community significance, particularly for serving members and veterans," they said.

The spokesperson said Defence would be engaging with the museum trust on the future of the collection as part of the site divestment planning.

For Mr Fleeton, he felt passionately about preserving the barracks and the museum.

"We've got three Victoria Crosses there. We have a 5-foot bronze statue of a soldier holding a bayonet made by the same primary sculptor of the Anzac Memorial in Hyde Park, George Rayner Hoff," he said.

"Victoria Barracks is a jewel."