Armytage House is situated on the Midland Highway at Bagdad, one of the earliest districts settled in Tasmania outside the major centres of Hobart and Launceston. Originally the home and seasonal migration route of the Palawa (aboriginal) people, the valley with its deep alluvial soils and cleared aspect drew the first colonists to formally settle the district around 1813.

George Armytage, a free settler, arrived in Hobart in 1816 and was granted 500 acres in the Bagdad valley in 1817. He planted grain and in the succeeding years built flour mills powered by water wheels from the Bagdad rivulet.

Armytage prospered and by the time he left for Victoria in 1847 had an estate (Milford) of 4000 acres and numerous buildings, many of which remain in their original stone and brick form. Armytage House built c.1830 as a store house for grain is the largest of the stone buildings remaining in close proximity to what is believed to be Armytage's principal residence now known as Milford manor.

Armytage House and its smaller neighbour were built of local sandstone and hand made bricks and mortar. Construction was done by two masons and was probably completed within a year. The building work would have involved elaborate scaffolding and pulleys to raise the sandstone blocks up to their resting place.

The buildings, with simple lines and consummate craftsmanship, remain a lasting legacy to the unknown masons who created them for George Armytage in the 1830's.