Under the new Crown ownership, the building was originally intended as a private retreat for King George III's wife, Queen Charlotte, and was accordingly known as The Queen's House. Re-modelling of the structure began in 1762.
