Henry Bennet constructed Arlington House on the site

When the improvident Goring defaulted on his rents Henry Bennet, 1st Earl of Arlington was able to purchase the lease of Goring House and he was occupying it when it burned down in 1674, following which he constructed Arlington House on the site—the location of the southern wing of today's palace—the next year. In 1698, John Sheffield, later the first Duke of Buckingham and Normanby, acquired the lease.