In the 19th Century Edward Hussey III built the house at the top of the hill and used the old castle below as a romantic folly.