The castle is thought to have been built by Alan, Earl of Richmond, soon after the Norman conquest of 1066 on the site of the old Roman fort, a date of 1087 indicating how strategic the location was to the Norman conquerors. It appears to have been involved in considerable conflict over the centuries with reports of it having been besieged as early as 1173 by King William of Scotland. It was apparently so badly damaged as to have required rebuilding in 1187. Around 1216 enemies of King John again besieged the castle, and again in 1322 in a regional feud between Henry Fitzhugh and the then Earl of Richmond.