William the Conqueror created the lordship of Holderness, a vast area from the Humber estuary to Bridlington. He gave it to Drogo de la Beauvriére, who had fought alongside him at the Battle of Hastings in 1066. The king needed a trusted follower there to control the area and the adjacent coastline.
Drogo built Skipsea castle as a principal residence and his successors, the counts of Aumale, held the seat for the next 130 years. In the 12th century, Count William le Gros created a fortified settlement or 'borough' possibly located on the ridge, where houses and shops fronted onto a roadway. It provided him with an income from tolls and rents. Use of the castle declined from about 1200 when the lords of Holderness moved their administrative residence to Burstwick. The castle was ordered to be destroyed in 1221 when Count William de Forz II rebelled against the king. By 1350, the castle was being leased for pasture and the borough was abandoned.