This site was fortified by the 1100s, and from around 1200 it was adopted as the main residence of the bishops and archbishops of St Andrews. As such, the Castle became the principal administrative centre of the Scottish Church and was the setting for some of the key events in Scottish history.

Little of this early Castle can be traced through the existing ruins. Certainly almost nothing remains of the earliest structure which suffered badly during the Wars of Independence and was finally rendered indefensible by the Scots in 1337 to avoid it again being held by the English.

Completed in about 1400, the "new" castle was the work of Bishop Trail. With steep cliffs protecting it to the north and east, thick curtain walls and rock cut ditches on its landward side it was built to be easily defended.

As a residence, St Andrews Castle saw many notable visitors, including the young James I who visited in 1410. The castle also served as a strong and grim prison. An especially striking remnant of this role is the Bottle Dungeon, a bottle shaped pit dug 22ft down into the rock below the Sea Tower and accessible only via the narrow neck opening through a trap door from the floor of tower vault. Into this prisoners could simply be lowered - or dropped - and forgotten.