The Heron family acquired the castle following the death of Odinel de Ford in c1276 and had a dwelling erected here. By 1287 this was probably a two story tower house. Sir William Heron was granted licence to crenellate in 1338.

The Castle was captured and dismantled by the Scots in 1385 but was repaired by William Heron. He was imprisoned by his enemies at Newcastle in 1388 one of these enemies being Henry Lilburn who captured Ford and killed the garrison.