These are the fortified remains of the British administrative buildings.
Most of the slave-holding facillities were actually destroyed when the British finally outlawed slavery.