Construction began in early spring 1218 during the period of the Fifth Crusade by the Knights Templar, replacing the earlier castle of Le Destroit which was situated slightly back from the coast. The castle was built on a promontory, with two main walls cutting the citadel off from the land. The outer wall was approximately 15 metres high and 6 metres thick, with three square towers situated about 44 metres apart, projecting out by 12 metres with a level platform on the roof probably for artillery. In front ran a shallow ditch dug at sea level cut into the bedrock. The inner wall was approximately 30 metres high by 12 metres, with two square towers, the north and south each approximately 34 metres tall. As the inner wall was taller than the outer wall, defenders were able to shoot at targets over the first wall allowing greater protection from return fire by the besiegers. Part of the design of the castle included a protected harbour on the south side of the promontory. It also had three fresh water wells within its enclosure. The castle was capable of supporting up to 4000 troops during a siege, as it did in 1220. The settlement of Atlit developed outside the castle's outer wall and was later fortified. The castle's position dominated the north-south coastal route, and surrounding countryside allowing it to draw revenue from tolls and rents, going some way to pay for the running costs of the castle; as well as providing protection for pilgrims. The castle probably got its name from pilgrims who volunteered their labour during its construction.
The castle was under the control of the Knights Templar and was never taken in siege due to its excellent location and design and its ability to be resupplied by sea. It was besieged in 1220 by the Ayyubids, under the command of al-Malik al-Mu'azzam. It came under siege by the Mamluks under Sultan Baybars in 1265, during which the settlement of 'Atlit was destroyed. With the fall of Acre and collapse of the Kingdom of Jerusalem by the Mamluks under Sultan al-Ashraf Khalil, the Knights Templar lost their main roles of defence of the Holy land and security of pilgrims to the Holy Sites. The castle could now only be resupplied by sea, so the castle was evacuated between 3 and 14 August 1291, the last crusader outpost in the Holy Land.
Mamluke and Ottoman Period
The castle was not demolished by the Mamluks as was their normal practice after capturing a crusader fortification and remained in good condition until it suffered serve damage during an earthquake in 1837, and was also further damaged by Ibrahim Pasha in 1840, who used it as a source of stone for Acre.
British Mandate Period
A major excavation sponsored by Mandate Authorities was undertaken by C. N. Johns between 1930 and 1934. The castle was part of the area used by the Mandate Authorities to house illegal refugees during the later Mandate period.
State of Israel
The castle is now in a closed military zone which is a training area for IDF Naval commandos.