The first castle in Vitré was built of wood on a feudal motte around the year 1000 on the Sainte-Croix hill.
The castle was burned down on several occasions,
and eventually was bequeathed to the Benedictine monks of Marmoutier Abbey.

The first stone castle was built by the baron Robert I of Vitré at the end of the 11th century.
The defensive site chosen, a rocky promontory, dominated the valley of the Vilaine.
A Romanesque style doorway still survives from this building.
During the first half of the 13th century, baron André III,
rebuilt it in its present triangular form,
following the contours of the rocks, surrounded with dry moats.