A Sengoku-period castle, Tahara-jō was first built in 1480 by Toda Munemitsu. At the time, the castle sat at the edge of Mikawa Bay and was the main seat of power on Atsumi Peninsula. Allied with the Imagawa clan against the Matsudaira, the Toda clan eventually fell under sway of the Tokugawa clan. After the siege of Odawara Castle in 1590, Toyotomi Hideyoshi stripped the Toda of their holdings in the Mikawa area, handing the castle to one of his vassals, who rebuilt and expanded the castle defenses. The Toda were reinstated by the Tokugawa Shogunate following its establishment in 1603, but were moved to Bungo Province in modern-day Kyūshū from 1664. Tahara-jō then came under the control of Miyake clan, which ruled over it until the Meiji Restoration. All the castle buildings were destroyed in 1872, with the grounds largely taken over by a shrine. Replicas of a feudal-era gate and yagura (tower) were built in 1990 to accompany the neighboring Tahara City Museum.