A castle was first built on the site of Iwamura-jō in the Kamakura period, most likely around 1195 by Tōyama Kagetomo. The Tōyama clan continued to rule it until Takeda Shingen successfully laid siege to it in 1572. This provoked the wrath of Oda Nobunaga, as Tōyama Kageto (the last of the Tōyama line) was one of his retainers. Nobunaga retook the castle in 1575, shortly after the Takedas' crushing defeat at Nagashino. The castle then passed to Kawajiri Hidetaka, who rebuilt and expanded it. Iwamura-jō changed hands numerous times throughout the Edo era, then was decommissioned and dismantled following the Meiji Restoration. The castle remains in ruins apart from a gate and drum tower restored in 1990, but the picturesque site is popular for the views over the surrounding area, as well as its autumn foliage.