The origin of Sumoto-jo came from the Atagi clan who led Kishu/Kumano water army in order to bring Osaka Bay under their control. After that, Fuyuyasu, the 3rd son of the Miyoshi clan in Awa became an adopted child of the Atagi clan during the Tenmon era and the castle was placed under the control of the Miyoshi clan.
(Awaji Province／Hyogo Prefecture)
Fuyuyasu was assassinated by his elder brother, Nagayoshi, in the 7th year of Eiroku era (1564) and his first son, Nobuyasu, then, the second son, Kiyoyasu, took over his post. When Hideyoshi Hashiba conquered Awaji in the 9th year of Tensho era (1582) under the order of Nobunaga Oda, Kiyoyasu surrendered but his territory was confiscated anyway.
Hidehisa Sengoku became the castle ruler in the next year (1583) and the castle fully played its role as the front line against Shikoku. When the Shikoku conquest was completed in the 13th year of Tensho (1586), Hidehisa was transfered to Sanuki Province in Shikoku and Yasuharu Wakizaka became the castle ruler instead. The dry stone on top of the mountain which remains today was built during those days.
Yasuharu allied with the eastern forces in the Battle of Sekigahara and ruled Awaji under the Tokugawa administration, but was transfered to Iyo Ohzu in the 14th year of Keicho (1609) and the territory was put under the control of the Ikeda clan in Harima Himeji. The Ikeda clan abandoned Sumoto-jo and moved their base to Yura-jo.
In the 1st year of Genna era (1615), the Hachisuka clan became the ruler of Awaji and ruled there until the late Tokugawa shogunate after that. Yura-jo was abandoned in the 8th year of Kan'ei (1631) and Sumoto-jo was used again. However, the area on top of the mountain was left unused and architectures such as a residence were built at its foot.