This castle was founded by Henry de Beaumont in 1107. Henry was a Norman Lord and the 1st Earl of Warwick. He acquired the Lordship of Gower in Wales around 1107 from the favour of King Henry I and subsequently built Swansea Castle. A few years later Henry returned to Normandy and entered the Abbey of Saint-Pierre de Préaux and became a monk. He died there on 20 June 1119. The castle Henry left behind was of timber construction but was rebuilt in stone, probably between 1221 and 1284 after it was unsuccessfully besieged in 1192 by Rhys ap Gruffydd. After a series of failed attacks, the castle fell in 1217 and was subsequently restored to the English in 1220.
Part of the interior of the castle, in particular the large motte, was demolished between 1909 and 1913 to make way for the construction of the newspaper office’s of the South Wales Daily Post which once employed the poet Dylan Thomas.