The castle stands at the end of a ridge with a sheer cliff on the eastern side. Haverfordwest castle was first established by Gilbert de Clare, earl of Pembroke in the mid-12th century, and remained an English stronghold throughout its history. The castle was first mentioned by Giraldus Cambrensis as one of the places he visited in 1188. The castle at that time, must have been of earth and timber construction of which little now survives, except, perhaps for the footings of a large square keep in the north-east corner of the inner ward.
The present form of the castle, probably reflects that of the original 12th-century castle.The medieval castle was converted to a prison in the 18th century, but the buildings of the inner ward and outer defences can still be appreciated.
Haverfordwest was probably a strong stone castle by 1220, when it withstood an attack by Llywelyn the Great who had already burned the town. It was acquired by Queen Eleanor (wife of Edward I) in 1289. Much of the existing masonry is late 13th-century in style and may well have been undertaken during the year before her death in 1290.
The inner ward has round towers on the north-west and south-west corners, while the south-east corner has a square tower with an additional projecting turret. The entrance lay on the west, protected by a gatehouse of which no trace survives. The remains of a spacious hall lie on the south, with large windows built high enough in the exterior wall to be safe from attack by besiegers equipped with scaling ladders. The south-west and south-east towers have three storeys, the latter with a basement equipped with a postern gate to allow access to a small terrace which could be used to counter-attack during a siege.
In the 14th century the castle was held by a series of owners, including Edward, the Black Prince, from 1359-67. In the hands of the crown from 1381-85, the castle was repaired. It was strong enough to repulse an attack in 1405 during Owain Glyndwr's war of Welsh independence. By the 16th century, however, the castle was derelict, but was hastily re-fortified during the Civil War. A story relates how in 1644 the nervous Royalists abandoned the castle, mistaking a herd of cows on a nearby hill for a Parliamentary army, thus allowing it to fall to Parliament without any resistance! It was later recaptured and held for the king for a year, but finally surrendered after the battle of Colby Moor, just to the west.
Situated within the Castle wall and previously the old prison governor’s house is the town museum. It houses exhibitions of art and local history and has artefacts relating to the town’s past.