|The castle was built on the site of a Roman fort ( Verterae) and was one of the first stone castles to be built in England. It shows signs of typical Norman masonry as some of the walls exhibit herringbone patterns in the stonework. It was started in 1092 by William Rufus and was left in a poor state after being set fire to by the Scots under William the Lion in 1174. Theobald de Valoignes rebuilt the castle in the 1180's adding a four storey keep.|
Edward I stayed there in 1300 on his way to fight the Scots at Caelaverock Castle near Dumfries. Like a few castles in the area it passed to the Clifford family in 1269, and it was enlarged by Robert Clifford, who was responsible for the building of Clifford's Tower. In 1521 it was again badly damaged by fire during Christmas festivities, but later restored by Lady Anne Clifford.
When Lady Clifford died in 1629 the castle passed to her grandson Thomas Tufton, who used stone from the castle for other building work including repairs to Appleby Castle.
In 1920 it was "made safe" by the Ministry of Works and is now in the care of English Heritage.