Castleton is built on a ridge of high ground overlooking the River Esk. It takes its name from the castle that Robert de Brus built when he was granted lands by the king at the end of the 11th century. He later acquired further lands in the North East and south west Scotland becoming the 1st Lord of Annandale and establishing a dynasty that produced a king, Robert the Bruce of Scotland.
But at Castleton he was a mere Baron. The Reverend Atkinson in his book Forty Years in a Moorland Parish recalls an old story that the baron “came in with the Conqueror”. As he is generally thought nowadays to have been born about 1070 if this story is true it must have been as as a twinkle in his father’s eye.
De Brus’s stay in Castleton must have been quite short for he next built Skelton Castle. This was a strategically better location that overlooked the lowest fording of Skelton Beck where the Baron could control the transport of fish and salt from the coast south. He was also a monastic founder, establishing Gisborough Priory.