Castles weren't just built for protection of the residents,as storage and/or military garrisons. They were very often strategically placed to guard the borders or nearby/outlying areas. Ignoring a castle was not difficult, but circumnavigating it only to later get attacked from the rear/flanks was.