Castle walls could be plastered and whitewashed to protect the walls and mortar. The White Tower in the Tower of London is named for such a reason. Whether a castle would be 'white' would really depend on the plaster used and I'd expect to vary by custom from county to country and from age to age.