A castle is fundamentally two things, a residence for a laird, and a place that offers significant protection. If it's only one or the other, that's problematic. A grand building with no defensive features would be a palace, chateau or country house.