To summarize, when times were dangerous, and defense was important, knights, seigneurs, lords, tended to live in castles. They were not the most luxurious accommodations (though they got better over time), but they were generally larger and more luxurious than the other buildings around and about.