The Normans were the first group of people to build castles in England, although the Romans before them had built forts that the Normans then expanded and improved. Initially, castles were built out of wood, but eventually, people made castles from stone because they were stronger and lasted longer.