Inside the Walls. The outer wall of a castle was called the Bailey. Inside the Bailey were buildings where the lord of the castle's cattle, horses and servants lived. Some of the soldiers needed to defend the castle might live in part of the gatehouse (known as the Barbican).