Since today's castle building techniques still contain much of the hand crafting of the old days, you should plan to allow more time to build a castle than a traditional modern building. There are many more factors that affect building time, but the previously mentioned factors seem to pop up most often.