Sir Henry Mill Pellatt wanted a medieval castle on the hill for him and his wife, Mary, to enjoy. He hired a grand architect of his time, E.J. Lennox, to create his palace on the hill with city views of Toronto. He began by building the stables in 1911.