The main physical challenges are the multiple days walking; even fit people will hit the 'wall' at some stage. The hardest day of the Camino is on the first day of the French Way, where you have to cross the great mountain range of the Pyrenees over the Napoleon Pass.