The month of March is calmer compared to April's weather when the winds are at their highest. So, head out in the morning and swim during the warmest part of the day if you want to avoid being too cold. You can swim just about any time of year in San Diego.