While I don't think you are crazy, and I think a trip with a newborn is doable, especially for a seasoned mama, for a first-time mom I might recommend you wait at least 8 weeks, but maybe even closer to 12-16 weeks. I have visited Disney while pregnant, but the youngest I have brought to Walt Disney World was 4 months.