Staying 2-4 days in Palermo is the perfect amount. With 1 or 2 days, you'll be able to fit in all the highlights, but your days will be long and busy. In my experience, 3-4 days is the sweet spot. It's enough time to get a solid feel for the city without rushing through your days.