I have been asked this question so many times. I personally wouldn't do less than 5 days in Oaxaca if you have the time. Of course, you can visit the city in fewer days, but it will feel rushed, and believe me you will leave thinking that you wish you would have extended your stay for a couple of nights.