As someone who's lived there, I can say that, since Boston is so small, you won't waste a lot of time “in transit,” so you can pack a lot into your days. Obviously, you can spend longer here (slow travel is the best travel), but for the first-time visitor, three to four days is enough.