If you want to avoid all traffic on both ends, then you have to leave in the middle of the night. If you leave DC by 2 AM, you can be in Manhattan by 6 AM. You will run into little to no traffic unless there is construction or accidents. Or otherwise, leave at 8 PM and get to Manhattan by Midnight.