They can't see it when they initially accept rides, only when they start the ride. Sounds like while you were in the middle of loading, he started it, saw the destination, and said no, if you literally just got into the car and it was the very first thing he said to you. Most can't.