There are in fact some places in the capital where you could spot a whale from the land. You will definitely need binoculars though. If you forgot yours at home, you can drop by at the Eiðisgrandi Beach in the west side of Reykjavik, where they just recently installed binoculars for whale watching.