You only have to maintain two-way communication in Class E airspace when flying IFR. All airports where there is an operating control tower requires two-way communication within 4 Nautical Miles of the airport housing the control tower at and below 2500 feet AGL at a minimum, regardless of airspace.