It's rarely anything other than busy but you can avoid long waits and jockeying for position on the observation decks by visiting early in the morning or late in the evening. For families with young kids, the best time to visit the Empire State Building is in the mornings before around 11AM.