About 650 families, including celebrities and mega stars, have homes on Fisher Island. Fewer than 20 percent of residents live on this secret haven year-round. Carl Fisher traded seven acres of Fisher Island to William K. Vanderbilt II in 1926 in exchange for a yacht named Eagle.