History of Fishers Island It has been part of Southold since 1879 when a joint commission from Connnecticut and New York decided the legal boundary of Connecticut stopped at the shoreline. The island lies about six miles northeast of Great and Little Gull Islands in Long Island Sound.