Folly Beach is a smaller, more laid-back community that is located on a barrier island to the south of Charleston. There are several key differences between Isle of Palms and Folly Beach, including their size, location, and character. Isle of Palms is a larger community, with a population of around 4,600 people.