Harlem became an area that held a portion of New York City's population that was relatively well-off. While the area had a wealthy white population, it also held a small black population since a few blacks had stayed from the time of slavery. New blacks moved in with the whites as domestic servants.