It's perfectly safe, but the first bit of Southwark Street after you get past Borough Market is very quiet (for pedestrians) at night and can feel a little creepy. So while it's objectively safe, your wife and daughter might feel better in a bus or tube.