It brings your overall average score down, and while you won't necessarily be banned by the service if it plummets – being kicked off Uber or Lyft is determined on a case-by-case basis – a low rating could mean fewer drivers will want to pick you up.