Dostoyevsky's “Notes from Underground” apparently had a big influence on the movie, and many people see similarities. It's a great book on its own, so I'd highly recommend reading it. Seconding Notes from the Underground. Screenwriter Paul Schrader said it was his biggest influence when writing Taxi Driver.

While Taxi Driver (1976) is an original screenplay by Paul Schrader, it wasn't inspired by a single book so much as a combination of literary influences, real-life events, and personal turmoil. Schrader has frequently cited "Diary of a Country Priest" by Georges Bernanos as a primary structural influence, particularly the concept of a lonely, suffering man recording his inner thoughts. The film was also heavily influenced by the diaries of Arthur Bremer, who attempted to assassinate presidential candidate George Wallace in 1972; Bremer's alienated, rambling writings helped shape Travis Bickle's erratic internal monologue. Furthermore, Schrader drew from existentialist literature, such as Jean-Paul Sartre's "Nausea" and Fyodor Dostoevsky's "Notes from Underground," both of which explore the psyche of a social outcast living on the fringes of society. These high-concept literary roots combined with Schrader's own experiences with insomnia and isolation in New York City created the gritty masterpiece we know today.