If seat is in Coach and assigned to someone the person may ask you to move out of their seat (especially if it's a full flight). If seat is in a first class/premium seating even if unassigned the Flight Attendants may ask you to see your boarding pass and move back to your assigned seat.