In polite conversation, yes, both would be acceptable. The one you choose depends on how much you know about the other's future plans. If you know they are continuing their travels, use 'have a nice trip'. Otherwise, 'enjoy your stay' work for most situations.
The terms going on holiday and taking a trip are often used interchangeably to describe a temporary break from work or routine, typically for the purpose of leisure or relaxation.
“Have a nice trip”: enjoy yourself, hope you have no travel difficulties, take lots of photos, visit the wonderful sites to see, relax, eat in great restaurants, take in a concert, take what the hosts will give you. “Have a safe trip”: a little more specific. Come back healthy with no injury or illness.