The term 'palace' comes from Rome's Palatine Hill, where the rich and famous built their sumptuous homes in Roman times. So when we're wondering what's the difference between a castle and a palace, the most important difference is that a palace is not fortified.