The short answer is mold, tsunami's and earthquakes is why they didn't build with stone. There are few 'ancient' examples of stone building but most of them came from the Chinese style for the temples that were constructed and you can see the resemblance in the Chinese Pagoda vs. the Japanese toba.