Just speculation, but I think that this is what the "baking" is supposed to periodically fix. The problem is, performance will tend to de-grade as all of this cached-on-the-fly terraforming data or whatever it is starts to accumulate.
Then it'll get baked, and everything will be peachy again, until the time to bake again draws near.