As the person that was helping Noble when we discovered this, and someone with experience with building and large tribes, I can see both sides of things here. It was frustrating to be the one that told him to just temporarily move his totem, something I saw as simple, only to turn around and have to rebuild so many things to continue building where we were. But I've also experienced the severe lag of a larger tribe and all the ghosts that can accompany it. So I understand why the change even though I don't particularly like it.
I'm facing issues now as I'd originally planned to be able to use one of my characters (that I have in my own tribe) to pop around with their totem when needed to build a bridge, and do some public lands fixing, clearing a large patch of paving, etc, but because I have structures at the outer limits of my tribe, I now can't pull a character out or I will have to rebuild to use those structures (some walls that I plan to build on, as well as my infirmary and mist trader at this time). I also was the person that would pull Jexa out to help anyone that needed it, either with building for them or dropping water barrels before returning to my own tribe. I now have to be more careful and don't know that I can, unless I expand more.
I do have a couple of possible suggestions that would maybe work for both sides.
We already have various timers associated with the totems; 6hrs for a toon to place a new totem from the time they placed the previous one. 20 minutes between joining new tribes. So what if there was a certain time limit where we could move a totem and return it without losing the 'ownership' of the structures. Say a tribe pulls their totem, moves it to do whatever, if that totem is put back in place by the same tribe within say 7 days (or some other set limit, 3 days) they retain their ownership of all structures and can continue on as if they'd never moved their totem. If the totem isn't replaced within the time frame, the ownership is lost.
Personally, I believe if the same toon that built a usable structure is in a tribe, even after a totem is moved, that toon should still be able to set the use. Ownership of 'use' and/or storage structures should be retained to some degree, at least to the original builder (if their tribe is also there). Having to dismantle and rebuild a storage structure is more involved than a wall. And it isn't just about the loss from dismantling, you must first remove any bins or items stored in the tent to dismantle it, before rebuilding and then putting those bins and items back. Often bins in a storage structure are very heavy, so dropping everything to the ground doesn't work well as they can't just be picked back up.
As for ghosts that are laid out, or decayed, disappearing when a totem goes (either through natural decay or someone picking up their totem) I can certainly see the benefit in the lag area. Is there a way to remove ghosts that naturally decayed, or were placed long ago - maybe to match the timer for tribe abandonment/decay. So, if an abandonment timer starts, all ghosts currently there would also be removed when the totem goes. And if a tribe pulls their totem, using the my previous suggestion of a timer for moving a totem, they have the 7 days (or whatever) where the ghosts stay in place, otherwise they're deleted after that time.
Another option, tho less helpful, would be to remove empty ghosts, but if someone has placed at least some items in the build window, then it doesn't get removed if they pull their totem. This would require ghosts that are naturally decayed to always lose all items (which I don't see as a problem tbh) but would not fix the issue if a tribe left some items in some ghosts but didn't build and then it naturally went abandoned.
I believe a timer for moving the totem would be far more useful and allow some flexibility behind the scenes to work future or unforeseen issues or to possibly help with other things, such as the ghost structures and removal.
Just a couple of my thoughts for suggestions so far.