Right now walls can't be destroyed by outsiders, so we do not know how long that will take or if it will take longer to destroy a masonry wall than a branch barricade. The rules for tribe warfare and when it will be introduced haven't been set.
Ravelli