If we aren't just talking taking down your own buildings, but taking down enemy buildings in tribal warfare this needs to be a very long drawn out process. It should really take some time to bust through walls using a hammer, and it shouldn't just be a crafting hammer, it should be a full on two handed sledge hammer made by a weapon smith. Otherwise, there won't be much place for things like torches, battering rams, catapults, cannons, etc.

However I think this method is entirely fine using a crafting hammer, if it can only be done by the tribe in control of the area, and it gives the highest rate or resource recovery, with things like cannons and torches obviously giving a very low recovery rate.