I think this is something to seriously take into consideration. One thing I loved about Haven and Hearth is I could wander off miles and miles in any direction and fine my own little corner by the river.

I would start by plugging away at a few baskets to store things in and with a little elbow grease and a campfire or two I could eventually clear chop my own little nook in the world and build my own little cabin. It took time, but it was possible with determination.

The feeling of solitude and accomplishment when you can form your own piece of paradise is definitely worth allowing in this game. Even if we cannot build real structures at least let us hermits have a decent yurt (tent) or at the very least a leanto. Even the ability to build fences. (honestly I'd like to be able to see the ability to clear land/plow/laystone and even lowgrade terraforming outside the limits of a tribe, but I'll take what I can get.)

Of course there's a risk associated with this survivalist experience, but it's worth it. You have to be clever and you'll end up quite tough and rough around the edges.

Fences, small dwellings, all doable.