You'll only need the minimum number of players to
start your tribe, be it 10, 5 or 3. After that your tribe size can reduce down to 2 if you have people leave.
So you could get together with some other small tribes and take turns forming tribes, or find some helpful people from larger tribes to make the numbers to plant your totem.
But very small tribes are going to find themselves at a disadvantage since they won't be able to cover all the necessary crafts needed for supporting building different structures.