I don’t think the vast majority of users will ever be interested in the sort of atypical content that niche communities provide. They generally prefer the more mainstream stuff, it’s kinda baked-in.

Also, I’m not talking about moderation, but federation. If you only federate with three instances, then only those three’s users can interact with you. This does not increase any moderation burdens.