I’m aware of the flaws. I don’t agree that means it shouldn’t exist. There’s certainly room for improvement, and I’m even open to the idea that it’s too early to roll it out.

This betrays a fundamental misunderstanding of the technology on your part. Large language models do not understand anything. They have no concept of truth or falsehood. They are not intelligent. The only thing they do is predict text. They’re more complex and realistic versions of the classic ELIZA program, not real AI. They will never be capable of filling the role Mozilla has shoehorned them into.

I’m not sure I understand the argument that this is somehow making them money.

The behavior of the Mozilla representatives strongly implies it. I have no idea how they intend to make money with this, and they may or may not succeed, but people don’t generally act like this unless they think they can strike it rich by doing so (and don’t care about the harm they’ll cause in the process).