The original argument is that Signal does not want to create a social graph of user accounts on their server and rather rely on the already existing one of the users phone book.

But that is very narrow thinking and ultimately counterproductive as others have pointed out here already.