I’m guessing he kept it all intentionally. He had the manifesto on him, probably expecting “accidental” suicide by cop in hopes that his message would continue and not be painted over by the media. Yeah, he could have ditched the gun, but again, perhaps he didn’t want there to be any shadow of a doubt that he is guilty. This was an intentional sacrifice in hopes of making a change.