I think part of it is an actual inability to distinguish the art from the artist, and equating the skillful performance of apparent incompetence with actual incompetence, which is then reinforced by them being depicted that way by other media.

Also, pantomime is the only comparable performance art I can think of, which is often just clowning: the musical.