Megan Garcia sued Character.AI in federal court after the suicide of her 14-year-old son, Sewell Setzer III, arguing the platform has "targeted the most vulnerable members of society – our children"
We are playing with some dark and powerful shit here.
We are social creatures. We’re primed to care about our social identity more than our own lives.
As the sociologist Brooke Harrington puts it, if there was an E = mc2 of social science, it would be SD > PD, “social death is more frightening than physical death.”
…yet we’re making technologies that tap into that sensitive mental circuitry.
And the cruel irony on top of it is:
Because we care so much about preserving our social status, we have a tendency to deny or downplay how vulnerable we all are to this kind of “obvious” manipulation.
Just think of how many people say “ads don’t affect me”.
<If I can find it, I’ll add a link to an interesting study on distracted driving and hands-free options. If I recall correctly, talking to someone in the passenger seat was only mildly distracting, but having the same conversation over a hands-free call was way more distracting. And their proposed explanation was that a passenger has the same context and naturally understands if the conversation needs to pause, but on a call there is no shared context so there is social pressure to keep going. Unnervingly, voice assistant interactions had the same problem. They trip the same part of our brain that worries about politeness at the expense of our physical safety.>
I’m worried we’re going to severely underestimate the extent to which this stuff warps our brains.
I was going to make a joke about how my social status died over a decade ago, but then I realized that no, it didn’t. It changed.
Instead of my social status being something amongst friends and classmates, it’s now coworkers, managers, and clients. A death in the social part of my world - work - would be so devastating that it motivates me to suffer just a little bit more. Losing my job would end a lot of things for me.
What we need is a human society predicated on affording human decency, rather than on taking it away to make profit for those who already have the most.
We are playing with some dark and powerful shit here.
We are social creatures. We’re primed to care about our social identity more than our own lives.
…yet we’re making technologies that tap into that sensitive mental circuitry.
And the cruel irony on top of it is:
Because we care so much about preserving our social status, we have a tendency to deny or downplay how vulnerable we all are to this kind of “obvious” manipulation.
Just think of how many people say “ads don’t affect me”.
<If I can find it, I’ll add a link to an interesting study on distracted driving and hands-free options. If I recall correctly, talking to someone in the passenger seat was only mildly distracting, but having the same conversation over a hands-free call was way more distracting. And their proposed explanation was that a passenger has the same context and naturally understands if the conversation needs to pause, but on a call there is no shared context so there is social pressure to keep going. Unnervingly, voice assistant interactions had the same problem. They trip the same part of our brain that worries about politeness at the expense of our physical safety.>
I’m worried we’re going to severely underestimate the extent to which this stuff warps our brains.
I was going to make a joke about how my social status died over a decade ago, but then I realized that no, it didn’t. It changed.
Instead of my social status being something amongst friends and classmates, it’s now coworkers, managers, and clients. A death in the social part of my world - work - would be so devastating that it motivates me to suffer just a little bit more. Losing my job would end a lot of things for me.
I need to reevaluate my life
What we need is a human society predicated on affording human decency, rather than on taking it away to make profit for those who already have the most.