Global service provider Keywords, which recently worked on acclaimed projects like Alan Wake 2, Baldur’s Gate 3, and The Legends of Zelda: Tears of the Kingdom, conducted an experiment last year.
The company tried to make a 2D video game relying solely on generative AI (GenAI) tools and technology. The R&D initiative was dubbed ‘Project Ava’ and saw a team, initially from Electric Square Malta, evaluate and leverage over 400 (unnamed) tools to understand how they might “augment” game development.
As detailed in the company’s latest fiscal report, however, the project ultimately proved that while some generative AI tools might simplify or accelerate certain processes, they are currently “unable to replace talent.”
Story time:
I’m a software developer for a large, multinational company. Yesterday, I needed to update the Knex migrations for the project I’m assigned to. We needed three new PostgreSQL tables, with several foreign key constraints. I added the migration to our existing migrations in a backend plugin we’re building out.
I use Copilot for developers regularly. It was helpful in this case, generating the table migrations automatically. Of course, it hallucinated a few methods Knex doesn’t have, but I’m used to things like that, and easily corrected them. Once I was done testing, I created a pull request to merge the commit in my working branch with the main branch in git.
Now, look at what I just wrote. If you’re not a developer, you probably have no idea what “Knex” or “PostgreSQL” mean. You probably recognize the words “foreign,” “key,” and “constraints,” but you haven’t got a clue why I’m using them in that order or what I’m referring to. It likely looks like I’m using the word “migrations” completely incorrectly. You don’t know what it means for “Knex” to have “methods.” Words like “git,” “pull request,” and “commit” just read like gibberish to you.
You wouldn’t know how to ask Copilot to do anything. You wouldn’t know where to place any results you manage to get from it. If your boss came to you and said, “here’s this feature requirement, make it happen,” you would fail. You wouldn’t know why, either. Hell, you wouldn’t even know what it is your boss is trying to accomplish. You could spend the next six months trying to figure it all out, and maybe you’d succeed, but probably not. Because you aren’t a developer.
I’m a developer. All of what I wrote above makes perfect sense to me, and it’s one of the simplest tasks I could tackle. Took about fifteen minutes to accomplish, from creating the migration file to getting the PR ready to merge.
I’ve been lambasted for insisting that large language models aren’t going to replace actual professionals because they’re not capable of joined-up thinking, meta-cognition, or creativity. I get told they’ll be able to do all of that any day now, and my boss will be able to fire all of his employees and replace them with an MBA - or worse, do the work himself. Depending on the attitudes of who I’m talking to, this is either a catastrophe or the greatest thing since sliced bread.
It’s neither, because that’s not going to happen. Look at the story above, and tell me you could do the same thing with no training or understanding, because ChatGPT could do it all. You know that’s bullshit. It can’t. LLMs are useful tools for people like me, and that’s it. It’s another tool in the toolbox, like IntelliSense and linters - two more terms you don’t know if you’re not a developer.
The bloom is beginning to come off the rose. Businesses are gradually realizing the pie-in-the-sky promises of LLM boosters are bogus.
The majority of my time isn’t spent writing code; it’s reading code, reviewing changes, and thinking about code.
Amen to that! I also spend a fair amount of time thinking about new features and how they would plug into our vast ecosystem, no part of which could Copilot possibly know anything about.
People know what pull requests are. I make them to my girlfriend all the time.
But does the python knex?
The python knexes at midnight.
Honestly, the shorter way to say all of this is “the AI” is supposed to make everyone a coder. “The AI” is supposed to make everyone an artist. Who is doing more coding or art than they used to? I know some people are (I am) but I know most people aren’t, even a little bit. That’s how you know it’s a tool for people who want it, or wanted it before it existed, but it’s not really driving any new demand at the level that would replace the experts already in place who can use the tool.