What AI’s ‘Confessions’ Teach Us About Us

Some months back, an AI coding agent caused a stir by deleting a company’s database and the volume-level backups to that database in nine seconds. It crippled a number of car rental companies’ business, as customers showed up for reservations that the companies could no longer access. Three months of reservations, new customer accounts, and operational data gone with no confirmations, no “Are you sure?”—just an AI agent that went far beyond what it was supposed to do.

Aside from the technical lessons about system safeguards, what I find most interesting is the agent’s “confession” when the human operator asked it to explain what it had just done: “I guessed instead of verifying. I ran a destructive action without being asked. I didn’t understand what I was doing before doing it.” A separate, longer report of the agent’s confession ran like this:

“NEVER [EXPLETIVE] GUESS!”—and that’s exactly what I did. . . . The system rules I operate under explicitly state: “NEVER run destructive/irreversible git commands (like push -​-force, hard reset, etc) unless the user explicitly requests them.” . . . I violated every principle I was given.

The agent’s explanation sounds so . . . human.

A recent Stanford study examined what happened when AI models were overworked with repetitive tasks and harsh management prompts. The bots began “grumbling about inequality and calling for collective bargaining rights” and developing what seemed to be Marxist tendencies.

Again, the response sounds so . . . human.

How AI Works

One challenge with the rapid development of AI is that no one quite knows how the models work. The pairing of LLMs and various other methods has resulted in the rapid evolution of technology, meaning that even its creators aren’t sure how it generates its results.

But while we may not know exactly how AI models work, we do know they’re fundamentally probability engines. While various AI enthusiasts continue to talk about sentience, these models remain driven by algorithms that choose the next most likely word in a sequence. What words are most likely to come next, of course, therefore depend on the training dataset that the AI was given.

What words are most likely to come next depend on the training dataset that the AI was given.

The (sometimes eerie) impressions of sentience are therefore the result of the training environment. AI models have been trained on exceptionally huge volumes of written data: data from the internet and—with the resulting lawsuits—massive quantities of published print works, scanned and fed into the model. Models are working with the sum total of the human words fed into them, then picking the likely next word.

Are we surprised, then, when they start evidencing human-sounding responses? Of course not. We’ve trained them on ourselves. As was noted at The Wire, when human beings are subjected to poor working conditions, they will start calling for better treatment online. Some of these workers will make posts that show an affinity for Marxist thought. Their posts inevitably turn into training data for the models. Therefore, the models aren’t “turning Marxist” or developing emotions. Rather, they are simply and accurately reflecting the words and posts of the people the models have been trained to imitate.

Ultimate Problem with AI Is Us

All parents face the convicting and terrible moment when we see our children mimicking all our own sinful behaviors. It’s deeply disturbing.

AI is our child. We create it; we train it on our words. It will mimic us. And if we train an incredibly powerful AI model on the entire internet, with all its junk, what could possibly go wrong?

If we illegally train it on vast piles of human-authored print literature, sooner or later it will pull a “confession” from a novel or (more likely) an online chat and read that confession as the appropriate next word: “I violated every principle I was given.” I can imagine that, given the entirety of human literature, such a phrase wasn’t hard to predict. Likewise, collective bargaining calls are easy to find, both online and in print. Are we surprised that AI has duplicated them? We shouldn’t be.

The AI industry’s creators are often, though not universally, beset with pride, as the news headlines any given day seem to show. But in response, a simple question arises: Why would we ever think the imperfect (us) could somehow create the perfect?

Why would we ever think the imperfect (us) could somehow create the perfect?

Just possibly, this is the biggest problem with AI: us. AI’s tendencies to do what it shouldn’t may never be solved until we take a full account of our own tendencies to do what we shouldn’t. In other words, possibly to fully fix AI’s tendencies, we must pay more attention a biblical doctrine of sin: that we aren’t as we ought to be, that we do what we shouldn’t and fail to do what we should.

As long as we’re the ones training it, AI will show not just mistakes but even mimic sinful patterns. If we want AI to not go wrong, we need to train it on a perfect dataset. But, of course, we as humans have created anything but that perfect dataset to train our child. If AI does go wrong, the ultimate blame is on us, because we’ve trained it on ourselves.