Self-Learning Machines - What Happens When AI Starts Learning from Itself?

In this episode of The Agile Software Engineering Deep Dive, Alessandro Guida explores one of the most important questions in the next phase of artificial intelligence: what happens when AI starts learning from itself?

For years, generative AI has been trained largely on human-created material from the internet. But the internet is changing. More and more text, images, code, summaries, documentation, and online content are now generated or heavily assisted by AI. That raises a difficult question: when future AI systems are trained on the output of earlier AI systems, will they become more capable, or will they slowly lose contact with the richness and diversity of human knowledge?

The episode examines both sides of the self-learning machine problem. On one side, poorly controlled recursive training may lead to model collapse, narrowing, and fluent but less grounded outputs. On the other side, well-designed self-learning loops may accelerate progress in areas such as strategic games, reasoning systems, medical treatment optimization, synthetic data generation, and scientific discovery.

The central distinction is simple but important: a bad loop says generate, consume, repeat; a good loop says generate, test, filter, learn, repeat. The future of AI may depend less on whether machines learn from machines, and more on whether those learning loops remain connected to reality, evidence, constraints, and human judgment.

Transcript

SPEAKER_01 0:30

Welcome to the Agile Software Engineering Deep Dive, the podcast where we unpack the ideas shaping modern software engineering. My name is Alessandro Guida, and I've spent most of my career building and leading software engineering teams across several industries. And today I want to talk about something that sounds almost like science fiction. Self-learning machines. It sounds like a phrase from an old movie poster where the computer becomes conscious, locks the laboratory doors, and politely informs humanity that its services are no longer required. But behind the dramatic wording, there is a very real engineering question. What happens when AI starts learning from itself? For years, generative AI has been trained on enormous amounts of human-created material. Books, articles, websites, documentation, code, discussions, images, videos, and all the strange, brilliant, messy, contradictory material that humans have produced online. But now the internet is changing. More and more content is generated, rewritten, summarized, translated, corrected, polished, or heavily assisted by AI. Some of it is excellent, some of it is mediocre, some of it is wrong, and much of it is plausible enough to be difficult to distinguish from human-created material. So the question is no longer only whether AI can learn from human knowledge. We know it can. The question is what happens when future AI systems are trained on material produced by earlier AI systems? Does that create a powerful acceleration loop, where AI improves by generating new hypotheses, testing new strategies, refining its reasoning, and expanding what machines can discover? Or does it create a slow degradation loop where models are increasingly trained on synthetic echoes of previous models, gradually losing contact with the richness, diversity, and unpredictability of real human experience? In other words, will self-learning machines become smarter? Or will they slowly start eating their own homework? In this episode, I explore why self-learning AI can be both powerful and dangerous. Why examples from gameplaying AI, reasoning systems, medicine, synthetic data, and scientific discovery matter, and why the real issue is not whether AI learns from AI, but whether the learning loop is grounded in reality. Because a bad loop says generate, consume, repeat. A good loop says generate, test, filter, learn, repeat. And that difference may become one of the most important engineering questions of the next phase of AI. Let's dive in.

SPEAKER_02 3:25

Imagine for a second, right, that you ask an AI a complex question.

SPEAKER_00 3:29

Okay.

SPEAKER_02 3:30

And it outputs this incredibly detailed, highly articulate answer. But then, and here's the catch-it, it files that very same answer away, permanently incorporating it into its own training data set so it can, you know, teach itself tomorrow.

SPEAKER_00 3:45

Right, which sounds amazing at first, like a perpetual motion machine for intelligence.

SPEAKER_02 3:50

Actually, a system that just gets smarter by talking to itself.

SPEAKER_00 3:53

Yeah.

SPEAKER_02 3:53

But uh software engineers and data scientists are starting to discover that this exact process might actually be a recipe for, well, digital brain damage.

SPEAKER_00 4:02

Yeah. We are basically staring down this massive architectural crossroads. Depending on how we build the math behind that learning loop, it's either going to be the most powerful engine for discovery we've ever made, or it's just the slow, silent degradation of human knowledge.

SPEAKER_02 4:16

Aaron Powell Wow. Okay, so welcome to today's deep dive. We are unpacking issue number 32 of the Agile Software Engineering newsletter. It's titled Self-Learning Machines: What happens when AI Starts Learning from Itself?

SPEAKER_00 4:27

It's a fantastic issue.

SPEAKER_02 4:28

It really is. We're going to be looking at the core mechanics of recursive machine learning, digging into the data architecture, and pulling from their highly curated reading list to kind of figure out where this is all heading.

SPEAKER_00 4:40

Aaron Powell Because it really affects everyone.

SPEAKER_02 4:42

Right. Whether you're a senior backend engineer writing, you know, deterministic code or just someone using language models to parse a PDF, this evolution is fundamentally shifting the digital ground you walk on.

SPEAKER_00 4:53

Aaron Powell It is. And to understand the gravity of that shift, we have to look back at how machine progression actually functions. I mean, the concept of software improving itself isn't totally new, right?

SPEAKER_02 5:04

Aaron Powell Right, like Ray Kurzweil's vision.

SPEAKER_00 5:06

Trevor Burrus, Jr. Exactly. Kurzweil talked about this years ago, the idea that technology accelerates exponentially because well, better machines build better machines.

SPEAKER_02 5:15

Aaron Powell Yeah, but the underlying architecture of how that happens has radically transformed.

SPEAKER_00 5:19

Aaron Powell Oh, completely.

SPEAKER_02 5:20

Trevor Burrus, Jr.: Like if we contrast those architectures, for decades, traditional software engineering was just deterministic logic. Human developer writes a strict set of rules, defines the parameters, and uh dictates the workflow.

SPEAKER_00 5:33

Aaron Powell You write a function, the computer executes it. Simple. Trevor Burrus, Jr.

SPEAKER_02 5:36

Right. And if there's an edge case you didn't account for, the program just fails or throws an error. It's never dynamically rewrites its own core logic. Trevor Burrus, Jr.

SPEAKER_00 5:45

Well, no, it's totally static. But then we transitioned into early machine learning, right? Those probabilistic models. Trevor Burrus, Jr.

SPEAKER_02 5:52

Right, where we feed it examples.

SPEAKER_00 5:53

Yeah. Instead of hard-coding the logic for, say, how to recognize a cat in an image, we feed the system a massive data set of human-curated examples. We give it the inputs, the expected outputs, and the model calculates the statistical weights to bridge the gap.

SPEAKER_02 6:10

Aaron Powell But there was a critical limitation there, which was the training data itself. The raw material still had to be manually generated, labeled, and validated by us.

SPEAKER_00 6:19

Yeah, humans were the bottleneck.

SPEAKER_02 6:20

We had to supply the historical records, tag the good outcomes, tag the bad outcomes.

SPEAKER_00 6:25

Which brings us to the real leap, right? True self-learning. And AlphaGo Zero is the perfect illustration of this.

SPEAKER_02 6:31

Oh, I love the AlphaGo Zero example because it didn't just learn faster, it eliminated us entirely.

SPEAKER_00 6:36

Entirely. It didn't ingest a single human-played game of Go. It was initialized with nothing but the raw rules of the board.

SPEAKER_02 6:44

Aaron Ross Powell So instead of studying our centuries of, you know, recorded strategies, it just generated its own data set from scratch by playing millions of games against itself.

SPEAKER_00 6:55

Yeah. Self-play. And the way it updated its internal neural network during that self-play is fascinating. Every time it finished a game, it ran a backpropagation algorithm.

SPEAKER_02 7:04

Aaron Powell Meaning what? Exactly.

SPEAKER_00 7:06

Meaning a win mathematically strengthened the specific neural pathways, those probabilistic weights that led to those exact decisions, and a loss penalized them. So it continuously optimized its own architecture based purely on the friction of the game.

SPEAKER_02 7:21

So it's kind of like it's the difference between studying someone else's flashcards to learn a language versus being dropped in a foreign country and just figuring it out by interacting and failing.

SPEAKER_00 7:30

That's a great analogy. You aren't just absorbing static data, you're experiencing the absolute boundaries of the system. But the reason that specific architecture yielded such incredible results, like literally shattering centuries of human tradition, is the nature of the environment.

SPEAKER_02 7:47

Because Go is a game.

SPEAKER_00 7:48

Exactly. Go is a perfect mathematical universe. It has an undeniably rigid rule set, perfect information on the board, and a binary 100% reliable win or loss signal at the end. The reward function is absolute.

SPEAKER_02 8:02

There is zero ambiguity. But uh when we talk about generative AI, the large language models we're embedding into search engines and our daily workflows, they operate in the realm of human language and abstract ideas.

SPEAKER_00 8:15

Aaron Powell And language is definitely not a closed-loop board game.

SPEAKER_02 8:17

No, it's messy.

SPEAKER_00 8:18

Very. There's no mathematical win state when you ask an AI to summarize a legal contract or draft an architecture review. The reality of human communication breaks that absolute reward function.

SPEAKER_02 8:28

Aaron Powell Right. Because if an AI outputs an architecture summary, it might be incredibly fluent, grammatically flawless, but it could subtly misrepresent how two microservices talk to each other.

SPEAKER_00 8:39

Or it could be really persuasive but lack deep technical nuance. The signal is just noisy.

SPEAKER_02 8:49

Which brings us to the core structural threat. Because historically, AI trained on a human-generated internet, right?

SPEAKER_00 8:56

Right. Our code repositories, forums, literature, all the brilliant, messy human stuff.

SPEAKER_02 9:01

Aaron Ross Powell But the Internet is changing fast. It's rapidly filling up with synthetic AI-generated content, automated SEO, synthetic code snippets, machine summarized docs.

SPEAKER_00 9:11

Yeah, which guarantees that the next generation of models is inevitably going to train on the output of the previous generation. They're going to eat their own synthetic data.

SPEAKER_02 9:20

Aaron Powell Okay, wait, I have to ask about the danger here. If an LLM is already highly capable and it outputs really articulate, well-structured text, why is it structurally dangerous for the next model to ingest that? Isn't it just reinforcing good grammar and clean formatting?

SPEAKER_00 9:37

Okay, think about it like this. Imagine taking a highly detailed photograph and running it through a copy machine.

SPEAKER_02 9:42

Okay.

SPEAKER_00 9:43

Then you take that photocopy, put it back in the machine, and copy it again. Repeat that 50 times. What happens?

SPEAKER_02 9:48

The sharp details blur into a gray smudge.

SPEAKER_00 9:51

Exactly. The finest textures, the subtle shadows they all wash out. In machine learning, this degenerative process is called model collapse.

SPEAKER_02 9:59

Model collapse. So the AI is effectively engaging in self-amplified mediocrity.

SPEAKER_00 10:04

Yes. Mathematically, it's a flattening of the probability distribution. An AI model naturally favors the most statistically likely sequences of tokens. It gravitates toward the center of the bell curve, the highly predictable patterns. Right. So when a new model trains on that generated output, the rare, complex edge cases at the tails of the distribution, they start getting assigned lower and lower probability weights until they vanish entirely.

SPEAKER_02 10:28

Aaron Ross Powell But those tails are exactly where the most valuable human knowledge lives, like minority viewpoints or highly unusual software bugs or deep specialist knowledge.

SPEAKER_00 10:38

Trevor Burrus Yes. All of that gets systematically washed out in favor of the safest, most generic statistical average. And the truly insidious part is that the syntax and grammar stay absolutely pristine. The output still looks incredibly authoritative.

SPEAKER_02 10:52

So you don't even see the degradation on the surface. We're essentially building a hall of mirrors, artificial content reflecting back into artificial content, quietly stripping away the actual connection to reality.

SPEAKER_00 11:04

That maps perfectly onto Melanie Mitchell's arguments. She's on the newsletter's reading list about what intelligence really entails. Just because a system outputs highly articulate sentences doesn't mean it fundamentally understands the concept.

SPEAKER_02 11:16

Right. It's just mimicking the structural shape of understanding.

SPEAKER_00 11:19

Exactly. And if it's only mimicking the shape of other synthetic outputs, it entirely loses the plot. It also brings up Brian Christian's work on the alignment problem.

SPEAKER_02 11:29

Which is what? Exactly.

SPEAKER_00 11:30

Well, if a model is self-learning from an increasingly polluted, flattened data set, its internal representations will mathematically misalign with the complex, messy reality of the human world it's supposed to operate in.

SPEAKER_02 11:42

So if recursive learning inevitably leads to model collapse, does that mean all synthetic data is just fundamentally toxic? Like, are we hitting a hard ceiling here?

SPEAKER_00 11:52

No, and the author of the Agile Software Engineering newsletter draws a brilliant engineering distinction here. Synthetic data itself isn't toxic. The danger is dictated entirely by how you engineer the learning loop. We have to separate sterile loops from productive loops.

SPEAKER_02 12:09

Okay, so a sterile loop is the hall of mirrors we just talked about. Generation devoid of reality. The model produces text, the next one consumes it, the weird stuff gets washed out.

SPEAKER_00 12:21

Right. But a productive loop introduces a critical architectural mechanism. Generation followed by rigorous evaluation against an external, meaningful constraint.

SPEAKER_02 12:31

Aaron Ross Powell The machine desperately needs friction.

SPEAKER_00 12:33

Yes, friction to test its hypotheses against.

SPEAKER_02 12:36

So what does that look like in practice? The newsletter gives a few specific ways engineers are building that friction back in. Like let's talk about reasoning systems, specifically star, the self-taught reasoner.

SPEAKER_00 12:46

Oh, star is a great example. It avoids the trap by treating language generation the way a math student treats scratch paper. The model generates multiple different reasoning paths, step-by-step logic, trying to solve a complex problem. But it doesn't just blindly train on all its own generated logic.

SPEAKER_02 13:02

Oh, it filters it.

SPEAKER_00 13:03

Yeah, it only updates its weights using the specific reasoning paths that ultimately arrive at a mathematically or factually correct answer verified by a hard external key.

SPEAKER_02 13:14

So it's allowed to generate synthetic logic, but that logic has to survive the friction of a reality check. And if it fails.

SPEAKER_00 13:22

If it fails, it's given the correct answer and forced to work backward to reconstruct a plausible path. It uses its own generation, but filters it through an absolute constraint.

SPEAKER_02 13:33

Aaron Powell I love that. So a sterile loop is like an echo chamber filled with yes men, but a productive loop is like a science lab.

SPEAKER_00 13:39

Yeah.

SPEAKER_02 13:39

Your hypothesis actually has to survive real-world friction. You're allowed to fail.

SPEAKER_00 13:44

That is exactly it. Another fascinating approach to preventing degradation is token level editing.

SPEAKER_02 13:49

Aaron Powell Right, because AI doesn't process whole words, it processes tokens, statistical chunks of characters.

SPEAKER_00 13:55

Exactly. So token level editing means the system isn't generating a massive synthetic document from a blank slate to train on.

SPEAKER_02 14:01

So instead of writing a whole new article that might, you know, hallucinate or drift toward the average, it acts more like a surgical editor.

SPEAKER_00 14:07

Aaron Powell Precisely. It takes a highly complex human-engineered data set and selectively perturbs specific tokens. It might swap out a few variables or alter a contextual parameter or change the tone of a single paragraph. It acts as a controlled mathematical mutation.

SPEAKER_02 14:24

Got it. So it preserves the messy human scaffolding, the real underlying ideas, but artificially multiplies the variation so the model has more robust edge cases to learn from.

SPEAKER_00 14:35

Yeah, it's brilliant.

SPEAKER_02 14:36

But I think the most profound example of a productive loop from the reading has to be the MIT glioblastoma study.

SPEAKER_00 14:42

Oh yeah. Applying reinforcement learning to medicine really shows the stakes here. Glioblastoma is an aggressive brain cancer. And obviously, you cannot run exhaustive trial and error reinforcement loops on human patients.

SPEAKER_02 14:53

No, you can't just experiment with dosing strategies on people. So the researchers built a highly detailed clinical simulation model. They created a digital environment mirroring human biology and then unleashed the AI to explore different treatment schedules.

SPEAKER_00 15:07

Right. And the goal was to reduce the toxicity of the drugs while keeping the treatment highly effective against the tumor.

SPEAKER_02 15:13

Aaron Powell And the engineering brilliance there is in the reward function, right?

SPEAKER_00 15:17

Yes. The AI was given a dual constraint. It received positive reinforcement for minimizing the simulated tumor size, but heavy mathematical penalties if a toxicity score climbed too high.

SPEAKER_02 15:29

And because it had the freedom to explore millions of permutations against those hard biological constraints, it didn't just default to the standard human assumption of, you know, more medicine equals better results.

SPEAKER_00 15:41

No, it actually discovered that in specific scenarios, administering lower doses at different counterintuitive intervals was far superior.

SPEAKER_02 15:49

It successfully challenged human intuition, but only because its synthetic hypotheses had to survive the brutal friction of the medical simulation.

SPEAKER_00 15:57

Exactly. If we pull this back to the Mac review, Stuart Russell, another aucker from the reading list, frequently points out the control problem in AI. Building beneficial constraints isn't just a safety feature, it's the fundamental mechanism that allows a self-learning system to generate novel knowledge without just spinning off into noise.

SPEAKER_02 16:16

So the dividing line between a degenerative hall of mirrors and a supercharged engine of human discovery is entirely defined by friction.

SPEAKER_00 16:24

Yep. Which forces a massive perspective shift on how we treat data. If the machine desperately needs reality and friction, then data is no longer just, you know, fuel we shovel into a furnace.

SPEAKER_02 16:36

Data is an evolving ecosystem. And that brings up the most critical question in modern AI architecture. What is the teacher? In AlphaGo, it was the game board. In early generative AI, it was the human internet.

SPEAKER_00 16:48

But if that ecosystem floods with automated, unverified content, then data provenance, knowing the exact origin, lineage, and verification status of a piece of data becomes the most important engineering problem of our time.

SPEAKER_02 17:00

Absolutely. If you're managing massive data lakes, you can't blindly ingest text anymore. You have to track provenance. Was this human created? Was it an AI summary? Was it experimentally verified?

SPEAKER_00 17:12

Max Tagmark discusses this too. What happens when systems aren't bound by slow biological evolution, but can rapidly design their own improvements?

SPEAKER_02 17:20

Aaron Powell And Pedro Domingo's work underscores that the algorithms themselves are only as good as the environment they optimize against. If we pollute the internet and our shared data, the models will mathematically degrade. It's a cultural problem now, not just computer science.

SPEAKER_00 17:36

Yeah. Every line of code a human verifies, every piece of complex documentation we author, every nuanced minority viewpoint we preserve online, it is literally shaping the evolutionary environment for the next generation of models.

SPEAKER_02 17:49

We aren't just passive consumers anymore. We are the architects of the friction. Which honestly brings us to a surprisingly optimistic conclusion from the newsletter.

SPEAKER_00 17:57

Right, Kurtzweil's accelerating feedback loop doesn't have to be a path to collapse.

SPEAKER_02 18:01

No. A recursive loop accelerates whatever it's pointed at. It can accelerate noise or it can accelerate discovery. The solution is engineering a shared loop. AI shouldn't substitute human intellect, it should amplify it.

SPEAKER_00 18:16

AI models can explore massive solution spaces millions of times faster than a human brain.

SPEAKER_02 18:22

But we have to inject the reality. We bring the creative intuition, the ethical constraints, the edge case curiosity. We define the reward functions and verify the outcomes against reality.

SPEAKER_00 18:32

We act as the ultimate constraint. We create new foundational knowledge, and that verified human data becomes the grounded training environment for the next generation. They don't learn in a vacuum. We push them, they push us, and we accelerate together.

SPEAKER_02 18:46

Provided we engineer the pipelines to preserve that friction.

SPEAKER_00 18:49

Exactly.

SPEAKER_02 18:50

There is an immense amount of technical nuance to these learning loops that we simply couldn't fit into this deep dive. To fully grasp the underlying architectures, you really need to read the full article.

SPEAKER_00 18:59

Highly recommend it.

SPEAKER_02 19:00

We strongly urge you to subscribe to the Agile Software Engineering newsletter. It's the best way to support rigorous, high-quality technical deep dives like this one, and their insights are absolutely essential right now.

SPEAKER_00 19:11

It really is mandatory reading for anyone building in this new ecosystem.

SPEAKER_02 19:15

So as we wrap up, I want to leave you with a structural question to think about. As AI becomes our primary interface for summarizing docs, generating code, and filtering the vastness of the internet, will we even possess the metrics to realize if the hall of mirrors has fully enclosed us? Or will that self-amplified mediocrity just slowly, silently become our new baseline for reality?

SPEAKER_01 19:49

Thank you for listening to the Agile Engineering Deep Dive Podcast. If you found this episode valuable, feel free to share it with someone who might benefit. A colleague, your team, or your network. You can access all episodes by subscribing to the podcast and find their written counterparts in the Agile Software Engineering newsletter on LinkedIn. And if you have thoughts, ideas, or stories from your own engineering journey, I'd love to hear from you. Your input helps shape what we explore next. Thanks again for tuning in, and see you in the next episode.