Agile Software Engineering
This podcast explores how craftsmanship, architecture, engineering rigor, and organizational practices come together in modern R&D environments. Each edition refines and deepens my earlier reflections, building a coherent and evolving body of knowledge around Agile Software Engineering
Agile Software Engineering
THINKING SLOW, THINKING FAST AND INTUITION IN SOFTWARE ENGINEERING
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode of The Agile Software Engineering Deep Dive, Alessandro Guida explores intuition in software engineering through Daniel Kahneman’s ideas about fast and slow thinking.
The episode looks at how years of experience can turn deliberate reasoning into fast pattern recognition, why this matters in debugging, code review and architecture, and where intuition can become dangerous when confidence grows faster than evidence.
It also explores the relationship between intuition and first-principles thinking, and asks what these human capabilities may mean in a world where AI can generate increasingly convincing software at remarkable speed.
And, of course, it answers one important question: why does John earn so much?
This Podcast is an audio version of the written Agile Software Engineering newsletter. If you want to go deeper, don't forget to subscribe the newsletter too.
Welcome to the Agile Software Engineering Deep Dive, the podcast where we unpack the ideas shaping modern software engineering. My name is Alessandro Guida, and I've spent most of my career building and leading software engineering teams across several industries. Today, I want to talk about something every experienced engineer probably knows, even if we rarely put a name to it. Intuition. You've probably worked with someone like John, the engineer who looks at a production problem for a few minutes, points in one direction, and somehow turns out to be right. So where does that ability come from? And when should we trust it? In this episode, I use Daniel Kahneman's ideas about fast and slow thinking to explore how experience becomes pattern recognition, why intuition can be incredibly valuable in debugging, code review, and architecture, and why the same intuition can sometimes mislead us. We will also look at the relationship between intuition and first principles thinking, and why this balance may become even more important as AI makes producing software faster and easier. And somewhere along the way, we may finally answer the question why does John earn so much? With that, let's dive in.
SPEAKER_02Exactly. They pull up this, you know, massive spreadsheet of complex alphanumeric error codes. Yeah. And then they just start meticulously testing the sensors one by one. I mean, it's it's an incredibly systematic, deliberate process of elimination.
SPEAKER_01Aaron Powell Well, it really has to be, right? Modern cars are basically just rolling computer networks at this point.
SPEAKER_02Oh, totally.
SPEAKER_01So the default approach is just to treat the problem like a complex data puzzle.
SPEAKER_02Aaron Powell Right. But then there's always that one like veteran mechanic in the shop. You know the guy, he's been covered in motor oil since like the late 90s.
SPEAKER_01Oh yeah. Everyone knows that guy.
SPEAKER_02Trevor Burrus And you walk past your car while it's just idling in the bay, doesn't even glance at the computer screen. He just tilts his head, listens for about three seconds, and casually says, Oh, it's the timing belt.
SPEAKER_01And it drives everyone else absolutely crazy.
SPEAKER_02It does. Because it just feels like magic. But today we are taking a deep dive into that exact same phenomenon. Except uh we are looking at it in the world of software engineering.
SPEAKER_01Aaron Powell Which is fascinating. Aaron Powell Yeah.
SPEAKER_02We're unpacking issue number 45 of an agile software engineering newsletter, and it's titled Thinking Slow, Thinking Fast, and Intuition in Software Engineering.
SPEAKER_01Aaron Ross Powell It is such a brilliant examination of the intersection of human psychology, code architecture, and really where the future of artificial intelligence is actually taking the whole tech industry.
SPEAKER_02Aaron Powell Yeah. And the mission of this deep dive is to unpack why the most highly paid, you know, most valuable engineers often seem to do the absolute least visible work and how a famous psychological framework explains this secret superpower of theirs.
SPEAKER_01Aaron Powell It's a great mission.
SPEAKER_02And hey, as we get into this, if you find yourself nodding along and wanting to see the actual visual models the author uses, uh you should really do yourself a favor and read the full article.
SPEAKER_01Aaron Powell Oh, highly recommend it. The visual flow charts in the source text really anchor the cognitive concepts we're about to explore.
SPEAKER_02Yeah. So definitely subscribe to the newsletter because we can really only scratch the surface of the underlying psychology here. Also, since this deep dive is completely free, it is a massive help to us if you hit like, subscribe to get all our future deep dives, and you know, forward this audio and the related article to your colleagues.
SPEAKER_01Aaron Powell Yeah, it's the kind of material that completely changes how a dev team approaches their next, like really complex problem.
SPEAKER_02Totally. So to understand how that master mechanic intuition actually operates in a highly technical environment, the author introduces this archetype. A character that exists in almost every single mature software organization. The text basically just calls him John.
SPEAKER_01Aaron Powell Yeah, if you work in tech, you definitely know a John. In fact, if you've been doing this long enough, you might actually be striving to become a John.
SPEAKER_02Aaron Powell Right. So let's put John in his natural habitat. Imagine a severe production outage. The core system is completely down, revenue is just bleeding by the minute, and a team of like really capable developers has been trapped in this desperate, panicked, troubleshooting meeting for two solid hours.
SPEAKER_01Aaron Powell Just an absolute nightmare scenario.
SPEAKER_02Oh, the worst. They are frantically digging through server logs, staring at complex metric dashboards, tracing network requests across a dozen microservices, and just, you know, throwing out increasingly desperate theories about what went wrong. Right. And then John logs into the call.
SPEAKER_01The veteran.
SPEAKER_02Yep. Someone quickly summarizes the absolute chaos for him. John looks at the shared screen for like a few minutes, nods slowly, and then simply says uh I would look at the database connection pool.
SPEAKER_01Just out of nowhere.
SPEAKER_02Right. And when the rest of the team asks him why, because you know they're expecting this rigorous mathematically sound chain of logic, he can't even really give them one. He just shrugs and says, I'm not sure. It just feels like that kind of problem.
SPEAKER_01Aaron Powell Which, I mean, that has to be infuriating to a room full of people who have been doing complex distributed systems analysis for two hours.
SPEAKER_02Aaron Powell Mildly irritating at best, right? Especially because like 20 minutes later, someone actually checks the connection pool metrics and they discover that John was absolutely right. The system recovers, the bleeding stops, and John just goes back to whatever he was doing.
SPEAKER_01Aaron Powell And here's the funny detail about John that the text points out, which I actually really loved. He is not necessarily the stereotypical 10x coder that pop culture likes to portray.
SPEAKER_02Right. He's not hacking into the mainframe in a hoodie.
SPEAKER_01Exactly. He doesn't type furiously fast in a dark terminal. He probably doesn't know the absolute newest, trendiest JavaScript framework. And brilliantly, the author notes, there is a very reasonable chance that someone occasionally has to help John find the mute button that Microsoft moved in the latest version of Teams.
SPEAKER_02That is the irony there, is perfect. He struggles with the UI of a chat application, yet he is just casually saving the company from a catastrophic multimillion dollar infrastructure failure.
SPEAKER_01Yeah, and because of that massive leverage, John earns considerably more money than the younger, faster developers.
SPEAKER_02Okay, let's unpack this. Is John's brain basically running a lightning fast background database query of, I don't know, 327 previous production incidents without him even consciously realizing it?
SPEAKER_01Aaron Powell I mean, that is precisely what is happening under the hood. And it brings us right to the core of Daniel Kahneman's work.
SPEAKER_02Oh, right.
SPEAKER_01Yeah. Kahneman was a Nobel laureate who really revolutionized our understanding of human judgment. In his famous book, Thinking Fast and Slow, he categorizes human thought into two distinct, constantly interacting systems.
SPEAKER_02The famous System One and System Two.
SPEAKER_01Right. System one is fast, automatic, intuitive, and highly associative. It takes virtually zero cognitive effort to run.
SPEAKER_02Aaron Powell, so for John, System One is a process that's actually looking at the broken software architecture.
SPEAKER_01Aaron Powell Yes, and we need to be clear about how this plays to code. When an experienced developer opens a file, System One activates instantly. Before they even consciously read the logic line by line, they get a feeling.
SPEAKER_02Like a literal physical reaction.
SPEAKER_01Almost, yeah. They spot a massive utility class or a method where the responsibilities are all mixed up, or some strange cyclic dependency between two components that shouldn't even be aware of each other. It just smells wrong.
SPEAKER_02Aaron Powell The classic code smell.
SPEAKER_01Exactly.
SPEAKER_02And they can't always articulate why right away, which is fascinating because software engineering, you know, it prides itself on being a purely logical, deterministic, data-driven field. Relying on vibes sounds almost mystical.
SPEAKER_01It does sound unscientific, but it is actually the result of massive subconscious data processing. That feeling is built on having seen hundreds of code bases, thousands of methods, and years of grueling production problems.
SPEAKER_02So it's not just a lucky guess.
SPEAKER_01Not at all. Jet isn't consciously opening a mental spreadsheet of 327 incidents. Instead, his system one is instantly pattern matching the current disaster against the abstract shape of every disaster he has ever lived through.
SPEAKER_02Wow. It's associative memory functioning at the highest possible level. But I guess that brings us to the flip side of the coin, right? System one is incredible for familiar problems. It's highly optimized for patterns it has seen before. But what happens when John faces a completely unfamiliar domain? Say the company is uh pivoting to a brand new technology stack, the business requirements are totally novel, and they openly contradict each other.
SPEAKER_01Aaron Powell Well, that is where relying purely on that fast gut instinct suddenly becomes a massive liability. And it's exactly where Economan System Two has to take over.
SPEAKER_02Okay, so what is System Two doing differently?
SPEAKER_01System two is the slower, deliberate, highly analytical mode of thinking.
SPEAKER_02Aaron Powell This is the cognitive, heavy lifting.
SPEAKER_01Exactly. It requires intense, sustained concentration. System two is classical, rigorous engineering. It's the part of the brain that stops and asks, wait, what are the actual business requirements here? Right. What underlying assumptions are we making about data volume? What happens if this specific third-party API just fails? You literally have to calculate, model the architecture in your head, and document all those decisions.
SPEAKER_02But the text points out something very humbling about human nature here, which is that we fundamentally resent using system two.
SPEAKER_01Oh, we avoid it at all costs. Because it is biologically and cognitively expensive. It burns actual calories. Humans just naturally prefer the path of least cognitive resistance. So we look for mental shortcuts to avoid the heavy lifting, which Kahneman calls heuristics.
SPEAKER_02And the software engineering industry is apparently absolutely riddled with these shortcuts. Oh, completely. The author lists some fantastic examples, like instead of deeply analyzing a new system's specific throughput needs using system two, an engineer will just lean on a system one heuristic and say, Well, we used Kafka on the last project and it worked.
SPEAKER_01Right. Or microservices naturally scale better.
SPEAKER_02Or the ultimate classic, we have always done it this way.
SPEAKER_01Yes. And those statements might technically be true in isolation, or you know, they might have been true five years ago. But in that moment, they're being used as substitutes for actual critical thinking. They are literally just a defense mechanism to avoid spinning up system two.
SPEAKER_02Which makes me think about the very fine line between actual hardwone intuition and just um guessing based on a cliche. How does a developer build the right kind of intuition so they aren't just blindly applying buzzwords?
SPEAKER_01Aaron Powell True engineering intuition is forged in the fires of failure. You really don't get it by reading a textbook. Right. You design a system, it fails spectacularly in production, you analyze y and you learn.
SPEAKER_02Yeah.
SPEAKER_01You deploy a feature expecting users to do X and they immediately do Y.
SPEAKER_02Because users always do Y.
SPEAKER_01Exactly. You learn that a relational database that purrs beautifully with 10,000 records suddenly grinds to an absolute halt and locks up when it hits 200 million records. You learn that the theoretical zero latency network you drew on the whiteboard is just a complete lie in the real world.
SPEAKER_02So it's the scars, it's the accumulated trauma of reality constantly crashing into theory.
SPEAKER_01Yes. Over time, those incredibly painful experiences accumulate. Architectural problems that once required agonizing, slow system two analysis basically become instantly recognizable geometric shapes to system one.
SPEAKER_02The author has a great line for this that summarizes it perfectly. Uh intuition is slow thinking that through experience has learned how to become fast.
SPEAKER_01It's a beautiful way to phrase it. But we do have to talk about the trap hidden inside that experience. The text asks a brilliant, somewhat brutal question. Does a senior engineer actually have 20 years of experience, or do they just have one year of experience repeated 20 times?
SPEAKER_02Oof. That is a harsh question, but I see exactly where it's going. Wait, isn't hiring for decades of experience the whole point? Why is experience suddenly being framed as a potential liability?
SPEAKER_01Because if an engineer solves the exact same kind of problem in the exact same environment over and over again, and they never get meaningful corrective feedback on whether their decisions were actually optimal long term, their intuition gets incredibly confident, but it doesn't get accurate.
SPEAKER_02Oh wow. So they become loudly and aggressively wrong.
SPEAKER_01Precisely. And in a corporate environment, seniority naturally creates authority, right? A senior architect might look at a new proposal for a serverless architecture and just say, I know for a fact this will not work, and shut down the entire discussion.
SPEAKER_02But they might be wrong.
SPEAKER_01Exactly. Their system one intuition might be based entirely on technological constraints and network bottlenecks from 15 years ago that simply no longer exist. Their experience has calcified into baggage. What's fascinating here is how the two systems must interact. System one is supposed to provide a hypothesis, but system two must be allowed to challenge and update that hypothesis regardless of how senior the person is.
SPEAKER_02That dynamic between the two systems seems incredibly fragile, especially since human brains are practically wired to find patterns everywhere, even where they don't exist at all. The text dives into how this delicate balance plays out in the daily trenches of an engineer's life, specifically starting with debugging.
SPEAKER_01Debugging is really where this cognitive tag team is put to the ultimate stress test. Humans are natural storytellers. We crave causality and narrative coherence. So let's say a core database is upgraded on Tuesday night, and on Wednesday morning a major deployment completely fails.
SPEAKER_02System One immediately shouts the database upgrade obviously caused the failure.
SPEAKER_01Exactly. Or like a new junior developer joins the team and a week later the defect rate spikes. It's the new guy's fault.
SPEAKER_00Naturally.
SPEAKER_01Right. Maybe it is, maybe it isn't. But our brains instantly build a coherent, satisfying story connecting those two chronological events.
SPEAKER_02And the source material points out this is exactly why debugging can get surprisingly emotional on dev teams. It's not just objective code anymore. Someone develops a theory about the bug, and very quickly it becomes their theory. Their ego gets attached to the narrative.
SPEAKER_01Which immediately triggers severe confirmation bias. Once you firmly believe the Tuesday database upgrade caused the Wednesday outage, you basically welcome any log file or metric that supports that theory. But any evidence that contradicts it, you fight it. You demand more investigation. Yeah. You explain it away as a glitch. The team stops objectively debugging the actual system and starts defending their own System One hypotheses.
SPEAKER_02So what does good debugging actually look like when we view it through Kahneman's lens?
SPEAKER_01Good debugging requires combining both systems fluidly. The application starts acting weird, and System One instantly recognizes the symptoms. This looks exactly like a race condition. That intuition is highly valuable. It gives you a starting point. Right. But immediately, system two needs to step in and apply rigorous friction. It has to ask, okay, if it's a race condition, what specific log entries should we see? Can we design a test to falsify this theory? What other architectural explanations fit these exact same symptoms?
SPEAKER_02Yeah, it makes total sense. It's like a detective getting a brilliant hunch about a suspect that's your system one, but still having to do the boring, meticulous work of actually dusting for fingerprints to prove it in court. That's system two.
SPEAKER_01That is a much better way to frame it. The best troubleshooters generate hypotheses incredibly fast, thanks to their massive library of System One intuition. But they are willing to ruthlessly abandon those hypotheses just as fast the second the scientific falsification proves them wrong.
SPEAKER_02And this interaction doesn't just happen when production is on fire. It happens during routine code review, too. You're looking at a colleague's pull request, the code compiles perfectly, all the automated tests pass, nothing is explicitly broken. But your system one looks at the structure and just goes, Oh, I don't like it, the vibes are completely off.
SPEAKER_01Yeah. Maybe it's a simple convenience function that is slowly bloated over six months into a massive pile of unmaintainable business logic. The experienced engineer smells the structural decay before they can even formally articulate the architectural principle being violated.
SPEAKER_02But as the author rightfully points out, you can't just leave a pull request review comment that says, I don't like it, rejecting doesn't help anyone.
SPEAKER_01No, it really doesn't. If we connect this to the bigger picture, the actual job of a senior engineer during a code review is translating that invisible internal intuition into explicit external knowledge. System one detects the structural discomfort, but system two has to step up and do the hard work of explaining it. Why does this feel wrong? What specific software design principle, like the single responsibility principle, is being violated here? What future feature request is this tight coupling going to make impossible? If the senior engineer doesn't do that slow translation work, the intuition just stays locked in their head, and the rest of the team never learns.
SPEAKER_02Which brings us right back to John and his seemingly disproportionate paycheck. This slow, deliberate process of validating hypotheses, catching subtle code smells before they merge, translating intuition for junior devs. Absolutely none of this looks like typing furiously on a mechanical keyboard.
SPEAKER_01No, it looks like sitting quietly and staring at a screen.
SPEAKER_02So what does this all mean? If avoided work is invisible by definition, how can managers or organizations possibly reward or track it?
SPEAKER_01That is the multi-million dollar question. And it exposes why traditional productivity dashboards are often completely broken.
SPEAKER_02He asks three slightly uncomfortable questions in a planning meeting. He suggests deleting a legacy feature instead of building a workaround for it. Yeah. By the end of the day, John has written exactly 30 lines of code. Meanwhile, a junior developer sitting right next to him has proudly cranked out 800 lines of boilerplate code.
SPEAKER_01But John's true output isn't measured in code volume. Often his absolute highest value output to the organization is avoided work.
SPEAKER_02Avoided work. It's such a powerful concept. Unpack that for us.
SPEAKER_01Well, it is profound because of the leverage involved. John notices an architectural flaw on that whiteboard diagram that, if built, would have made the next five features an absolute nightmare of technical debt. He stops a subtle data exposure security flaw before it ever reaches the production servers. He asks a simple clarifying question about a business requirement, and the product manager realizes the customer doesn't even actually need the feature.
SPEAKER_02So the company isn't paying John for those 30 lines of code. They are paying him for the thousands of catastrophic paths he has walked down before and actively steers the team away from. I won't quote the text directly here because it synthesizes this so perfectly. Thirty years of slow thinking have gradually become 30 seconds of fast thinking.
SPEAKER_01That is the ultimate retirement investment for an engineer's entire career. But it requires a delicate balance. The author introduces a crucial tension here between trusting that deep intuition and utilizing first principles thinking. Exactly. It operates almost in direct opposition to intuition. Intuition says, I've seen a problem roughly shaped like this before, just apply solution X. First principles say, forget the past, strip away the assumptions, let's build the logic from the ground up. And the text argues that the real skill is knowing when to trust intuition and when to use first principles.
SPEAKER_02Like knowing which cognitive tool to deploy.
SPEAKER_01Right. If a database is crashing the exact same way it did 20 times before, use your system one intuition. It saves massive amounts of time. But if you are designing a completely novel distributed architecture for a scale your company has never faced, blindly importing past patterns relying on that System One baggage will blind you to better, more efficient solutions. You have to know when to stop John's gut instinct and say, wait, let's prove why.
SPEAKER_02And just when we think we have a solid handle on the immense value of human judgment and balancing these systems, the text introduces a massive paradigm shift, a technological shift that completely upends the economics of writing code. We are talking about a tool that can write those 800 lines of boilerplate code not in a day, but in seconds.
SPEAKER_01We have to talk about artificial intelligence and large language models.
SPEAKER_02Because AI changes the math entirely. For decades, the sheer physical and mental act of writing the implementation translating logic into type syntax was the expensive, time consuming bottleneck of software engineering. Now any developer can write a prompt and get hundreds of lines of code instantly. Writing code is basically becoming a commodity.
SPEAKER_01It is. And this shifts the entire value proposition of a human software engineer. If generating the implementation is cheap and instantaneous, what becomes valuable? The answer is judgment. Judgment becomes exponentially more critical.
SPEAKER_02Because you have to evaluate what the AI just gave you. You have to ask, does this generated architecture actually make sense for our specific domain? Did the AI solve the right problem or just the problem it thought I asked?
SPEAKER_01Exactly.
SPEAKER_02What hidden assumptions did the model bake into this memory management? What is going to happen when this seemingly perfect function runs at massive concurrent scale?
SPEAKER_01This raises an important question, though. As machines become exponentially faster at producing incredibly convincing solutions, how do you know if it's fundamentally flawed at a structural level? The author argues that you have to rely on that internal alarm bell. Does something about this solution simply feel wrong?
SPEAKER_02We're right back to the code, Smell. System one.
SPEAKER_01Yes. An experienced engineer looks at AI-generated code and their hard-won intuition instantly flags that the abstraction is overly complicated, or that a hidden assumption about database logging is going to cause a catastrophic deadlock down the line. Intuition alone isn't enough to fix it. They still absolutely need system two to investigate, falsify, and explain why it's wrong. But without the intuition, they wouldn't even know where to look in the first place. The AI code just looks too confident, too plausible.
SPEAKER_02Here's where it gets really interesting. When AI coding assistants first hit the scene, the narrative was that they would make coding so easy and automated that companies wouldn't need to pay for expensive veteran engineers like John anymore.
SPEAKER_01Right, that was the big fear.
SPEAKER_02They could just hire juniors to prompt the AI. But it turns out the exact opposite might be true. AI might make John more indispensable than ever as the ultimate supervisor.
SPEAKER_01It is the great paradox of automation. The most vital human skill is knowing when a perfectly convincing solution shouldn't actually be trusted.
SPEAKER_02And that, right there, is why John earns the big bucks. The company isn't paying him for his typing speed or even his encyclopedic knowledge of syntax. They're paying him for his highly calibrated judgment. So, to recap this incredible journey, we started with John fixing a broken system just by sensing the connection pool was off. We explored the complex psychological dance between Kahneman System 1 and System 2, how fast associative thinking gives us brilliant hypotheses, but slow, deliberate, scientific thinking is required to validate them and keep our biases in check.
SPEAKER_01We saw how this dynamic plays out in the emotional trenches of debugging, and why the real job of code review is translating invisible vibes into explicit engineering principles.
SPEAKER_02We wrestled with the invisible but massive economic value of avoided work, how 30 years of slow thinking condenses into come seconds of fast thinking. And finally, we looked at how the rise of AI code generation doesn't replace that human judgment, but actually elevates it to the single most critical skill an engineer can possess in the modern era.
SPEAKER_01It's a powerful, much needed reframing of what it actually means to be an expert in any technical complex field today.
SPEAKER_02It really is. And before we wrap up, I want to give one final reminder. If this discussion made you rethink how your team approaches debugging or how you handle your next pull request, please go check out the original article in the Agile Software Engineering newsletter. The diagrams alone will clarify everything we just talked about.
SPEAKER_01Oh, they're essential.
SPEAKER_02Subscribe to their work, hit that like button on this deep dive, and forward this audio and the text to your lead architect or your engineering manager. It is going to spark exactly the kinds of conversations your team needs to be having.
SPEAKER_01It provides the vocabulary to finally explain the invisible work that holds software systems together.
SPEAKER_02Absolutely. But I want to leave you with a lingering thought. A question that builds on all of this but wasn't directly answered in the text. We've established that true intuition that magical 30 seconds of fast thinking requires 30 years of struggling through slow thinking. It requires making mistakes, deploying bad architectures, and manually writing the code to see exactly how it fails under pressure. So, how will the next generation of engineers ever develop their own System One intuition if they spend their entire early careers just prompting an AI to do all that hard, slow educational work for them? When John finally retires, will the tech industry run out of John's entirely? Keep diving deep. We'll catch you next time.
SPEAKER_00Thank you for listening to the Agile Engineering Deep Dive Podcast. If you found this episode valuable, feel free to share it with someone who might benefit. A colleague, your team, or your network. You can access all episodes by subscribing to the podcast and find their written counterparts in the Agile Software Engineering newsletter on LinkedIn. And if you have thoughts, ideas, or stories from your own engineering journey, I'd love to hear from you. Your input helps shape what we explore next. Thanks again for tuning in. And see you in the next episode.
Podcasts we love
Check out these other fine podcasts recommended by us, not an algorithm.
Darknet Diaries
Jack Rhysider