From Puzzles to Product

Note: this is a keynote from Velocity Conf 2019, San Jose. A text version of the material follows, and then references.

VelocityConf video

I got into programming as a career because it was easy. I could solve puzzles all day, then go home at 5:30 and drink with my friends. Twenty years later, I stay in software because it is hard. And it keeps getting harder, and more interesting.

I want to take you on that journey with me. How I moved from caring about puzzles to products, about correctness to change, about tech to the domain. And then at the very end, I’ll tell you my secret hope about how the software industry might help the world. (hint: it is not disruption)

Provisioning Puzzles

My first job was in telecom. A customer needs a phone, and to activate it they call customer service. A representative works in the customer service app, which calls out to provisioning on new activations. That was my piece. The switches need to know which phones go with which phone numbers, and we told them.

It was fun! We received requests, split them up and populated them according to instructions in our own external DSL. We forked off processes to talk to each switch — for concurrency! and failure boundaries! — and talked to them over sockets.

I had all kinds of fun puzzles. I could make new functions in the internal DSL, make new switch-talking processes. We even wrote little programs to unit test new functionality… and then threw them away.

For me, it was puzzle solving. Puzzle solving is safe and satisfying.

I knew what “correct” is because someone told me. I had defined means available: we worked in C, on Unix, with the standard library, database drivers, and Oracle. Those are the tools at hand. For bonus points, make the code look like the rest of it.

It is tempting to think this is what software development is like. We certainly interview like it is.

But I’m here now because this ability to solve puzzles in code is only a stepping stone to the real work.

Provisioning Problems

A few years later, we got a new phone company client, a new implementation of our software to customize.

This time I got to participate sooner and at a higher level, not just solving puzzles, but helping define them.

My system view included our customer, including not only the customer service reps but also the business owners who had other needs.

It needs to be Service Oriented Architecture, so we’ll introduce queuing. Now, we still had a shared database, but I didn’t know any better at the time, so I didn’t scream.

And political cover: if we’re going to replace the old system we’d better have feature parity. That’s a whole list of new requirements.

Each concern we took back to the implementation level. But now they’re not puzzles, they’re more flexible than that. It’s closer to problem-solving. I start an implementation, and if it’s not taking this system in a healthy direction, I go back to our customers, and I’m like, this feature the old system had, what do y’all use it for? “Oh, we do this one thing with it.” How about we do that one thing another way?

I zoom in and out from detail to context. Implementation informs design.

We think of the feedback loop as design, implementation, user feedback, more design.

But you, developer, are the first user of the design. And your experience informs it. Design is negotiated in this way.

And since then, since 2001 when I worked on this project, there’s even more impact, because we don’t do most of the implementation ourselves. We import libraries, we pay for software as a service.

How each of these works — how queuing works, how the database works, what a framework supports — informs our implementation and design.

But if all the fun puzzles have been solved already, what do we get to do? Build on them.


To illustrate this, I want to turn this diagram into a map. The difference is, in a map, physical location on the page has meaning. So let’s add some axes.

The vertical axis is visibility. The user is at the top, deep support at the bottom.

The horizontal axis is increasing standardization. Reusability. To the right, there is a strict definition of correct. To the left, more exploring. I can use this to talk about what is worth implementing internally. And to anchor the area of software I’m talking about.

This is called a Wardley map. It is useful for strategy.

At the far right we have utilities. Electricity is here. The network switches are over here, because they’re very well specified. From where I am, We don’t change them, and we trust them to work. The database is close to a utility, because the SQL specification is well defined, and we count on it. Best practices are well established here.

A bit more to the left, we have someone else’s product. Another company releases this library, or provides this service.

Both someone else’s products and utilities represent puzzles that have been solved for us. We can use these as building blocks for everything to the left of them. Stable dependencies. Things I love about products: what we can do with them, and what we don’t have to know.

In Why Information Grows, César Hidalgo defines a product as embodied information. A product enfolds practical knowledge about the world, in a way that we can use that knowledge without having it.

With a pencil, I have the power to write on paper, without knowing how to mine graphite or write it in wood. I get to spend my brain thinking about what letters to write.

The product teaches us how to use it, and after that we get the power. We get the power of queuing without understanding the intricacies of this distributed system.

Software as a service is the best kind of product, because it comes with a team of human experts; it takes humans to make a system resilient. Also I love that someone is making this component of my system better, without me having to think about it.

This is crucial because the limitation of our work is not how much we can do. Our work is not typing. (You can tell, because we don’t type all day, and most of what we type, we delete.) Our work is making decisions. Our limitation is how much we can know. There is a lot to know here!

Only so much fits in a human head. That’s why we have teams. And that’s why we build as much of our system as we can out of someone else’s products. Each of these, each of these nodes in this map, represents a capability. Other people’s products represent capabilities we can get with very little knowledge, and some money. Money is a lot easier to move around than knowledge.

This frees us to focus on what other companies don’t do.

Internal products

We do most of our work over here, in custom software, in systems that are specific to our company.

Over here, the capabilities also embody information. The really good software systems are like an internal product in that way.

The rest of this org has a provisioning capability to activate phone service without details of communication switches, that’s encapsulated in the provisioning software and team.

Think about what capability you’re adding to your organization. Is it something special to your business, or the context or people inside your business? Or is it a capability many businesses need? Because that common stuff, it gets abstracted out and it moves to the right over time. It belongs over here to the right, as we figure out how to do it well it goes this way.

Four years ago I worked at a biotech company. I got to work on creating an internal platform as a service on top of the utilities offered by AWS. Did the company need this? Eeehhh. Was it specific to this company? No, except for some policies encoded in the UI and default setups.

Since then, there’s been tons of PaaS pop up in Someone Else’s Products. Yet, the thing that’s really made the difference, that hit the right level of abstraction, is Kubernetes. It is not a PaaS. It is the platform that you build your platform on. So every company can have a custom PaaS, with the UI and policies that fit in that business, without building as much of the common stuff, the instances and the load balancing etc etc. Now is a much better time to build your PaaS, because big chunks of it have moved to the right.

Custom software is expensive. In money, and also in the knowledge you need to add and grow inside the company.

Why write custom software instead of buying something? we can change it. we can update the software as our business learns.

Because companies these days are made of people, policies, and software. Hell, I’m made partly of software here days. I am not the same person when I don’t have my robot brain. If the software can learn (hint: change and learn are the same thing) then the business can learn, it can get more useful to its customers.

So change is essential. More than essential, it’s the whole point. This is very different from puzzles.

See, puzzles have an end state, they have a done. I don’t want to be done! I want to keep providing capabilities to the organization. The goal state is, still playing!

See, puzzle solving is a closed game. Like a board game, or one game of baseball. There is an end state, a defined means, and a brief feeling of accomplishment.

Product development an open game. We don’t write it, we grow it. Like a career in baseball, or a game of pretend. We want to keep being useful.

And grow more useful, useful to newer to versions of the business. We have at our disposal all the puzzles that have been solved for us, and: change.

Our job is not to write software. It is to change software. And not just code! Our job is to change the system. Change gives us access to every state except this one.

Custom software gives us the potential to change more quickly than utilities or someone else’s product. But we don’t automatically get that ability. What are the obstacles to change? We know about obstacles like code quality. Lack of tests, or too many tests.

It’s easy to focus on internals because we control those. And indeed, some obstacles are there. but most of the meaningful changes are visible from outside the system — they’re new features, or better UI. And you know where the biggest obstacles to change are there? They’re outside the system.

They’re all the people and software with expectations of our software. All the systems bigger than our piece that we are really afraid to break. Software needs backwards compatibility. People need training.

Oh and then there’s the really nasty obstacles that are both internal and external: like when customer service reports reach into our database tables. Suddenly even our internal parts are frozen in fear of breaking systems more important than ours.

This is a process! We are not designing static code. We are designing change.

Designing change is: where is the next increment of where we want to go?

How will the whole system get there? This means data migrations, deprecation, feature flags. Documentation, versioning. How will we bring the adjacent systems along? Get them to use the new features? Get them to upgrade auth so we can improve security. We need to do advocacy with our users.

How will we know whether it had the effect we wanted, and what other effects it had?

With puzzles, if nobody uses it, whatever, I did my bit. With products, with designing change, we care very much.

A positive is: When you think of code this way, feature-flag if-statements aren’t ugly. It’s OK if your code isn’t at the tip of the target architecture. It is the movement that matters. Deprecation is beautiful. Show me the movement.


Movement is harder the more users you have, and the more distant your interactions with them. Look at utilities, who are used by everyone, and everyone has expectations. When S3 goes down, the world has problems.

The bigger the system, the slower the timescale it moves at. Utilities move at the timescale of the industry. In custom software, we can move at the timescale of the one company, if it’s centralized, or the team, if it’s decentralized.

So think about the timescale of change you need. Industry? Company? Team?

For instance, take delivery. Since “change is our job”, delivery is crucial. Designing change only starts with a code push.

Delivery is custom, because it’s contextual. We can’t wait on any other company to accommodate both our existing architecture targets and our next one.

Fortunately, parts of delivery are common. Everybody needs triggering, needs the new code cloned. You don’t need to figure out how to do that: use someone else’s product. Save your knowledge for what is common across the company (run security checks) and specific to a team (format the code). Whatever’s centralized can move at the speed of the company, which can be faster than an external product but will be slower than the speed of a single team. So decentralize what you can.

Think about where the industry is on any particular puzzle solution. Where you can go to other people’s products, where you’re doing something specific to your business, where you need control.

Because control is expensive. Custom software is expensive, and it’s extremely hard.

Hard like rocks, hard like mud

Custom software, growing products, is incredibly hard. Yet “hard” is not specific enough. Solving puzzles is hard. It’s intellectually challenging, it’s work, and it feel satisfyingly productive at the end of the day. It’s hard like rocks. Rock climbing is very technical, and very hard work. And when you get to the top, you feel great! And then what? You go back down. You go home.

Growing a product is hard in a different way. It’s dealing with people, with ambiguity, with probabilities instead of absolutes. It’s hard like mud. Like wading through goopy mud.

Why would you want to wade through mud?

Why does a fish want to swim through water? It’s all around us. Politics, interrelationships, context, ambiguity — this is the world we swim through. If we acknowledge this and work with it, we can add real value, make real change.

We can also find surprises.

At the extreme of wading through mud, beyond finding answers to questions that don’t hold still, there’s finding whole new questions. That “genesis” phase over here on the left of the Wardley map. Here is more ambiguity. Here we can deliver something faster than we can define it. Every time you find an idea you need to bring it back to reality with some sort of test. And then let go of it.

I get to work here now, with our CEO, Rod Johnson. Using the capabilities in Atomist’s Custom column, we’re exploring, what kind of questions can we ask about the condition of code across hundreds of repositories at once?

Once we find a promising problem to solve, then it moves into custom development, where we make it real at scale and connect it with everything else. That is serious work, growing this idea into a product.

This is where I want to be most of the time.

Why move into growing products? Why choose to wade through mud?

The mud is the reality of your business, and this is where you are most valuable to the company. This is where we need the real talented developers, in the core domain that is this company’s secret sauce. The highly specific, challenging to nail down, supple domain work.

Talented developers like to move to the more general, abstract infrastructure projects where they can find some hope of defining “correct.” This gives them status among developers. Gold stars on GitHub. Stuff they can speak about at conferences because it’s not secret, because any company might have these same problems.

In the business domain, you can find the shortest path to meaningful work. I define “meaning” as activity that has significance at higher levels of the system you identify with. Identify with the business you’re part of. This gives you better decision-making power than if you identify generically as “a software craftsman.” You can zoom out to context of use, rather than resorting to generic “code quality” mantras.

This can be very satisfying, if the situation is right.

You need:

  • Visibility into impact. Who is using your product, and which parts? How is it working for them? Are they using it more over time?
  • Efficacy. The work you do goes somewhere. This is why I like working on software that’s already in production.
  • Autonomy. Your team chooses the changes to design and how to implement them, with input from others. Be easy to convince, but be convinced. If you take someone else’s word for what needs to be done next, then you aren’t doing it for the right reasons. You won’t be able to make the decisions correctly, because you don’t have the “why.” You can’t zoom out past “so and so told me to.”

How do we become good at growing products?

  • Be ready to be wrong. Eager, even. “Correct” is the enemy of “better.”
  • Get good at zooming in and zooming out
  • Seek out stories about the history of the software and the business.
  • Make friends who know the business. make friends all over the company.
  • Think about deltas. Design change, not code.
  • Be accountable to tell a story. Not for outcomes – there are a million influences on that KPI besides what you’re doing, and what you’re doing influences a million things besides the API. (Please care about those things.) Tell the story of your team’s decisions and what happened, so the whole organization can learn.Your intentions are not revealed by any one thing you do, but by your responses to the world’s responses to what you do. Action is a process.

I don’t know all the technology, but I know something about the business I’m in.I can’t find the solution, but I’m learning to design change.I can’t solve the puzzle, but I can wrestle with problems.

In software, we get the chance to change complex systems at a rate humans never could before. We get the opportunity to learn about complexity, in a laboratory where we can inject clues for ourselves, to find out how it’s working.

My secret hope is: if we can get good at designing change in a system too complex to understand, then maybe that will help us, as a species, to change our world without destroying it.

References & Further Reading

  • Products and knowledge as a limitation: Why Information Grows by César Hidalgo
  • Closed and open games: The Grasshopper, by Bernard Suits
  • Infinite games: Finite and Infinite Games, by James Carse
  • If the business doesn’t believe in growing products: Project to Product, by Mik Kersten
  • John von Neumann and Norbert Wiener, by Steve J Heims
  • How Reason Almost Lost Its Mind, by Erickson, Klein, Daston, Lemov, Sturm, Gordin