Biological evolution is a processing of ideas. Each gene represents an idea. More useful ideas win against other ideas, and new ideas rise out of these. Each builds on the ideas before it. Where is it all going?
It isn’t. These ideas came from somewhere, but they are not going anywhere in particular. There is no larger goal, only useful ideas crowding out less useful ideas. When Darwin put forth the theory of evolution, people hated it for this. They wanted humanity to be a pinnacle, a destination. Nope: we’re just one particularly useful idea.
When ideas are bigger than one person, these also build from other ideas, rather than building toward a goal. Science is ideas that sprang out of earlier ideas, and the most useful ideas won out. Where is science going? Look at old science fiction and it’s pretty clear: we have no fucking clue. Fifty years ago people thought we’d be populating space, flying in cars, teleporting. Instead we have cell phones and the internet. Scientific communities, like bacteria, grow ideas in an evolutionary style. There are small steps and big leaps, but each advancement builds on the ones before in a direction we can’t predict.
Biological computation, mechanical computation, and community computation – it is all consciousless information processing. In the information processing we do in our brains, there is one huge difference: imagination. We can look forward. We can plan and build toward something. Everything else in nature, at scales larger and smaller than us, can only take the next step it finds useful.
No wonder the scale of the individual human is special! It is only we who can aim for something even if life gets worse before it gets better. Consciousness, reason, free will: it is all about imagining consequences and choosing among them.
Meanwhile, to make our lives easier in software, remember that the natural order is unable to see the big picture. Constant improvement is the rule, small useful changes so that the project won’t be snapped up by some predator before it can germinate. A community can’t see the end goal, although some individuals may. Let your project evolve with continuous delivery of useful features, even if that is less efficient than driving through the valley to get to the mountaintop. Build from ideas to better ideas, and life will go on.