Andrian,
My criticism of this model would focus on criticizing points 4 and 5.
Thanks for addressing the topic, I do appreciate it.
My criticism of point 4 is that we don't know how many of these "existential scale powers" as you call them are actually possible.
That's true, agreed. But then we only need one such power, and we already have one. Genetic engineering and nano-tech come to mind as possibilities. And your argument can just as easily be used to support my case as challenge it. Not knowing how many such powers will emerge could also mean there will be many.
Consider the environment all new powers now operate in. At the turn of the 20th century most humans lived on the land and knew how to garden, hunt, forage etc. Now most humans are urbanized, and few know how to do more than swipe a credit card at the grocery store. In a crisis great masses of people will hit the panic button with days of the disruption, and that can become an existential power in itself. Point being, we are far more vulnerable than we were previously, thus it now takes less power to upset the civilization apple cart.
My criticism of point 5 is that what may be an "existential scale power" today may not be tomorrow. If, for example, we develop space colonies of some kind, nuclear weapons become a much less credible threat to all of civilization.
Ha, ha! This comes up a lot. I first heard it from some real scientists on a science forum, who were absolutely sure we definitely could NOT edit our relationship with knowledge, but were confident of colonizing the galaxies.
But to address your point, it's true, and has already been granted, that we very well may successfully manage a number of existential powers. However to focus on that is to miss the point entirely. Again, all that is needed for chaos is that
one existential power run out of control one time. Should that happen, it won't matter a bit that a hundred other such powers are being safely managed.
What you're really proposing, as is everyone else who comments on this topic, is that we can create an unlimited number of powers of existential scale and successfully manage them every day forever. Respectfully, there is simply nothing in human history that suggests such a perfect record of success is at all realistic.
I understand where you're coming from with your belief that "more is better" when it comes to knowledge might need to be changed, but I think that it's not actually practical to adopt any other approach.
We've already done it with food. For endless centuries we had a "more is better" relationship with food, due to chronic scarcities and hunger. Then we entered a revolutionary age where food is plentiful across wide swaths of the planet, and now we are learning to adopt a new more nuanced relationship with food. Eating as much as we can as often as we can is now seen to be a major threat to health.
Again, you're making all the same mistakes that are always made on this topic. You've thought about this for a few minutes and don't see a solution, and thus are concluding no solution is possible. Next you will ask me if I have a solution, and when I tell you I don't, that will close the deal and you will walk away satisfied that the status quo is correct. But, the fact that nobody currently has a solution simply doesn't prove that we don't have a problem.
First, it's hard to know in advance which discoveries will lead to which applications. Who could have predicted, for example, that studying bird feathers under a microscope would lead to the invention of velcro? In addition, knowledge is sort of a "high risk, high rewards" kind of game. Our knowledge of radioactivity may have led to nuclear weapons, but it's also brought us nuclear medicine, nuclear power, insights into astronomy and astrophysics, and radiometric dating.
And on the day that the nukes arc over the poles, or some other existential scale power runs wild, all of the benefits of the science age will vanish. Those who argue for the status quo are casually putting all those benefits at risk, nearly guaranteeing their destruction.
In addition, I don't think it's feasible to try and restrict the kinds of things we research.
Nature has a very simple but ruthless rule.
Adapt or die.
We live in revolutionary times which will only become ever more revolutionary at ever faster rates, due to the exponential nature of knowledge development.
The critics of this theory are essentially saying, let's not adapt to this radically new environment, let's keep doing things the same way we always have. The great irony is that they see themselves as supporters of change and progress, when really they are clinging to an outdated stagnant mode of thinking that can only work so much longer.
In it's essence this theory is incredibly simple.
Why do we not give guns to kids? Because we know if we did it would be only a matter of time until something catastrophic happened, due to the scale of power a gun provides, where one mistake can spell the end.
As adult members of the modern world, we're in exactly that situation. No one can say how or when, but if we keep giving ourselves bigger and bigger powers at faster and faster rates sooner or later someone will make a mistake.
I have adjusted my view over the years after having discussed this so many times. I now have little hope we can reason our way out of this box, and so instead pray for a near miss that will wake us up and give us a chance to learn and adapt. If the near miss is scary enough we might see a radical change in global consciousness at an impressive rate.
Thank for enduring my rant!
If the things we want to hear could take us where we want to go, we'd already be there.