Philosophy Discussion Forums | A Humans-Only Club for Open-Minded Discussion & Debate

[Ormond]

Thanks for the intelligent posts above from everyone, and thanks to Steve for re-energizing this discussion. A point by point reply to all the posts above would quickly become incredibly wordy, so perhaps the following will be more efficient at focusing the conversation.

QUESTION: Do members think that we can create...

1) ever more...
2) ever greater existential scale powers, at an...
3) ever faster rate, and...
4) manage them all successfully...
5) every day, forever?

Existential scale power is defined as any technology which could bring down modern civilization. Successful management is defined as avoiding such a collapse. (As Steve suggest, species extinction is set aside.)

Members are intelligently comparing the benefits of knowledge development against the risks in a rational manner. Ok, that's good, but...

What members are failing to include in their calculations is that unless you reject my theory completely and answer the question above affirmatively, then there aren't going to be any future benefits to weigh in your calculations. Yes, if we continue with the status quo we will eventually cure cancer etc, and that's a good thing, agreed. But what difference does this make if 32 years later we crash modern civilization?

Focusing the conversation:

ACCEPT: If you accept my theory, then you face the challenge of coming up with radical solutions which match the revolutionary nature of the situation we already find ourselves in, being 30 minutes away from civilization collapse.

REJECT: If you reject my theory, then you face the challenge of showing how human beings will successfully manage a growing number of existential scale powers which emerge at an ever faster rate, the logical outcome of allowing the exponential nature of knowledge development to continue.

It may help to focus the conversation if members will accept or reject my theory, and then make the arguments which are required for the position which has been chosen.

Steve, can you aim us at members of the scientific community who are discussing this issue?

[Fooloso4]

Steve:

But if there were no more scientific advances then I doubt whether we would be extinct in anything like the foreseeable future.

If we are unable to curtail pathogens then it is a possibility, although large scale die offs might be more likely than extinction. The technological link is the ease of travel.

Antibiotic resistance is a growing problem. The role of the development of antibiotics is instructive. It is a clear case of where knowledge is beneficial but the ignorant use of that knowledge has caused new problems.

[LuckyR]

Thanks for the intelligent posts above from everyone, and thanks to Steve for re-energizing this discussion. A point by point reply to all the posts above would quickly become incredibly wordy, so perhaps the following will be more efficient at focusing the conversation.

QUESTION: Do members think that we can create...

1) ever more...
2) ever greater existential scale powers, at an...
3) ever faster rate, and...
4) manage them all successfully...
5) every day, forever?

Existential scale power is defined as any technology which could bring down modern civilization. Successful management is defined as avoiding such a collapse. (As Steve suggest, species extinction is set aside.)

Members are intelligently comparing the benefits of knowledge development against the risks in a rational manner. Ok, that's good, but...

What members are failing to include in their calculations is that unless you reject my theory completely and answer the question above affirmatively, then there aren't going to be any future benefits to weigh in your calculations. Yes, if we continue with the status quo we will eventually cure cancer etc, and that's a good thing, agreed. But what difference does this make if 32 years later we crash modern civilization?

Focusing the conversation:

ACCEPT: If you accept my theory, then you face the challenge of coming up with radical solutions which match the revolutionary nature of the situation we already find ourselves in, being 30 minutes away from civilization collapse.

REJECT: If you reject my theory, then you face the challenge of showing how human beings will successfully manage a growing number of existential scale powers which emerge at an ever faster rate, the logical outcome of allowing the exponential nature of knowledge development to continue.

It may help to focus the conversation if members will accept or reject my theory, and then make the arguments which are required for the position which has been chosen.

Steve, can you aim us at members of the scientific community who are discussing this issue?

Your post and thread remind me of riding the chair lift. You know how if you are at an amusement park and are going to ride any sort of mechanized ride (more adventurous than the merry go round), the car will come equipped with a seat belt, a bar across your midsection and probably a door enclosure. Obviously it is for safety. No one wants to take the chance that some kid (or an extremely careless or drunk adult) is going to slip out of the seat and fall to their death, or worse. Makes complete sense, it's totally logical and amusement park incidents are uncommon but not rare. OTOH, at a ski resort, folks who drink alcohol served on the premises, routinely get on a bench seat, no seat-belt, no mandatory lap bar, no door enclosure and ride way, way the hell up in the sky, winds can be howling, swinging the chairlift back and forth. I suppose there are skiers who fall to their deaths but it is rare enough that the lawyers haven't mandated a change in practice. WHY IS THAT? It makes no sense when observed superficially. However, it does make sense when you realize that your average skier is tons more responsible in this way than your average amusement park patron (regardless of age). Same thing, the folks in charge of the nuclear arsenal are way more responsible than your average person... until recently... OK I see your point.

[Nick_A]

I began following this thread and would like to comment.

Indeed knowledge can be detrimental. It becomes detrimental when our Being does not come close to the level of our knowledge. By 'Being' I mean, literally, what we are. If what we are is savages, then knowledge of how to build nuclear weapons can be very detrimental. This means that as Knowledge grows Being must grow along side it. And if it falls behind beyond a certain critical threshold then it guarantees our destruction. Our level of Being falling way below our level of Knowledge means that in essence we are apes with machine guns. And this is not a desirable state at all....

You are right IMO and one of the horrors of the human condition is that egoism associated with it prevents us from realistically and impartially examining the human condition within ourselves. I've really come to believe that the survival of humanity will depend on this minority who have questioned as you just did and acted upon it. It isn't a matter of denying science but of acquiring a human perspective in which it can serve Man as opposed to Man serving the scientific expression of technology.

You are probably aware of the esoteric meaning of Noah's Ark. Water refers to knowledge which can consume us. The ark refers to a quality of consciousness which remains on top of the flood. As the waters of knowledge and their expression through technology consume humanity, will there be enough of a conscious influence to avoid the lawful results of the most horrific catastrophe? Logically I doubt it. Of course I have hope but as a chess player this situation looks like a simple mate in three. No amount of speeches will change it. The position is what it is. I don't see what new variables will appear that can change it. It seems we will have to hit a collective bottom and start again if we are still able.

[Ormond]

Same thing, the folks in charge of the nuclear arsenal are way more responsible than your average person... until recently... OK I see your point.

Actually, there have been a number of near misses throughout the years. The Cuban Missile Crisis is just the most famous. As example, once somebody mistakenly loaded a training tape in to the NORAD system, and for a few precious minutes the entire government thought a massive Russian attack was incoming. The documentary Countdown To Zero offers other such near miss examples.

But the larger point is that even if we fixed the nuke problem completely, which would be great, it only buys us time until more existential scale technologies come online. As the knowledge explosion continues to accelerate, there will be less and less time available between one new existential scale technology and the next. Nobody can predict what exactly will happen when, but the bottom line is ruthlessly simple. More and more knowledge at a faster and faster rate equals accelerating levels of power, and thus less and less room for error. If we stay on the current path, sooner or later we'll arrive at the situation we can't handle, it's only a matter of time.

-- Updated February 6th, 2017, 3:57 pm to add the following --

A great imaginary sage once indignantly bellowed....

When readers realize that there are no rational arguments for unrestrained knowledge growth, and that they can't defend the foundational premise of science culture, they quietly vanish. Not just on this forum, on all forums, and throughout our culture, to the highest levels...

[Steve3007]

Ormond:

QUESTION: Do members think that we can create...

1) ever more...
2) ever greater existential scale powers, at an...
3) ever faster rate, and...
4) manage them all successfully...
5) every day, forever?

I think it would be impossible to do anything other than ACCEPT and then discuss both the premises and conclusions of the above.

The development of nuclear weapons is often used as an example because it is an easily understandable threat of collapse. But as knowledge and technological ability grows then clearly the general ability to create and to destroy grows in equal measure. As you have pointed out, there is a basic asymmetry here in that a single large act of destruction halts the entire process in a way that a single large act of construction does not. If we think of it using the simple metaphor of a pendulum swinging with greater and greater amplitude left and right (construction to one side, destruction to the other), then perhaps we can visualize that asymmetry as a big switch placed on just one side of the pendulum, but not the other, which when hit by the pendulum causes the entire experiment to be destroyed, halting any future swinging!

ACCEPT: If you accept my theory, then you face the challenge of coming up with radical solutions which match the revolutionary nature of the situation we already find ourselves in, being 30 minutes away from civilization collapse.

Well, in that previous post I started talking about a vague and not particularly radical thing that doesn't seem to be an obvious solution. But maybe it's a start.

Steve, can you aim us at members of the scientific community who are discussing this issue?

Not without doing pretty much the same research that others would have to do. Like many people I studied a physical science at University and work in an engineering/technology area, but am not really involved in an academic environment in which I have any special insight into how or whether this subjects is being discussed in anything that might be called the scientific community.

-- Updated Mon Feb 06, 2017 11:14 pm to add the following --

To me it seems clearly true that our ever increasing knowledge of how to both create and destroy on ever larger scales will indeed lead to a catastrophic destruction which will cause the survivors to take a very long time before they reach the end of the next cycle and do it all again. But if we can't find any concrete practicable solutions then, no matter how serious it is, we're forced to be fatalistic and resort to gallows humour. Just treat it as one of the many things that are going to wipe us out, like that big volcano under Yellowstone Park, or the next big meteor impact, or "grey goo" or homicidal robots from the future (who, ironically, could never be president because they weren't born in the USA).

[Ormond]

The development of nuclear weapons is often used as an example because it is an easily understandable threat of collapse.

Yes, if nuclear weapons can be said to have a value it is that they are simple. A box that goes boom.

But as knowledge and technological ability grows then clearly the general ability to create and to destroy grows in equal measure.

I agree that the further development of knowledge will bring new tools for managing other tools. I agree that this process will be successful most of the time. But...

As you have pointed out, there is a basic asymmetry here in that a single large act of destruction halts the entire process in a way that a single large act of construction does not.

Yes, that's it. The vast scale of existential powers renders the old formula of two steps forward, one step back obsolete. We used to be able to make mistakes, learn from them and adjust, and then keep moving forward. What's changing is that as the scale of the powers increases, the room for error decreases.

Well, in that previous post I started talking about a vague and not particularly radical thing that doesn't seem to be an obvious solution. But maybe it's a start.

Sorry, I didn't mean to ignore your comments about education. That's what I'm arguing for too, but probably on a different scale. Your idea seems much like what the environmental movement has been doing since the sixties. It is having an impact, but SUV sales are still through the roof. I'm skeptical we have enough time left to see such a process play out over many generations. Thus...

I prefer education in the form of the President announcing 50% of federal research dollars will be redirected to nukes and climate change, ie. fixing past knowledge based mistakes, and proving that we are capable of doing so. This would raise a huge controversy and public discussion which would at least suggest the urgency of the situation. Even if such an effort failed as it likely would, we'd at least be on the right subject.

Not without doing pretty much the same research that others would have to do. Like many people I studied a physical science at University and work in an engineering/technology area, but am not really involved in an academic environment in which I have any special insight into how or whether this subjects is being discussed in anything that might be called the scientific community.

Ok, I understand. Well, keep your ears open. I see a good deal of discussion by experts on limited topics such as the ethical challenges presented by genetic engineering. But I don't see anybody anywhere addressing the big picture as we are here. Thus, like you, I'm at a loss as to how to elevate the discussion.

To me it seems clearly true that our ever increasing knowledge of how to both create and destroy on ever larger scales will indeed lead to a catastrophic destruction which will cause the survivors to take a very long time before they reach the end of the next cycle and do it all again.

To me, the best hope may be mini-disasters that wake us up without crashing the whole system. As example, consider the American reaction to 9/11, two wars launched at the cost of something like 5 trillion dollars. And that was after only 3,000 dead. A terrorist nuke attack or small scale nuclear war somewhere might impact the group consensus in a dramatic manner and cause some major re-evaluations. Maybe. Dunno....

But if we can't find any concrete practicable solutions then, no matter how serious it is, we're forced to be fatalistic and resort to gallows humour. Just treat it as one of the many things that are going to wipe us out, like that big volcano under Yellowstone Park, or the next big meteor impact, or "grey goo" or homicidal robots from the future (who, ironically, could never be president because they weren't born in the USA).

Yes, this is probably the most common reaction to such ideas, other than just ignoring them entirely. This is actually in line with my idea that knowledge is best seen not as a human creation but as a force of nature, like the Yellowstone volcano. We aren't driving the wave of knowledge, but riding it.

I'll be dead soon myself, so part of me is content to leave it at gallows humor. Another part of me remembers my parent's generation and how they stood up and faced the fascists in the face of near overwhelming odds and didn't shrink from a major challenge of the 20th century. Maybe their children and grandchildren have become too fat and soft and just don't have the grit to face the challenge of the 21st century. Trying and failing is one thing, giving up without a fight is another.

[Sy Borg]

But if we can't find any concrete practicable solutions then, no matter how serious it is, we're forced to be fatalistic and resort to gallows humour. Just treat it as one of the many things that are going to wipe us out, like that big volcano under Yellowstone Park, or the next big meteor impact, or "grey goo" or homicidal robots from the future (who, ironically, could never be president because they weren't born in the USA).

Yes, this is probably the most common reaction to such ideas, other than just ignoring them entirely. This is actually in line with my idea that knowledge is best seen not as a human creation but as a force of nature, like the Yellowstone volcano. We aren't driving the wave of knowledge, but riding it.

Agreed x 2. It's the tragedy of the commons. Recently a relative pointed me to a Joe Rogan interview with Sam Harris about AI. Sam H and others worry the technology could easily spiral out of control because the strategic advantage for the nation that first develops a general AI will be significant. SH thinks it would be pretty well game over in a short time because by the time the AI has self improved a number of times in a short time it could be present throughout most networks, in which case, as with The Terminator's Skynet, the only solution would be to shut down everything, which is not solution at all. It's not as though there's a lot of jungles and bountiful forests to support urban novice survivalists.

The knowledge was always going to grow. Any group that tries to check its knowledge growth is a group destined to be taken over by more technologically advanced counterparts.

[Ormond]

Hi Greta the Great,

SH thinks it would be pretty well game over in a short time because by the time the AI has self improved a number of times in a short time it could be present throughout most networks, in which case, as with The Terminator's Skynet, the only solution would be to shut down everything, which is not solution at all.

I don't see why the Internet and AI couldn't be shut down. We had neither most of my life, and life could still be grand. When I was growing up in a medium sized American city we had access to 2 TV stations, one radio station, one newspaper, and a dozen or two national magazines. That was our media landscape, and nobody cared or complained. We put men on the moon using computers less powerful than what's in your cell phone.

Ideally we would develop AI and put it to constructive use. But if that's not going to work, then why build it?

Substantial people seem to be worried that hackers will take down our electric grid. Ok, but why is our electric grid connected to the Net??? Again, most of my life there was no Internet except for experimental military purposes, and everybody still had electricity. Nothing in the average person's life was connected to the Net prior to 1994 or so, and people still fell in love, got laid, raised children, pursued their dreams etc etc.

The knowledge was always going to grow. Any group that tries to check its knowledge growth is a group destined to be taken over by more technologically advanced counterparts.

I agree this is an entirely reasonable position, but propose it's unworthy of a philosophy forum, and our culture in general. What phenomena like AI and the Net prove is that we are capable of remarkable achievements once we put our mind to something. We've done the seemingly impossible again and again and again.

It simply isn't rational to embrace defeat and sit around waiting for the end to come before we've even tried to meet the challenge. With this mindset we don't even need existential scale tools because if everyone concludes the game is up and there's nothing we can do about it, that perspective then becomes the engine of our demise. What's the point of planning for the future if we've concluded we have no future?

Why do the U.S. and Russia limit their nuke production? Why not make as many bombs as possible, wouldn't that give the best producer the advantage? We limit our nukes, we don't do something we could do, because we and the Russians recognize it's in our self interest to not compete to the fullest degree possible.

The problem as I see it is that we don't yet have that realization in regards to the larger picture, knowledge development as a whole. That doesn't mean we can NEVER have such an understanding, it just means we don't have it now, at least on the scale necessary.

So, I reject defeatism. It very well might be true that this challenge is more than we can handle, but it's better to go down fighting than to just sit around waiting for the end to come. At least then our final days, if that's what they are, can be filled with hope instead of despair.

And honestly, I truly think that the defeatist attitude, not just here but throughout our culture, is mostly just an excuse to avoid some hard work. We can't find the solution to an epic historic challenge in our first hour of effort, so let's just quit and die, cause that's easier.

So here's a specific game plan. If you 1) feel this theory is generally correct, and 2) if you reject defeatism, the next step might be....

How do we elevate this conversation off one of the world's most obscure forums on to a larger stage?

[Nick_A]

What is a man to do? It is bad enough when both his wife and his girlfriend cheat on him but what if science lies to him as well? This problem needs more than good scotch to deal with. Would you believe that there is deception in global warming statistics? I know, who could do such a thing - but it happens. Yes, sometimes a man must deny his wife, his girlfriend, and science. But fear not; you can always trust me. Send money.

http://www.foxnews.com/science/2017/02/ ... arges.html

[Steve3007]

Ormond:

How do we elevate this conversation off one of the world's most obscure forums on to a larger stage?

I think one thing that we need to do is find something "concrete" to say. A proposed change to government funding and education policy that people can definitely either agree or disagree with. As a starter, you've suggested that 50% of US federal research funding could be redirected:

I prefer education in the form of the President announcing 50% of federal research dollars will be redirected to nukes and climate change, ie. fixing past knowledge based mistakes, and proving that we are capable of doing so. This would raise a huge controversy and public discussion which would at least suggest the urgency of the situation. Even if such an effort failed as it likely would, we'd at least be on the right subject.

OK. People will certainly have a view as to whether any such thing is likely to actually happen, particularly under the present administration. But I think we can set aside, for now, considerations of what is actually likely to become government policy and talk first about what we think ought to be government policy. Once we've done that we can figure out how it might be brought about. But I think we're still a long way from doing that. It's not at all clear how this massive education program about threats like nukes and climate change would work. It's not clear precisely what state of mind we want to bring about in the students. I've tried a bit myself but am so far just as vague as you are about it. But with a slightly different emphasis, I think.

(also from your previous post)

Your idea seems much like what the environmental movement has been doing since the sixties.

In the post to which you're referring here I wasn't really suggesting education specifically on the dangers of any particular threat. I was suggesting a more joined-up approach to teaching political history and science so that people can better understand the rational reasons why decisions are made which have lead to both the benefits and the dangers of technology. As I was saying, I think it's only by realizing that the nuclear arms race (for example) is not a symptom of human madness but a symptom of humans rationally reacting to the concerns of their time that we can unpick such things. In order to change course we have to first properly understand the reasons for the present course. Greta touched on a part of it with her comments about the "tragedy of the commons".

I take on board what you said earlier about how we may not have time for several generations of education on this. But on the issue of timescale I really don't think we have a choice. One thing that I've learnt from various experiences in project management is that if you're constantly watching the clock you end up creating an inferior product that isn't fit for purpose and which takes even more time to fix.

That's still a vague and incomplete post. But it'll have to do for now.

[Ormond]

Hello again Professor Steve-o,

I think one thing that we need to do is find something "concrete" to say.

Ok, in that spirit here's a concrete proposal for your review.

1) First, we might work together to write a document that outlines this theory.

2) Next, we might make that document the centerpiece of a new blog and/or forum dedicated to this topic.

3) Finally, we might focus on expanding this conversation beyond the three or so people currently engaged here. If we're going to meaningfully address such a fundamental human issue as this, we certainly need to bring far more brainpower to the table.

Implementing the above would involve a lot of very concrete work, but it wouldn't necessarily involve the participants in this thread being the ones to deliver the solutions.

The bottom lines I see are 1) does a person understand and generally agree with this description of the problem and 2) do they care? Both of these conditions need to be met before a person will consider investing effort in to the subject.

As example, I'm solid on #1, but pretty soft on #2. Thus, if I'm going to invest effort in this I need assistance from others in helping me stay engaged. On the other hand, I can provide assistance to those who are solid on #2 but soft on #1. Everybody has a role to play, none of us are going to fix this on our own.

It's not clear precisely what state of mind we want to bring about in the students. I've tried a bit myself but am so far just as vague as you are about it.

Perhaps the focus might be on the idea that there is no free lunch, everything of value has a price tag. In the case of thought and knowledge, we get huge benefits, and a correspondingly large price tag. A place to start maybe...

In the post to which you're referring here I wasn't really suggesting education specifically on the dangers of any particular threat. I was suggesting a more joined-up approach to teaching political history and science so that people can better understand the rational reasons why decisions are made which have lead to both the benefits and the dangers of technology. As I was saying, I think it's only by realizing that the nuclear arms race (for example) is not a symptom of human madness but a symptom of humans rationally reacting to the concerns of their time that we can unpick such things.

I hear you, and don't disagree, but to go all philosophical for a moment, I might put it a bit differently. I don't see "madness" and "intelligence" as being polar opposites.

Some people are "mad" because they are too intelligent to fall comfortably in to the group consensus dream. As example, thanks to our dullness, most of us can keep death at a safe distance psychologically. Some people are too intelligent to do that, so their real world functioning may be negatively affected, ie. they appear "mad".

And intelligence, ie. thought, is the primary source of madness. Madness is built in to thought, which is why very intelligent highly educated people with good intentions can find themselves doing things like building nukes. Seen this way, holistically, the act of building nukes was both rational and mad, at the same time. But, this is admittedly a quite esoteric point which will likely do more to muddle the waters than add clarity, so I'm happy to let it go and move on.

I take on board what you said earlier about how we may not have time for several generations of education on this. But on the issue of timescale I really don't think we have a choice.

It depends. If we are going to reason our way out of this box, then yes, a great deal of time will be involved. But I predict there is pretty much no chance that reason alone can address this, and I see some big crisis coming to our aid sooner or later. Look at how America's relationship with the world changed in a single day on 9/11.

Thus, one game plan can be to quietly prepare for the coming crisis so that when it happens we don't have to start thinking about all of this from scratch. Prepare the ground so to speak. At least that seems the most realistic prospect to me at the moment.

Thanks again for engaging! I predict Nobel Prizes for both of us!

[Fooloso4]

How do we elevate this conversation off one of the world's most obscure forums on to a larger stage?

It is a conversation that is already going on. It has gone on for as long technology has altered the ways of life of human being. More recently:

Biological sciences: there are a variety of issues in genetics and stem cell research, which are mainstays of courses in bioethics, and have been widely discussed in both the media and government committees. In addition to discussing such matters both the government and research organizations have developed guidelines and prohibitions.

Artificial Intelligence: all one needs to do is Google potential dangers of AI. It is widely discussed.

[Ormond]

It is a conversation that is already going on.

Perhaps the mods will allow me to reply to your highly predictable gotcha post (you are capable of much greater contributions) with one of my own. The point you are making has already been made above, and already agreed to. What you're saying here has already been said, by me, in this thread, just a few posts back. It is agreed that there are conversations underway in various fields regarding ethical concerns about different technologies etc. We don't need to debate this obvious fact.

If you know of conversations that stand back from the particulars of this or that field and address the knowledge explosion as a whole, and where it is leading, then please share the specifics, as you are usually asking us to do. Where exactly might we find such conversations? This isn't a debating point, I'd sincerely like to know so I could travel there and attempt to join in.

Here's why a broad focus on knowledge matters....

Say we figured out how to get rid of nukes entirely. Imagine that all concerns about genetic engineering and AI and nano-bots, indeed all currently imagined technologies, were somehow completely resolved. All of that would be great, but it doesn't solve the problem, because the knowledge explosion continues to race forward at ever greater speeds. Thus, the problems just hypothetically solved in the last few sentences are almost immediately replaced with new challenges.

So, if our focus is limited to this field or that field, we will find ourselves trapped in a never ending game of whack-a-mole, a game we will eventually lose.

One problem with the broader focus that I would agree to is that addressing knowledge itself, instead of just this or that particular technology, makes the conversation more abstract, which tends to limit the number of people who can and will engage.

I see a useful role for philosophers here in the art of translating the abstract which comes naturally to us in to more accessible language suitable for a wider audience. We should be aiming for the simplest more direct common sense every day language that can explain the challenge, instead of using the topic as a vehicle for displaying how educated and sophisticated we are, or think we are, or wish we were etc.

I don't object to the process of challenge, I welcome it, am begging for it. But please try to make it constructive, somehow leading towards a solution however imperfectly, instead of merely reactive.

[Steve3007]

One thing that does appear to have started to change recently in the US is people engaged in some areas of scientific research becoming more active politically than they previously were. At least so I've read. The apparent determination of the Trump administration to pretty much kill off the concept of environmental protection has perhaps forced them into action. Maybe every cloud has a silver lining?

Anyway, more on some of Ormond's points:

Why do the U.S. and Russia limit their nuke production? Why not make as many bombs as possible, wouldn't that give the best producer the advantage? We limit our nukes, we don't do something we could do, because we and the Russians recognise it's in our self interest to not compete to the fullest degree possible.

The problem as I see it is that we don't yet have that realisation in regards to the larger picture, knowledge development as a whole. That doesn't mean we can NEVER have such an understanding, it just means we don't have it now, at least on the scale necessary.

If we imagine trying to enact a kind of "knowledge limitation treaty", analogous to a nuclear arms limitation treaty, then perhaps we can explore the differences between the two.

The obvious one is that knowledge cannot be simply quantified in the way that nuclear weapons can. The sides in an arms limitation treaty could agree to limit themselves to some quantity of megatons of TNT equivalent each. The obvious equivalent with knowledge might be to agree not to research a particular area. But which one(s)? The fact that we don't know what technology will result in the next nuke-equivalent super-weapon means that, by definition, we don't know which currently obscure area of research will lead to it.

When Ernest Rutherford was firing alpha particles at gold leaf or, going further back, when Faraday was laying the groundwork for the whole of electromagnetism, they couldn't possibly have dreamed that this would lead to Hiroshima. How far back do we decide to nip potentially dangerous technologies in the bud? Should we have forgone the benefits of electric lighting in order to stop nuclear weapons?