node created 2012/06/17
last changed 2013/02/04
I cannot tell why the spokesmen I have cited want the developments I forecast to become true. Some of them have told me that they work on them for the morally bankrupt reason that "If we don't do it, someone else will." They fear that evil people will develop superintelligent machines and use them to oppress mankind, and that the only defense against these enemy machines will be superintelligent machines controlled by us, that is, by well-intentioned people. Others reveal that they have abdicated their autonomy by appealing to the "principle" of technological inevitability. But, finally, all I can say with assurance is that these people are not stupid. All the rest is mystery.
What is finished... is the idea that this great country is dedicated to the freedom and flourishing of every individual in it. It's the individual that's finished. It's the single, solitary human being that's finished. It's every single one of you out there that's finished, because this is no longer a nation of independent individuals. It's a nation of some 200-odd million transistorized, deodorized, whiter-than-white, steel-belted bodies, totally unnecessary as human beings, and as replaceable as piston rods... Well, the time has come to say, is dehumanization such a bad word. Because good or bad, that's what is so. The whole world is becoming humanoid - creatures that look human but aren't. The whole world not just us. We're just the most advanced country, so we're getting there first. The whole world's people are becoming mass-produced, programmed, numbered, insensate things...
When once you were in the grip of the Party, what you felt or did not feel, what you did or refrained from doing, made literally no difference. Whatever happened you vanished, and neither you nor your actions were ever heard of again. You were lifted clean out of the stream of history. And yet to the people of only two generations ago this would not have seemed all-important, because they were not attempting to alter history. They were governed by private loyalties which they did not question. What mattered were individual relationships, and a completely helpless gesture, an embrace, a tear, a word spoken to a dying man, could have value in itself. The proles, it suddenly occurred to him, had remained in this condition. They were not loyal to a party or a country or an idea, they were loyal to one another. For the first time in his life he did not despise the proles or think of them merely as an inert force which would one day spring to life and regenerate the world. The proles had stayed human. They had not become hardened inside. They had held on to the primitive emotions which he himself had to re-learn by conscious effort. And in thinking this he remembered, without apparent relevance, how a few weeks ago he had seen a severed hand lying on the pavement and had kicked it into the gutter as though it had been a cabbage-stalk.
Software will absorb new humans as just another object, and map them into some form of homelessness, because that's the most efficient use of us.

Personally I'd feel happier if I grew up on a community surrounded by people that I knew and cared for, instead of being routed around like a tinder hookup, endlessly bidding for cheaper container storage.

I think that's something worth sacrificing a little bit of algorithmic efficiency for.
"The Olive Tree" (1936)
Let nothing be called natural
In an age of bloody confusion,
Ordered disorder, planned caprice,
And dehumanized humanity, lest all things
Be held unalterable!
"The Exception and the Rule" (1937)
Until now the totalitarian belief that everything is possible seems to have proved only that everything can be destroyed. Yet, in their effort to prove that everything is possible, totalitarian regimes have discovered without knowing it that there are crimes which men can neither punish nor forgive. When the impossible was made possible it became the unpunishable, unforgivable absolute evil which could no longer be understood by the evil motives of self-interest, greed, covetousness, resentment, lust for power, and cowardice; and which therefore anger could not revenge, love could not endure, friendship could not forgive. Just as the victims in the death factories or the holes of oblivion are no longer "human" in the eyes of their executioners, so this newest species of criminals is beyond the pale even of solidarity in human sinfulness.

It is inherent in our entire philosophical tradition that we cannot conceive of a "radical evil." And this is true both for Christian theology, which conceded even to the Devil himself a celestial origin, as well as for Kant, the only philosopher who, in the word he coined for it, at least must have suspected the existence of this evil even though he immediately rationalized it in the concept of a "perverted ill will" that could be explained by comprehensible motives. Therefore, we actually have nothing to fall back on in order to understand a phenomenon that nevertheless confronts us with its overpowering reality and breaks down all the standards we know. There is only one thing that seems to be discernible: we may say that radical evil has emerged in connection with a system in which all men have become equally superfluous.
"The Origins of Totalitarianism"
What gets me though is most IT people can't even grok people's facial expressions, but they'll trust anything that claims it measures “the average user”.
Just as bombs follow oil, and drones follow drought, so boats follow both: boats filled with refugees fleeing homes on the aridity line ravaged by war and drought. And the same capacity for dehumanising the other that justified the bombs and drones is now being trained on these migrants, casting their need for security as a threat to ours, their desperate flight as some sort of invading army. Tactics refined on the West Bank and in other occupation zones are now making their way to North America and Europe.
We don't know a perfected totalitarian power structure, because it would require the control of the whole planet. But we know enough about the the still preliminary experiments of total organization to realize that the very well possible perfection of this apparatus would get rid of human agency in the sense as we know it. To act would turn out to be superfluous for people living together, when all people have become an example of their species, when all doing has become an acceleration of the movement mechanism of history or nature following a set pattern, and all deeds have become the execution of death sentences which history and nature have given anyway.
"Elemente und Ursprünge totaler Herrschaft" p. 683
The salvation of the world depends only on the individual whose world it is. At least, every individual must act as if the whole future of the world, of humanity itself, depends on him. Anything less is a shirking of responsibility and is itself a dehumanizing force, for anything less encourages the individual to look upon himself as a mere actor in a drama written by anonymous agents, as less than a whole person, and that is the beginning of passivity and aimlessness.
They do not feel any enmity against me as an individual, nor I against them. They are ‘only doing their duty’, as the saying goes. Most of them, I have no doubt, are kind-hearted law-abiding men who would never dream of committing murder in private life.