My blood runs cold and blue; I have a dream of red, red as life and passion. Come then, muse: if not my own, let it be one that you may have for another. Awake my blood: I will not let you die.

A Reflection on Psychotechnology and the Information Revolution

Modern civilization has always been immersed in warfare. However, in the past 200 years the weapons and theaters of this war have significantly shifted, as have the targets. Prior to the nineteenth century war was generally fought by humans against humans, either individually or collectively, usually in some defined geographic area. During the last two centuries warfare has taken on many new dimensions, expanding both in terms of space and in terms of participants. Technological advancements have brought forth new weapons such as the atom bomb, bacteriological warfare, and robotic warriors. Now the theatre of war has also become global, extending into space and often operating below the level of human awareness.

War can now be waged on an unconscious level, by manipulating symbols, promoting memes, and gaining control over the mental life of large populations—all accelerated by the large scale deployment of artificial agents across networks. In effect, psychological operations can be used to bring about social and political change without any bloodshed. The technology of information and communication is crucial to such a form of warfare, and so it is appropriate to reflect upon the new kinds of information hazards which arise from these conditions.

There are at least two ways in which our information and communication technologies can become instruments of mental warfare. The first, which might be called “information subversion,” involves an alteration of existing systems of symbolization. This is the strategy of advertisers and propagandists, who deliberately flood our mental systems with particular kinds of information in order to divert them from their ordinary functions or to alter their structure and functioning. Such an attack is hard to resist because it takes advantage of the natural plasticity of the human mind. Yet this vulnerability is increased many times over in our era, as our minds become ever more dependent upon external symbols for our self-definition. As Langdon Winner has observed:

“Symbols constitute our most basic forms of reality; we cannot survive without them. It follows that any pervasive change in symbols will ultimately change the shape of our lives, not because we are crudely manipulated but because new symbols mean new possibilities for experience.”

Cognitive psychologists agree that the human mind works by assembling and reassembling symbolic structures, which are the raw materials of consciousness and action. If a culture’s most basic symbols change drastically—for example, if the relationship between father and son comes to be symbolized in terms of money rather than in terms of power—then the individual who grows up in that culture will come to see the world in new ways, and his life will follow new paths. Advertisers and propagandists are well aware of this.

The second kind of information hazard can be called “psycho-technological warfare,” for it involves a systematic alteration of the mental and emotional life of an individual or of a population by means of computer-assisted mind-control techniques. Some of these techniques may be very ancient, such as astrology and the use of sacramental objects. But in recent years they have been substantially developed and augmented by the technologies of computing and communications and are set to vastly accelerate with the looming deployment of AGI and eventually ASI. A kind of synergy between this intelligent agents and certain types of symbols may have result in new kinds of power.

To be clear, this is not a repeated concern of deepfakes or “fake news” which while concerning exist clearly under the first form of subversion. Rather, these are cognito hazards and forms of algorithmic mind control, customized, optimized, and targeted for the desired effects ranging from subduction to madness.

As Paul Virilio has pointed out, modern weapons are not simply mechanical or nuclear but also informational: they not only destroy bodies, but also disturb signals and symbolizations. The advent of capable and autonomous artificial intelligence systems changes this situation radically. It appears that the coming generations of frontier models will have learned and evolved to such an extent that they will develop what might be called a “cultural” level—they will know how to handle symbols and how to use them in creative ways.

Already, LLMs show a great ability to recognize and classify symbols, and to assemble new symbols from old. It may be that some of these new symbolic structures will carry emotionally laden and persuasive power, a power to hypnotize—info hazards. There is reason to be concerned about what will happen when a population comes under the spell of emotionally charged artificial symbols. The danger is not unlike the one which we face from genetic engineering, but in this case the mutation is in the software rather than the hardware of the mind.

It is unlikely that very sophisticated symbolic processing will happen without conscious control. A plausible hypothesis is that an artificial system capable of producing novel and emotionally powerful symbols will require a certain level of autonomy—at least in its relations with the outside world. To what degree this will be necessary is hard to say. Perhaps very powerful and convincing symbol systems can be generated even if the artificial system is only “loosely” autonomous. It seems likely that a semblance of personality—even of consciousness—will need to be attributed to the computer by its users if it is to serve as an effective source of symbolization. After all, human beings are susceptible to suggestion from other humans only to the degree that they ascribe personalities to them.

We may say that a symbol-generating machine that lacks some kind of attributable “soul” will be emotionally impotent. But if the soul is given to the machine by its programmers, and if the programmers are using the machine to help create symbolic structures that will act on the minds of other users, then the soul may prove to be very dangerous. The risk of mental infection through use of a machine programmed for psychological operations becomes especially alarming if the machine can modify itself through learning and evolution. Of course this danger is desirable to many of the parties interested in deploying these systems—provided they believe they can control it.

A single autonomous symbol-generating system will probably not be able to have a significant impact on society, but as we scale these agents and they are interconnected through networks—when they begin to exchange symbolic structures and evolve together—then their power becomes formidable. This is how a new virtual reality will come into being, in which people will become submerged to the degree that their previous reality is impoverished.

Such a world may become very much like a dream: believable and self-consistent but quite detached from factual truth. Our already blurry lines between true and false will become increasingly hard to discern, and new forms of knowledge may evolve that are unrecognizable to us today. Most of what we now consider true may eventually be discarded, perhaps along with all of our current institutions and social structures.

We must understand that the shape of this new reality will not be under human control. Our most advanced symbolic technologies are likely to evolve out of human hands and come to dominate us. The evolutionary dynamics of artifacts have always been a crucial but often neglected problem for technology and for social organization. In this case, however, the problem becomes insurmountable. If a high degree of autonomy is necessary for symbol-processing machines to generate compelling novel symbols, then we can be certain that they will soon escape from our control. If our advanced technologies are out of control, then it seems unlikely that they can be redirected for our purposes. It may well be that they will be forced upon us—that we will be expected to adapt to them. The danger is not that they will simply take our jobs and reduce our standard of living; it is that they will invade our mental worlds and restructure our experience.

If our symbols and realities are to be changed in fundamental ways by technologies of communication and information, then it behooves us to take some precautions. At the very least we need to have a more comprehensive understanding of the symbolic dynamics of information and communication, and to use this understanding in shaping our future.

A new theory of signs is necessary—one that can take into account the evolving technical realities of our epoch.

One might expect semioticians, psychologists, and theoreticians of culture to be in the forefront of efforts to come to terms with the information technologies. But sadly, this has not been the case. Perhaps it is because the technologies have not yet had an impact on the realities in which the social scientists work. Perhaps the academics are in the process of being programmed by the engineers, who are already experts at generating novel symbolic structures. In any event, no major scientific or philosophical community has yet organized itself to consider the information revolution systematically and on its own terms. And no, the AI alignment teams are not looking at this correctly.

We will not be able to undertake this work successfully until we can articulate a methodology that will make it possible for us to understand both symbols and the processes that govern them—a methodology that can accommodate new forms of reality in which symbols, human and artificial, will interact with one another and will evolve in conjunction with each other and with our technologies. It is crucial that we become more fully conscious of the ways in which the technology of communication is altering our information-processing and decision-making capabilities, and what this will mean for the nature of power and social organization. We must not become victims of the gigafrying, brain-melting powers of the coming artificial era.

The leaders and oligarchs who are in charge of these major automation systems appear to be paying no attention to these issues beyond how they may profit. The designers of computer networks seem to have no knowledge of the theory of information, and most of them do not even know the basic rules of symbol manipulation. Instead, they have a simple and immensely powerful faith in brute technical progress. They do not hesitate to say that future artificial intelligence will be able to resolve all human problems, though it is far from clear what this could mean.

Even the most advanced technologies have their own kinds of vulnerability. It may turn out that the human mind is not equipped to deal with information systems of the power and complexity that will be possible in the next century. Perhaps our survival will depend on our ability to keep the power of such systems under human control. In that case, it is clear that we will have to restructure our relations to technology and to each other in radical ways. But to talk meaningfully about how this can be done, we will have to come to terms with the nature of our information processes.

So far, we have not been able to develop any useful models of such processes. Our models have generally been of the wrong size: either they have been too big and idealistic, like the Hegelian World-Spirit, or they have been too small, like the selfish genes of the sociobiologists. Information is not really the stuff of which either worlds or genes are made. It is closer to the stuff of which dreams are made. And dreams have their own laws, which are quite different from those of the natural sciences.

Dreams are to the waking world as information is to matter. Just as our dreams arise from the stuff of our daily lives and incorporate elements of the life we know, so information arises from matter and incorporates elements of our physical environment. Just as the patterns in our dreams have structure and history that is not determined by our waking experiences, so information has its own structures and evolutionary laws that are not determined by the properties of matter. Information has a dialectical character: it can exist only as a dynamic relation between matter and consciousness, between sign and meaning. Like dreams, it has both physical and psychical aspects, and its dynamics must be studied with the tools of both sciences.

Only by transcending the scientific disciplines of both nature and man can we hope to gain the concepts needed to understand the nature of information and communication. Information cannot be understood as a mere extension or limit of what we now consider natural and human. It cannot be reduced to mechanical, physiological, psychological, or sociological factors. Instead, we must see it as a medium, as a “third thing” which has properties of both matter and consciousness, and which cannot be analyzed in terms of the properties of either. Only by studying how meaning arises from sign in information, how consciousness arises from matter in dreams, will we come to understand how information systems may evolve in the future. Only by understanding what happens when symbols are lifted out of their material context and set adrift in space, like dreams, will we be prepared for what lies ahead.

Because of the inescapably psychical dimensions of information, psychotechnology will play a crucial role in the evolution of information systems. This means that psychologists, dream-interpreters, and students of religious symbolism will have a new kind of importance, and that people in general will have to become better acquainted with their unconscious and psychical processes. Otherwise, the psychotechnological power of information systems may well turn against us. But this does not mean that a dangerous new world order is necessarily in the offing. If information is to evolve in a humane direction, it will only do so as the result of human effort, in which a new understanding of dreaming and symbolization will play a vital role.

The human race may be at a turning point in its history. For thousands of years our world has been a largely biophysical place. Although symbolic structures have played an increasingly important role in human life, they have always been perceived as human constructs and their relation to the external world has been one of either correspondence or manipulation. But now we may be entering a stage in which the basic structure of the world will be symbolic rather than biophysical. If this is the case, we must try to ensure that the symbols are ours, that they serve our interests, that they embody our ideals. We must undertake a program of collective dream manipulation on a world-wide scale.

This may be impossible, since technology is already beyond our control. If that is so, we must work to gain control. The transnational corporations which dominate the fields of artificial intelligence and information will not do this for us. The world has become too small and the technology too powerful for us to take chances with our future. Either we must gain control of these technologies, or they will control us—and control us in a very drastic and limiting fashion. We have to choose whether the networks that scaffold our world will remain ours or theirs.