Technology and its unanticipated consequences

1. Introduction

When the successful cloning of a lamb called Dolly was announced in February of this year by Scottish researchers, it set off a spate of anxious questions. Many of them concerned the ethics of cloning, but another set asked about the unanticipated consequences. If we go down the cloning road, where will it lead? The answer is that we don't know. All of our technological roads twist and turn, and we can never see around the bend or through the fog.

The purpose of this paper is to investigate the ubiquitous phenomenon of unanticipated consequences. We begin with a look at some definitions which shed light on the matter, and then consider the nature of change. This leads to a broadening of the definition of the word 'technology', and a look at what was one of our earliest examples of unanticipated consequences. We then address the crucial question of why we have such consequences. Some additional examples follow, and we then look at what society does in the face of unanticipated consequences. The paper concludes with a discussion of some of the ethical implications of acting when we know that there can be unanticipated consequences to our actions.

2. Some Definitions

It is important here to distinguish between unanticipated and undesired consequences. The former are consequences which are not foreseen and dealt with in advance of their appearance. Undesired consequences are those which we are harmful, but which we are willing to accept, or accept the risk of occurring. Consequences may be:

Anticipated

• intended and desired
• not desired but common or probable
• not desired and improbable

Unanticipated

• desirable
• undesirable

As an example, consider the development of a nuclear power plant at an ocean site. The anticipated and intended goal or consequence is the production of electric power. The undesired but common and expected consequence is the heating of the ocean water near the plant. An undesired and improbable consequence would be a major explosion, and we would associate the term 'risk' with this outcome, but not with the heating of the water.

An unanticipated and desirable consequence might be the discovery of new operating procedures which would make nuclear power safer. An unforeseen and undesirable consequence might be the evolution of a new species of predator fish, in the warmed ocean water, which destroy existing desired species.

This paper concentrates on unanticipated consequences of our technologies. Anticipated negative consequences have been dealt with extensively in the literature on risk. See, for example, Margolis (1996), and Bernstein (1996). The latter emphasizes the role of mathematics in risk assessment.

Two brief points should be made before we proceed. The first is that change is always with us. Even without the intervention of human beings, nature changes constantly. Continents move, weather changes, species evolve, new worlds are born and old ones die. The second point is that all change seems to involve unanticipated consequences. Hence, the unanticipated is a part of life. There is no absolute security. Unanticipated consequences can be mitigated, largely through the gaining of additional information or knowledge, but not eliminated. That's the nature of our life, natural and human.

3. A Broader Definition of Technology

Although we focus here on the term 'technology' as it is usually taken, it is worth pointing out that human beings do much that has unanticipated consequences, in all areas of life, certainly including, for example: medicine, business, law, politics, religion, education, and many more. Because of the parallels among these fields it is useful to think of a broader definition of technology, such as "...that which can be done, excluding only those capabilities that occur naturally in living systems." (Benziger) This matter is dealt with in some detail by in Technopoly: The Surrender of Culture to Technology (Postman).

Seen in the light of this broader definition, writing is one of our first technologies. Postman recalls the story of Thamus and the god Theuth, from Plato's Phaedrus, as an example of unanticipated consequences. Theuth had invented many things, including: number, calculation, geometry, astronomy and writing. Theuth claimed that writing would improve both the memory and the wisdom of humans. Thamus thought otherwise.

Theuth, my paragon of inventors, the discoverer of an art is not the best judge of the good or harm which will accrue to those who practice it. So it is in this: you, who are the father of writing, have out of fondness for your off-spring attributed to it quite the opposite of its real function. Those who acquire it will cease to exercise their memory and become forgetful; they will rely on writing to bring things to their remembrance by external signs instead of by their internal resources. What you have discovered is a receipt for recollection, not for memory. And as for wisdom, your pupils will have the reputation for it without the reality: they will receive a quantity of information without proper instruction, and in consequence be thought very knowledgeable when they are for the most part ignorant. And because they are filled with the conceit of wisdom instead of real wisdom they will be a burden to society.

It is true in all of our technologies, the discoverer of an art, or the designer of a new system is not usually the best judge of the good or harm which will accrue to those who practice it. And yet, paradoxically, it is often the designer to whom we must go to ask for the likely outcomes of her work. This is a problem with which society must wrestle, and we shall discuss later how it does so.

4. Why Do We Have Unintended Consequences?

Dietrich Dorner has recently analyzed systems in a way that can help us see why they can be so difficult to understand, and hence why consequences are unanticipated. Dorner has identified four features of systems which make a full understanding of any real system impossible. These are:

• complexity
• dynamics
• intransparence
• ignorance and mistaken hypotheses

Complexity reflects the many different components which a real system has and the interconnections or interrelations among these components. Our system models necessarily neglect many of these components or features, and even more so their interrelations, but there is always a danger in doing so, because it is from such interrelations that the unanticipated may arise. Our economic system is an example of a highly complex system. Not only are there many players, but the players are also interrelated in many ways which are difficult to identify and define. If Player A sets this price, how will Player B respond, and what will Player C think and do when she observes the actions of A and B?

Many devices and systems exhibit dynamics, that is, the property of changing their state spontaneously, independent of control by a central agent in charge of the system. One of the most fascinating examples of our time is the Internet, an extraordinarily dynamic system, with no one in charge. There is no way to model the Internet system in a way which will predict its future and the future of the people and things which will be impacted by the Internet. Many of our complex technological systems have this property. Examples might include: a new freeway system, nuclear power, high definition television, genetic engineering. For example, a freeway system is dynamic because a large number of players initiate actions beyond any central control. Driver A slows down to observe an accident, Driver B responds in an unpredictable way, depending on his skills, state of mind, sobriety perhaps, and other factors. The system, though structured to some degree, is in many ways on its own.

Intransparence means that some of the elements of a system cannot be seen, but can nevertheless affect the operation of the system. More complex systems can have many contributors to intransparence. In the Internet, for example, the list would include almost all of the users at a particular time, equipment failures at user sites, local phenomena, such as weather, which affect use of the Internet at other locations. We need to understand that what you can't see might hurt you.

Finally, ignorance and mistaken hypotheses are always a possibility. Perhaps our model is simply wrong, faulty, misleading. This last problem is particularly interesting and important, because it is the one we can do something about. We can take steps to reduce our ignorance, to increase our understanding, as we shall discuss in Section 6. And in Section 7 we argue that we are obliged to do so.

Lets look next at some other perspectives on this problem. Peter Bernstein has addressed the matter from the viewpoint of probabilities and economics. He points out that economists have sometimes believed that deterministic forces drive our societies and their enterprises. More contemporary economists have seen less order. Bernstein puts it this way.

The optimism of the Victorians was snuffed out by the senseless destruction of human life on the battlefields (of the First World War), the uneasy peace that followed, and goblins let loose by the Russian Revolution. Never again would people accept Robert Browning's assurance that "God's in his heaven:/All's right with the world." Never again would economists insist that fluctuations in the economy were a theoretical impossibility. Never again would science appear so unreservedly benign, nor would religion and family institutions be so unthinkingly accepted in the western world. ...

Up to this point, the classical economists had defined economics as a riskless system that always produced optimal results....

Such convictions died hard, even in the face of the economic problems that emerged in the wake of the war. But a few voices were raised proclaiming that the world was no longer what once it had seemed. Writing in 1921, the University of Chicago economist Frank Knight uttered strange words for a man in his profession: 'There is much question as to how far the world is intelligible at all...It is only in the very special and crucial cases that anything like a mathematical study can be made.'

Edward Tenner takes still another perspective on the phenomenon of unanticipated and unintended consequences. He sees in some of our technologies a "revenge effect" in which our perverse technologies turn against us with consequences which exceed the good which had been planned.

Security is another window on revenge effects. Power door locks, now standard on most cars, increase the sense of safety. But they have helped triple or quadruple the number of drivers locked out over the last two decades - costing $400 million a year and exposing drivers to the very criminals the locks were supposed to defeat.

We shall return to this issue of perversity in Section 6 when we see how society attempts to deal with unintended consequences.

For Dorner on engineering, for Bernstein on economics, for Tenner's perverse technologies the message is the same. The world is not knowable and predictable. It's complexities are too great, its uncertainties beyond our understanding. Some unanticipated consequences are a necessary feature of all of our enterprises. But this does not mean that we should give up the effort to reduce uncertainty. We shall return to this in Section 7 on ethical implications.

In the next section we turn to some examples of such consequences. Then in Section 6 we consider how society responds to the problem of unanticipated consequences.

5. Some Examples

In this section we consider some anecdotes, some cases, of consequences which were not anticipated. We follow a historical path in this effort. We have already reached back to a time before the dawn of human history for a story of the invention of writing. Now we jump forward to the last two hundred years, touching on some of the effects of the Industrial Revolution, and moving on to questions which are being asked today about newly proposed technologies.

James Beniger's The Control Revolution: Technological and Economic Origins of the Information Society, traces in some detail the evolution of technological development over the past two centuries, particularly in the United States. While Beniger stresses the role and need for control in technology, he does not pay a great deal of explicit attention to consequences. But, the implicit implications of the changes in speed brought on by the Industrial Revolution are clear.