The Glass Cage: Automation and Us (28 page)

The introduction of a new class of armaments always alters the nature of warfare, and weapons that can be launched or detonated from afar—catapults, mines, mortars, missiles—tend to have the greatest effects, both intended and unintended. The consequences of autonomous killing machines would likely go beyond anything that’s come before. The first shot freely taken by a robot will be a shot heard round the world. It will change war, and maybe society, forever.

T
HE SOCIAL
and ethical challenges posed by killer robots and self-driving cars point to something important and unsettling about where automation is headed. The substitution myth has traditionally been defined as the erroneous assumption that a job can be divided into separate tasks and those tasks can be automated piecemeal without changing the nature of the job as a whole. That definition may need to be broadened. As the scope of automation expands, we’re learning that it’s also a mistake to assume that society can be divided into discrete spheres of activity—occupations or pastimes, say, or domains of governmental purview—and those spheres can be automated individually without changing the nature of society as a whole. Everything is connected—change the weapon, and you change the war—and the connections tighten when they’re made explicit in computer networks. At some point, automation reaches a critical mass. It begins to shape society’s norms, assumptions, and ethics. People see themselves and their relations to others in a different light, and they adjust their sense of personal agency and responsibility to account for technology’s expanding role. They behave differently too. They
expect
the aid of computers, and on those rare instances when it’s not forthcoming, they feel bewildered. Software takes on what the MIT computer scientist Joseph Weizenbaum termed a “compelling urgency.” It becomes “the very stuff out of which man builds his world.”
¹⁰

In the 1990s, just as the dot-com bubble was beginning to inflate, there was much excited talk about “ubiquitous computing.” Soon, pundits assured us, microchips would be everywhere—embedded in factory machinery and warehouse shelving, affixed to the walls of offices and shops and homes, buried in the ground and floating in the air, installed in consumer goods and woven into clothing, even swimming around in our bodies. Equipped with sensors and transceivers, the tiny computers would measure every variable imaginable, from metal fatigue to soil temperature to blood sugar, and they’d send their readings, via the internet, to data-processing centers, where bigger computers would crunch the numbers and output instructions for keeping everything in spec and in sync. Computing would be pervasive; automation, ambient. We’d live in a geek’s paradise, the world a programmable machine.

One of the main sources of the hype was Xerox PARC, the fabled Silicon Valley research lab where Steve Jobs found the inspiration for the Macintosh. PARC’s engineers and information scientists published a series of papers portraying a future in which computers would be so deeply woven into “the fabric of everyday life” that they’d be “indistinguishable from it.”
¹¹
We would no longer even notice all the computations going on around us. We’d be so saturated with data, so catered to by software, that, instead of experiencing the anxiety of information overload, we’d feel “encalmed.”
¹²
It sounded idyllic. But the PARC researchers weren’t Pollyannas. They also expressed misgivings about the world they foresaw. They worried that a ubiquitous computing system would be an ideal place for Big Brother to hide. “If the computational system is invisible as well as extensive,” the lab’s chief technologist, Mark Weiser, wrote in a 1999 article in
IBM Systems Journal
, “it becomes hard to know what is controlling what, what is connected to what, where information is flowing, [and] how it is being used.”
¹³
We’d have to place a whole lot of trust in the people and companies running the system.

The excitement about ubiquitous computing proved premature, as did the anxiety. The technology of the 1990s was not up to making the world machine-readable, and after the dot-com crash, investors were in no mood to bankroll the installation of expensive microchips and sensors everywhere. But much has changed in the succeeding fifteen years. The economic equations are different now. The price of computing gear has fallen sharply, as has the cost of high-speed data transmission. Companies like Amazon, Google, and Microsoft have turned data processing into a utility. They’ve built a cloud-computing grid that allows vast amounts of information to be collected and processed at efficient centralized plants and then fed into applications running on smartphones and tablets or into the control circuits of machines.
¹⁴
Manufacturers are spending billions of dollars to outfit factories with network-connected sensors, and technology giants like GE, IBM, and Cisco, hoping to spearhead the creation of an “internet of things,” are rushing to develop standards for sharing the resulting data. Computers are pretty much omnipresent now, and even the faintest of the world’s twitches and tremblings are being recorded as streams of binary digits. We may not be encalmed, but we are data saturated. The PARC researchers are starting to look like prophets.

There’s a big difference between a set of tools and an infrastructure. The Industrial Revolution gained its full force only after its operational assumptions were built into expansive systems and networks. The construction of the railroads in the middle of the nineteenth century enlarged the markets companies could serve, providing the impetus for mechanized mass production and ever larger economies of scale. The creation of the electric grid a few decades later opened the way for factory assembly lines and, by making all sorts of electrical appliances feasible and affordable, spurred consumerism and pushed industrialization into the home. These new networks of transport and power, together with the telegraph, telephone, and broadcasting systems that arose alongside them, gave society a different character. They altered the way people thought about work, entertainment, travel, education, even the organization of communities and families. They transformed the pace and texture of life in ways that went well beyond what steam-powered factory machines had done.

Thomas Hughes, in reviewing the consequences of the arrival of the electric grid in his book
Networks of Power
, described how first the engineering culture, then the business culture, and finally the general culture shaped themselves to the new system. “Men and institutions developed characteristics that suited them to the characteristics of the technology,” he wrote. “And the systematic interaction of men, ideas, and institutions, both technical and nontechnical, led to the development of a supersystem—a sociotechnical one—with mass movement and direction.” It was at this point that technological momentum took hold, both for the power industry and for the modes of production and living it supported. “The universal system gathered a conservative momentum. Its growth generally was steady, and change became a diversification of function.”
¹⁵
Progress had found its groove.

We’ve reached a similar juncture in the history of automation. Society is adapting to the universal computing infrastructure—more quickly than it adapted to the electric grid—and a new status quo is taking shape. The assumptions underlying industrial operations and commercial relations have already changed. “Business processes that once took place among human beings are now being executed electronically,” explains W. Brian Arthur, an economist and technology theorist at the Santa Fe Institute. “They are taking place in an unseen domain that is strictly digital.”
¹⁶
As an example, he points to the process of moving a shipment of freight through Europe. A few years ago, this would have required a legion of clipboard-wielding agents. They’d log arrivals and departures, check manifests, perform inspections, sign and stamp authorizations, fill out and file paperwork, and send letters or make phone calls to a variety of other functionaries involved in coordinating or regulating international freight. Changing the shipment’s routing would have involved laborious communications among representatives of various concerned parties—shippers, receivers, carriers, government agencies—and more piles of paperwork. Now, pieces of cargo carry radio-frequency identification tags. When a shipment passes through a port or other way station, scanners read the tags and pass the information along to computers. The computers relay the information to other computers, which in concert perform the necessary checks, provide the required authorizations, revise schedules as needed, and make sure all parties have current data on the shipment’s status. If a new routing is required, it’s generated automatically and the tags and related data repositories are updated.

Such automated and far-flung exchanges of information have become routine throughout the economy. Commerce is increasingly managed through, as Arthur puts it, “a huge conversation conducted entirely among machines.”
¹⁷
To be in business is to have networked computers capable of taking part in that conversation. “You know you have built an excellent digital nervous system,” Bill Gates tells executives, “when information flows through your organization as quickly and naturally as thought in a human being.”
¹⁸
Any sizable company, if it wants to remain viable, has little choice but to automate and then automate some more. It has to redesign its work flows and its products to allow for ever greater computer monitoring and control, and it has to restrict the involvement of people in its supply and production processes. People, after all, can’t keep up with computer chatter; they just slow down the conversation.

The science-fiction writer Arthur C. Clarke once asked, “Can the synthesis of man and machine ever be stable, or will the purely organic component become such a hindrance that it has to be discarded?”
¹⁹
In the business world at least, no stability in the division of work between human and computer seems in the offing. The prevailing methods of computerized communication and coordination pretty much ensure that the role of people will go on shrinking. We’ve designed a system that discards us. If technological unemployment worsens in the years ahead, it will be more a result of our new, subterranean infrastructure of automation than of any particular installation of robots in factories or decision-support applications in offices. The robots and applications are the visible flora of automation’s deep, extensive, and implacably invasive root system.

That root system is also feeding automation’s spread into the broader culture. From the provision of government services to the tending of friendships and familial ties, society is reshaping itself to fit the contours of the new computing infrastructure. The infrastructure orchestrates the instantaneous data exchanges that make fleets of self-driving cars and armies of killer robots possible. It provides the raw material for the predictive algorithms that inform the decisions of individuals and groups. It underpins the automation of classrooms, libraries, hospitals, shops, churches, and homes—places traditionally associated with the human touch. It allows the NSA and other spy agencies, as well as crime syndicates and nosy corporations, to conduct surveillance and espionage on an unprecedented scale. It’s what has shunted so much of our public discourse and private conversation onto tiny screens. And it’s what gives our various computing devices the ability to guide us through the day, offering a steady stream of personalized alerts, instructions, and advice.

Once again, men and institutions are developing characteristics that suit them to the characteristics of the prevailing technology. Industrialization didn’t turn us into machines, and automation isn’t going to turn us into automatons. We’re not that simple. But automation’s spread is making our lives more programmatic. We have fewer opportunities to demonstrate our own resourcefulness and ingenuity, to display the self-reliance that was once considered the mainstay of character. Unless we start having second thoughts about where we’re heading, that trend will only accelerate.