Superintelligence: Paths, Dangers, Strategies (50 page)

52
. Rubin and Watson (2011).

53
. Elyasaf et al. (2011).

54
. KGS (2012).

55
. Newell et al. (1958, 320).

56
. Attributed in Vardi (2012).

57
. In 1976, I. J. Good wrote: “A computer program of Grandmaster strength would bring us within an ace of [machine ultra-intelligence]” (Good 1976). In 1979, Douglas Hofstadter opined in his Pulitzer-winning
Gödel, Escher, Bach
: “Question: Will there be chess programs that can beat anyone? Speculation: No. There may be programs that can beat anyone at chess, but they will not be exclusively chess programs. They will be programs of general intelligence, and they will be just as temperamental as people. ‘Do you want to play chess?’ ‘No, I’m bored with chess. Let’s talk about poetry’” (Hofstadter [1979] 1999, 678).

58
. The algorithm is minimax search with alpha-beta pruning, used with a chess-specific heuristic evaluation function of board states. Combined with a good library of openings and endgames, and various other tricks, this can make for a capable chess engine.

59
. Though especially with recent progress in learning the evaluation heuristic from simulated games, many of the underlying algorithms would probably also work well for many other games.

60
.
Nilsson (2009, 318). Knuth was certainly overstating his point. There are many “thinking tasks” that AI has not succeeded in doing—inventing a new subfield of pure mathematics, doing any kind of philosophy, writing a great detective novel, engineering a coup d’état, or designing a major new consumer product.

61
. Shapiro (1992).

62
. One might speculate that one reason it has been difficult to match human abilities in perception, motor control, common sense, and language understanding is that our brains have dedicated wetware for these functions—neural structures that have been optimized over evolutionary timescales. By contrast, logical thinking and skills like chess playing are not natural to us; so perhaps we are forced to rely on a limited pool of general-purpose cognitive resources to perform these tasks. Maybe what our brains do when we engage in explicit logical reasoning or calculation is in some ways analogous to running a “virtual machine,” a slow and cumbersome mental simulation of a general-purpose computer. One might then say (somewhat fancifully) that a classical AI program is not so much emulating human thinking as the other way around: a human who is thinking logically is emulating an AI program.

63
. This example is controversial: a minority view, represented by approximately 20% of adults in the USA and similar numbers in many other developed nations, holds that the Sun revolves around the Earth (Crabtree 1999; Dean 2005).

64
. World Robotics (2011).

65
. Estimated from data in Guizzo (2010).

66
. Holley (2009).

67
. Hybrid rule-based statistical approaches are also used, but they are currently a small part of the picture.

68
. Cross and Walker (1994); Hedberg (2002).

69
. Based on the statistics from TABB Group, a New York- and London-based capital markets research firm (personal communication).

70
. CFTC and SEC (2010). For a different perspective on the events of 6 May 2010, see CME Group (2010).

71
. Nothing in the text should be construed as an argument against algorithmic high-frequency trading, which might normally perform a beneficial function by increasing liquidity and market efficiency.

72
. A smaller market scare occurred on August, 1, 2012, in part because the “circuit breaker” was not also programmed to halt trading if there were extreme changes in the
number
of shares being traded (Popper 2012). This again foreshadows another later theme: the difficulty of anticipating all specific ways in which some particular plausible-seeming rule might go wrong.

73
. Nilsson (2009, 319).

74
. Minsky (2006); McCarthy (2007); Beal and Winston (2009).

75
. Peter Norvig, personal communication. Machine-learning classes are also very popular, reflecting a somewhat orthogonal hype-wave of “big data” (inspired by e.g. Google and the Netflix Prize).

76
. Armstrong and Sotala (2012).

77
. Müller and Bostrom (forthcoming).

78
. See Baum et al. (2011), another survey cited therein, and Sandberg and Bostrom (2011).

79
. Nilsson (2009).

80
. This is again conditional on no civilization-disrupting catastrophe occurring. The definition of HLMI used by Nilsson is “AI able to perform around 80% of jobs as well or better than humans perform” (Kruel 2012).

81
. The table shows the results of four different polls as well as the combined results. The first two were polls taken at academic conferences:
PT-AI
, participants of the conference
Philosophy and Theory of AI
in Thessaloniki 2011 (respondents were asked in November 2012), with a response rate of 43 out of 88; and
AGI
, participants of the conferences
Artificial General Intelligence
and
Impacts and Risks of Artificial General Intelligence
, both in Oxford, December 2012 (response rate: 72/111). The
EETN
poll sampled the members of the Greek Association for Artificial Intelligence, a professional organization of published researchers in the field, in April 2013 (response
rate: 26/250). The
TOP100
poll elicited the opinions among the 100 top authors in artificial intelligence as measured by a citation index, in May 2013 (response rate: 29/100).

82
. Interviews with some 28 (at the time of writing) AI practitioners and related experts have been posted by Kruel (2011).

83
. The diagram shows renormalized median estimates. Means are significantly different. For example, the mean estimates for the “Extremely bad” outcome were 7.6% (for
TOP100
) and 17.2% (for the combined pool of expert assessors).

84
. There is a substantial literature documenting the unreliability of expert forecasts in many domains, and there is every reason to think that many of the findings in this body of research apply to the field of artificial intelligence too. In particular, forecasters tend to be overconfident in their predictions, believing themselves to be more accurate than they really are, and therefore assigning too little probability to the possibility that their most-favored hypothesis is wrong (Tetlock 2005). (Various other biases have also been documented; see, e.g., Gilovich et al. [2002].) However, uncertainty is an inescapable fact of the human condition, and many of our actions unavoidably rely on expectations about which long-term consequences are more or less plausible: in other words, on probabilistic predictions. Refusing to offer explicit probabilistic predictions would not make the epistemic problem go away; it would just hide it from view (Bostrom 2007). Instead, we should respond to evidence of overconfidence by broadening our confidence intervals (or “credible intervals”)—i.e. by smearing out our credence functions—and in general we must struggle as best we can with our biases, by considering different perspectives and aiming for intellectual honesty. In the longer run, we can also work to develop techniques, training methods, and institutions that can help us achieve better calibration. See also Armstrong and Sotala (2012).

1
. This resembles the definition in Bostrom (2003c) and Bostrom (2006a). It can also be compared with Shane Legg’s definition (“Intelligence measures an agent’s ability to achieve goals in a wide range of environments”) and its formalizations (Legg 2008). It is also very similar to Good’s definition of ultraintelligence in
Chapter 1
(“a machine that can far surpass all the intellectual activities of any man however clever”).

2
. For the same reason, we make no assumption regarding whether a superintelligent machine could have “true intentionality” (
pace
Searle, it could; but this seems irrelevant to the concerns of this book). And we take no position in the internalism/externalism debate about mental content that has been raging in the philosophical literature, or on the related issue of the extended mind thesis (Clark and Chalmers 1998).

3
. Turing (1950, 456).

4
. Turing (1950, 456).

5
. Chalmers (2010); Moravec (1976, 1988, 1998, 1999).

6
. See Moravec (1976). A similar argument is advanced by David Chalmers (2010).

7
. See also Shulman and Bostrom (2012), where these matters are elaborated in more detail.

8
. Legg (2008) offers this reason in support of the claim that humans will be able to recapitulate the progress of evolution over much shorter timescales and with reduced computational resources (while noting that evolution’s unadjusted computational resources are far out of reach). Baum (2004) argues that some developments relevant to AI occurred earlier, with the organization of the genome itself embodying a valuable representation for evolutionary algorithms.

9
. Whitman et al. (1998); Sabrosky (1952).

10
. Schultz (2000).

11
. Menzel and Giurfa (2001, 62); Truman et al. (1993).

12
. Sandberg and Bostrom (2008).

13
. See Legg (2008) for further discussion of this point and of the promise of functions or environments that determine fitness based on a smooth landscape of pure intelligence tests.

14
. See Bostrom and Sandberg (2009b) for a taxonomy and more detailed discussion of ways in which engineers may outperform historical evolutionary selection.

15
.
The analysis has addressed the nervous systems of living creatures, without reference to the cost of simulating bodies or the surrounding virtual environment as part of a fitness function. It is plausible that an adequate fitness function could test the competence of a particular organism in far fewer operations than it would take to simulate all the neuronal computation of that organism’s brain throughout its natural lifespan. AI programs today often develop and operate in very abstract environments (theorem provers in symbolic math worlds, agents in simple game tournament worlds, etc.).

A skeptic might insist that an abstract environment would be inadequate for the evolution of general intelligence, believing instead that the virtual environment would need to closely resemble the actual biological environment in which our ancestors evolved. Creating a physically realistic virtual world would require a far greater investment of computational resources than the simulation of a simple toy world or abstract problem domain (whereas evolution had access to a physically realistic real world “for free”). In the limiting case, if complete micro-physical accuracy were insisted upon, the computational requirements would balloon to ridiculous proportions. However, such extreme pessimism is almost certainly unwarranted; it seems unlikely that the best environment for evolving intelligence is one that mimics nature as closely as possible. It is, on the contrary, plausible that it would be more efficient to use an artificial selection environment, one quite unlike that of our ancestors, an environment specifically designed to promote adaptations that increase the type of intelligence we are seeking to evolve (abstract reasoning and general problem-solving skills, for instance, as opposed to maximally fast instinctual reactions or a highly optimized visual system).

16
. Wikipedia (2012b).

17
. For a general treatment of observation selection theory, see Bostrom (2002a). For the specific application to the current issue, see Shulman and Bostrom (2012). For a short popular introduction, see Bostrom (2008b).

18
. Sutton and Barto (1998, 21f); Schultz et al. (1997).

19
. This term was introduced by Eliezer Yudkowsky; see, e.g., Yudkowsky (2007).

20
. This is the scenario described by Good (1965) and Yudkowsky (2007). However, one could also consider an alternative in which the iterative sequence has some steps that do not involve intelligence enhancement but instead design simplification. That is, at some stages, the seed AI might rewrite itself so as make subsequent improvements easier to find.

21
. Helmstaedter et al. (2011).

22
. Andres et al. (2012).

23
. Adequate for enabling instrumentally useful forms of cognitive functioning and communication, that is; but still radically impoverished relative to the interface provided by the muscles and sensory organs of a normal human body.