Will machines outsmart man?Scientists believe the point of 'Singularity' – where artificial intelligence surpasses that of

humans – is closer than we thought

Wendy M Grossman - guardian.co.uk, Wednesday 5 November 2008

http://www.guardian.co.uk/technology/2008/nov/06/artificialintelligenceai-engineeringThey are looking for the hockey stick. Hockey sticks are the shape technology startups hope their sales graphs will assume: a modestly ascending blade, followed by a sudden turn to a near-vertical long handle. Those who assembled in San Jose in late October for the Singularity Summit are awaiting the point where machine intelligence surpasses that of humans and takes off near-vertically into recursive self-improvement.The key, said Ray Kurzweil, inventor of the first reading machine and author of 2005's The Singularity Is Near, is exponential growth in computational power - "the law of accelerating returns". In his favourite example, at the human genome project's initial speed, sequencing the genome should have taken thousands of years, not the 15 scheduled. Seven years in, the genome was 1% sequenced. Exponential acceleration had the project finished on schedule. By analogy, enough doublings in processing power will close today's vast gap between machine and human intelligence.This may be true. Or it may be an unfalsifiable matter of faith, which is why the singularity is sometimes satirically called "the Rapture for nerds". It makes assessing progress difficult. Justin Rattner, chief technology officer of Intel, addressed a key issue at the summit: can Moore's law, which has the number of transistors packed on to a chip doubling every 18 months, stay in line with Kurzweil's graphs? The end has been predicted many times but, said Rattner, although particular chip technologies have reached their limits, a new paradigm has always continued the pace."In some sense - silicon gate CMOS - Moore's law ended last year," Rattner said. "One of the founding laws of accelerating returns ended. But there are a lot of smart people at Intel and they were able to reinvent the CMOS transistor using new materials." Intel is now looking beyond 2020 at photonics and quantum effects such as spin. "The arc of Moore's law brings the singularity ever closer."

Judgment dayBelief in an approaching singularity is not solely American. Peter Cochrane, the former head of BT's research labs, says for machines to outsmart humans it "depends on almost one factor alone - the number of networked sensors. Intelligence is more to do with sensory ability than memory and computing power." The internet, he adds, overtook the capacity of a single human brain in 2006. "I reckon we're looking at the 2020 timeframe for a significant machine intelligence to emerge." And, he said: "By 2030 it really should be game over."Predictions like this flew at the summit. Imagine when a human-scale brain costs $1 - you could have a pocket full of them. The web will wake up, like Gaia. Nova Spivack, founder of EarthWeb and, more recently, Radar Networks (creator of Twine.com), quoted Freeman Dyson: "God is what mind becomes when it has passed beyond the scale of our comprehension."

1

Listening, you'd never guess that artificial intelligence has been about 20 years away for a long time now. John McCarthy, one of AI's fathers, thought when he convened the first conference on the subject in 1956, that they'd be able to wrap the whole thing up in six months. McCarthy calls the singularity, bluntly, "nonsense".Even so, there are many current technologies, such as speech recognition, machine translation, and IBM's human-beating chess grandmaster Deep Blue, that would have seemed like AI at the beginning. "It's incredible how intelligent a human being in front of a connected computer is," observed the CNBC reporter Bob Pisani, marvelling at how clever Google makes him sound to viewers phoning in. Such advances are reminders that there may be valuable discoveries that make attempts at even the wildest ideas worthwhile.Dharmendra Modha, head of the cognitive computing group at IBM's Almaden research lab, is leading a "quest" to "understand and build a brain as cheaply and quickly as possible". Last year, his group succeeded in simulating a rat-scale cortical model - 55m neurons, 442bn synapses - in 8TB memory of a 32,768-processor IBM Blue Gene supercomputer. The key, he says, is not the neurons but the synapses, the electrical-chemical-electrical connections between those neurons. Biological microcircuits are roughly essentially the same in all mammals. "An individual human being is stored in the strength of the synapses."

Smarter than smartModha doesn't suggest that the team has made a rat brain. "Philosophically," he writes on the subject, "any simulation is always an approximation (a kind of 'cartoon') based on certain assumptions. A biophysically realistic simulation is not the focus of our work." His team is using the simulation to try to understand the brain's high-level computational principles.But computational power is nothing without software. "Would the neural code that powers human reasoning run on a different substrate?" the sceptical science writer John Horgan asked Kurzweil, who replied: "The key to the singularity is amplifying intelligence. The prediction is that an entity that passes the Turing test and has emotional intelligence ... will convince us that it's conscious. But that's not a philosophical demonstration."For intelligence to be effective, it has to be able to change the physical world. The MIT physicist Neil Gershenfeld was therefore at the summit to talk about programmable matter. It's a neat trick: computer science talks in ones and zeros, but these are abstractions representing the flow or interruption of electric current, a physical phenomenon. Gershenfeld, noting that maintaining that abstraction requires increasing amounts of power and complex programmning, wants to turn this on its head. What if, he asked, you could buy computing cells by the pound, coat them on a surface, and run programs that assemble them like proteins to solve problems?Gershenfeld is always difficult for non-physicists to understand, and his video of cells sorting was no exception. Two things he said were clear. First: "We aim to create life." Second: "We have a 20-year road map to make the Star Trek replicator."Twenty years: 2028. Vernor Vinge began talking about the singularity in the early 80s (naming it after the gravitational phenomenon around a black hole), and has always put the date at 2030. Kurzweil likes 2045; Rattner, before 2050.

Turning back timeThese dates may be personally significant. Rattner is 59; Vinge is 64. Kurzweil is 60, takes 250 vitamins and other supplements a day, and believes some of them can turn back ageing. If curing

2

all human ills will be a piece of cake for a superhuman intelligence, then the singularity carries with it the promise of immortality - as long as you're still alive when it happens.It is in this connection between the singularity and immortality, along with the idea that sufficiently advanced technology can solve every problem from climate change to the exhaustion of oil reserves, that gives the summit the feel of a religious movement. Certainly, James Miller, assistant professor of economics at Smith College, sounded evangelical when he reviewed how best to prepare financially. He was optimistic, reviewing investment strategies and assuming retirement funds won't be needed.HowStuffWorks founder Marshall Brain, by contrast, explained why 50 million people will lose their jobs when they can be replaced by robots. "In the whole universe, there is one intelligent species," he said. "We're in the process of creating the second intelligent species."The anthropologist Jane Goodall may disagree. She sees a different kind of singularity - the growing ecological devastation of Africa - and worries about the disconnection between human minds and hearts. "If we're the most intellectual animal," she said, "why are we destroying our only home?"If Goodall's singularity comes first, the other one might never happen at all - one of those catastrophes that Vinge admits as the only thing he can imagine that could stop it.

Watch the Singularity Summit videos at:http://www.vimeo.com/siai/videos/sort:oldest

. . . and go to the Summit home page at:

The Singularity Institute is pleased to host the Singularity Summit 2009, a rare gathering of thinkers to explore the rising impact of science and technology on society. The summit has been organized to further the understanding of a controversial idea – the singularity scenario.Founded in 2006 by Tyler Emerson, Ray Kurzweil, and Peter Thiel, the inaugural summit was held at Stanford, the first academic symposium focused on singularity dialogue.

http://www.singularitysummit.com/

ARTIFICIAL INTELLIGENCE: THE SINGULARITYFor over a hundred thousand years, the evolved human brain has held a privileged place in the expanse of cognition. Within this century, science may move humanity beyond its boundary of intelligence. This possibility, the singularity, may be a critical event in history, and deserves thoughtful consideration

OVERVIEW: WHAT IS A SINGULARITY?

What is the Singularity?The Singularity is the technological creation of smarter-than-human intelligence. There are several technologies that are often mentioned as heading in this direction. The most commonly mentioned is probably Artificial Intelligence, but there are others: direct brain-computer interfaces, biological augmentation of the brain, genetic engineering, ultra-high-resolution scans of the brain followed by computer emulation. Some of these technologies seem

3

likely to arrive much earlier than the others, but there are nonetheless several independent technologies all heading in the direction of the Singularity – several different technologies which, if they reached a threshold level of sophistication, would enable the creation of smarter-than-human intelligence.

A future that contains smarter-than-human minds is genuinely different in a way that goes beyond the usual visions of a future filled with bigger and better gadgets. Vernor Vinge originally coined the term "Singularity" in observing that, just as our model of physics breaks down when it tries to model the singularity at the center of a black hole, our model of the world breaks down when it tries to model a future that contains entities smarter than human.

Human intelligence is the foundation of human technology; all technology is ultimately the product of intelligence. If technology can turn around and enhance intelligence, this closes the loop, creating a positive feedback effect. Smarter minds will be more effective at building still smarter minds. This loop appears most clearly in the example of an Artificial Intelligence improving its own source code, but it would also arise, albeit initially on a slower timescale, from humans with direct brain-computer interfaces creating the next generation of brain-computer interfaces, or biologically augmented humans working on an Artificial Intelligence project.

Some of the stronger Singularity technologies, such as Artificial Intelligence and brain-computer interfaces, offer the possibility of faster intelligence as well as smarter intelligence. Ultimately, speeding up intelligence is probably comparatively unimportant next to creating better intelligence; nonetheless the potential differences in speed are worth mentioning because they are so huge. Human neurons operate by sending electrochemical signals that propagate at a top speed of 150 meters per second along the fastest neurons. By comparison, the speed of light is 300,000,000 meters per second, two million times greater. Similarly, most human neurons can spike a maximum of 200 times per second; even this may overstate the information-processing capability of neurons, since most modern theories of neural information-processing call for information to be carried by the frequency of the spike train rather than individual signals. By comparison, speeds in modern computer chips are currently at around 2GHz – a ten millionfold difference – and still increasing exponentially. At the very least it should be physically possible to achieve a million-to-one speedup in thinking, at which rate a subjective year would pass in 31 physical seconds. At this rate the entire subjective timespan from Socrates in ancient Greece to modern-day humanity would pass in under twenty-two hours.

Humans also face an upper limit on the size of their brains. The current estimate is that the typical human brain contains something like a hundred billion neurons and a hundred trillion synapses. That's an enormous amount of sheer brute computational force by comparison with today's computers – although if we had to write programs that ran on 200Hz CPUs we'd also need massive parallelism to do anything in realtime. However, in the computing industry, benchmarks increase exponentially, typically with a doubling time of one to two years. The original Moore's Law says that the number of transistors in a given area of silicon doubles every eighteen months; today there is Moore's Law for chip speeds, Moore's Law for computer memory, Moore's Law for disk storage per dollar, Moore's Law for Internet connectivity, and a dozen other variants.

4

By contrast, the entire five-million-year evolution of modern humans from primates involved a threefold increase in brain capacity and a sixfold increase in prefrontal cortex. We currently cannot increase our brainpower beyond this; in fact, we gradually lose neurons as we age. (You may have heard that humans only use 10% of their brains. Unfortunately, this is a complete urban legend; not just unsupported, but flatly contradicted by neuroscience.) An Artificial Intelligence would be different. Some discussions of the Singularity suppose that the critical moment in history is not when human-equivalent AI first comes into existence but a few years later when the continued grinding of Moore's Law produces AI minds twice or four times as fast as human. This ignores the possibility that the first invention of Artificial Intelligence will be followed by the purchase, rental, or less formal absorption of a substantial proportion of all the computing power on the then-current Internet – perhaps hundreds or thousands of times as much computing power as went into the original Artificial Intelligence.

But the real heart of the Singularity is the idea of better intelligence or smarter minds. Humans are not just bigger chimps; we are better chimps. This is the hardest part of the Singularity to discuss – it's easy to look at a neuron and a transistor and say that one is slow and one is fast, but the mind is harder to understand. Sometimes discussion of the Singularity tends to focus on faster brains or bigger brains because brains are relatively easy to argue about compared to minds; easier to visualize and easier to describe. This doesn't mean the subject is impossible to discuss; section III of our "Levels of Organization in General Intelligence" does take a stab at discussing some specific design improvements on human intelligence, but that involves a specific theory of intelligence, which we don't have room to go into here.

However, that smarter minds are harder to discuss than faster brains or bigger brains does not show that smarter minds are harder to build – deeper to ponder, certainly, but not necessarily more intractable as a problem. It may even be that genuine increases in smartness could be achieved just by adding more computing power to the existing human brain – although this is not currently known. What is known is that going from primates to humans did not require exponential increases in brain size or thousandfold improvements in processing speeds. Relative to chimps, humans have threefold larger brains, sixfold larger prefrontal areas, and 98. 4% similar DNA; given that the human genome has 3 billion base pairs, this implies that at most twelve million bytes of extra "software" transforms chimps into humans. And there is no suggestion in our evolutionary history that evolution found it more and more difficult to construct smarter and smarter brains; if anything, hominid evolution has appeared to speed up over time, with shorter intervals between larger developments.

But leave aside for the moment the question of how to build smarter minds, and ask what "smarter-than-human" really means. And as the basic definition of the Singularity points out, this is exactly the point at which our ability to extrapolate breaks down. We don't know because we're not that smart. We're trying to guess what it is to be a better-than-human guesser. Could a gathering of apes have predicted the rise of human intelligence, or understood it if it were explained? For that matter, could the 15th century have predicted the 20th century, let alone the 21st? Nothing has changed in the human brain since the 15th century; if the people of the 15th century could not predict five centuries ahead across constant minds, what makes us think we can outguess genuinely smarter-than-human intelligence?

5

Because we have a past history of people making failed predictions one century ahead, we've learned, culturally, to distrust such predictions – we know that ordinary human progress, given a century in which to work, creates a gap which human predictions cannot cross. We haven't learned this lesson with respect to genuine improvements in intelligence because the last genuine improvement to intelligence was a hundred thousand years ago. But the rise of modern humanity created a gap enormously larger than the gap between the 15th and 20th century. That improvement in intelligence created the entire milieu of human progress, including all the progress between the 15th and 20th century. It is a gap so large that on the other side we find, not failed predictions, but no predictions at all.

Smarter-than-human intelligence, faster-than-human intelligence, and self-improving intelligence are all interrelated. If you're smarter that makes it easier to figure out how to build fast brains or improve your own mind. In turn, being able to reshape your own mind isn't just a way of starting up a slope of recursive self-improvement; having full access to your own source code is, in itself, a kind of smartness that humans don't have. Self-improvement is far harder than optimizing code; nonetheless, a mind with the ability to rewrite its own source code can potentially make itself faster as well. And faster brains also relate to smarter minds; speeding up a whole mind doesn't make it smarter, but adding more processing power to the cognitive processes underlying intelligence is a different matter.

But despite the interrelation, the key moment is the rise of smarter-than-human intelligence, rather than recursively self-improving or faster-than-human intelligence, because it's this that makes the future genuinely unlike the past. That doesn't take minds a million times faster than human, or improvement after improvement piled up along a steep curve of recursive self-enhancement. One mind significantly beyond the humanly possible level would represent a Singularity. That we are not likely to be dealing with "only one" improvement does not make the impact of one improvement any less.

Combine faster intelligence, smarter intelligence, and recursively self-improving intelligence, and the result is an event so huge that there are no metaphors left. There's nothing remaining to compare it to.

The Singularity is beyond huge, but it can begin with something small. If one smarter-than-human intelligence exists, that mind will find it easier to create still smarter minds. In this respect the dynamic of the Singularity resembles other cases where small causes can have large effects; toppling the first domino in a chain, starting an avalanche with a pebble, perturbing an upright object balanced on its tip. (Human technological civilization occupies a metastable state in which the Singularity is an attractor; once the system starts to flip over to the new state, the flip accelerates.) All it takes is one technology – Artificial Intelligence, brain-computer interfaces, or perhaps something unforeseen – that advances to the point of creating smarter-than-human minds. That one technological advance is the equivalent of the first self-replicating chemical that gave rise to life on Earth.

For more information, continue with "Why Work Toward the Singularity

6

http://www.singinst.org/overview/whatisthesingularity

WHY WORK TOWARD A SINGULARITY?

If you traveled backward in time to witness a critical moment in the invention of science, or the creation of writing, or the evolution of Homo sapiens, or the beginning of life on Earth, no human judgment could possibly encompass all the future consequences of that event – and yet there would be the feeling of being present at the dawn of something worthwhile. The most critical moments of history are not the closed stories, like the start and finish of wars, or the rise and fall of governments. The story of intelligent life on Earth is made up of beginnings.

Imagine traveling back in time to witness a critical moment in the dawn of human intelligence. Suppose that you find an alien bystander already on the scene, who asks: "Why are you so excited? What does it matter?" The question seems almost impossible to answer; it demands a thousand answers, or none. Someone who valued truth and knowledge might answer that this was a critical moment in the human quest to learn about the universe – in fact, the beginning of that quest. Someone who valued happiness might answer that the rise of human intelligence was a necessary precursor to vaccines, air conditioning, and the many other sources of happiness and solutions to unhappiness that have been produced by human intelligence over the ages. There are people who would answer that intelligence is meaningful in itself; that "It is better to be Socrates unsatisfied than a fool satisfied; better to be a man unsatisfied than a pig satisfied." A musician who chose that career believing that music is an end in itself might answer that the rise of human intelligence mattered because it was necessary to the birth of Bach; a mathematician could single out Euclid; a physicist might cite Newton or Einstein. Someone with an appreciation of humanity, beyond the individual humans, might answer that this was a critical moment in the relation of life to the universe – the beginning of humanity's growth, of our acquisition of strength and understanding, eventually spreading beyond Earth to the rest of the galaxy and the universe.

The beginnings of human intelligence, or the invention of writing, probably went unappreciated by the individuals who were present at the time. But such developments do not always take their creators unaware. Francis Bacon, one of the critical figures in the invention of the scientific method, made astounding claims about the power and universality of his new mode of reasoning and its ability to improve the human condition – claims which, from the perspective of a 21st-century human, turned out to be exactly right. Not all good deeds are unintentional. It does occasionally happen that humanity's victories are won not by accident but by people making the right choices for the right reasons.

Why is the Singularity worth doing? The Singularity Institute for Artificial Intelligence can't possibly speak for everyone who cares about the Singularity. We can't even presume to speak for the volunteers and donors of the Singularity Institute. But it seems like a good guess that many supporters of the Singularity have in common a sense of being present at a critical moment in history; of having the chance to win a victory for humanity by making the right choices for the right reasons. Like a spectator at the dawn of human intelligence, trying to answer directly why superintelligence matters chokes on a dozen different simultaneous replies; what matters is the entire future growing out of that beginning.

7

But it is still possible to be more specific about what kinds of problems we might expect to be solved. Some of the specific answers seem almost disrespectful to the potential bound up in superintelligence; human intelligence is more than an effective way for apes to obtain bananas. Nonetheless, modern-day agriculture is very effective at producing bananas, and if you had advanced nanotechnology at your disposal, energy and matter might be plentiful enough that you could produce a million tons of bananas on a whim. In a sense that's what nanotechnology is – good-old-fashioned material technology pushed to the limit. This only begs the question of "So what?", but the Singularity advances on this question as well; if people can become smarter, this moves humanity forward in ways that transcend the faster and easier production of more and more bananas. For one thing, we may become smart enough to answer the question "So what?"

In one sense, asking what specific problems will be solved is like asking Benjamin Franklin in the 1700s to predict electronic circuitry, computers, Artificial Intelligence, and the Singularity on the basis of his experimentation with electricity. Setting an upper bound on the impact of superintelligence is impossible; any given upper bound could turn out to have a simple workaround that we are too young as a civilization, or insufficiently intelligent as a species, to see in advance. We can try to describe lower bounds; if we can see how to solve a problem using more or faster technological intelligence of the kind humans use, then at least that problem is probably solvable for genuinely smarter-than-human intelligence. The problem may not be solved using the particular method we were thinking of, or the problem may be solved as a special case of a more general challenge; but we can still point to the problem and say: "This is part of what's at stake in the Singularity."

If humans ever discover a cure for cancer, that discovery will ultimately be traceable to the rise of human intelligence, so it is not absurd to ask whether a superintelligence could deliver a cancer cure in short order. If anything, creating superintelligence only for the sake of curing cancer would be swatting a fly with a sledgehammer. In that sense it is probably unreasonable to visualize a significantly smarter-than-human intelligence as wearing a white lab coat and working at an ordinary medical institute doing the same kind of research we do, only better, in order to solve cancer specifically as a problem. For example, cancer can be seen as a special case of the more general problem "The cells in the human body are not externally programmable." This general problem is very hard from our viewpoint – it requires full-scale nanotechnology to solve the general case – but if the general problem can be solved it simultaneously solves cancer, spinal paralysis, regeneration of damaged organs, obesity, many aspects of aging, and so on. Or perhaps the real problem is that the human body is made out of cells or that the human mind is implemented atop a specific chunk of vulnerable brain – although calling these problems raises philosophical issues not discussed here.

Singling out "cancer" as the problem is part of our culture's particular outlook and technological level. But if cancer or any generalization of "cancer" is solved soon after the rise of smarter-than-human intelligence, then it makes sense to regard the quest for the Singularity as a continuation by other means of the quest to cure cancer. The same could be said of ending world hunger, curing Alzheimer's disease, or placing on a voluntary basis many things which at least some people would regard as undesirable: illness, destructive aging, human stupidity, short lifespans. Maybe death itself will turn out to be curable, though that would depend on whether the laws of

8

physics permit true immortality. At the very least, the citizens of a post-Singularity civilization should have an enormously higher standard of living and enormously longer lifespans than we see today.

What kind of problems can we reasonably expect to be solved as a side effect of the rise of superintelligence; how long will it take to solve the problems after the Singularity; and how much will it cost the beneficiaries? A conservative version of the Singularity would start with the rise of smarter-than-human intelligence in the form of enhanced humans with minds or brains that have been enhanced by purely biological means. This scenario is more "conservative" than a Singularity which takes place as a result of brain-computer interfaces or Artificial Intelligence, because all thinking is still taking place on neurons with a characteristic limiting speed of 200 operations per second; progress would still take place at a humanly comprehensible speed. In this case, the first benefits of the Singularity probably would resemble the benefits of ordinary human technological thinking, only more so. Any given scientific problem could benefit from having a few Einsteins or Edisons dumped into it, but it would still require time for research, manufacturing, commercialization and distribution.

Human genius is not the only factor in human science, but it can and does speed things up where it is present. Even if intelligence enhancement were treated solely as a means to an end, for solving some very difficult scientific or technological problem, it would still be worthwhile for that reason alone. The solution might not be rapid, even after the problem of intelligence enhancement had been solved, but that assumes the conservative scenario, and the conservative scenario wouldn't last long. Some of the areas most likely to receive early attention would be technologies involved in more advanced forms of superintelligence: broadband brain-computer interfaces or full-fledged Artificial Intelligence. The positive feedback dynamic of the Singularity – smarter minds creating still smarter minds – doesn't need to wait for an AI that can rewrite its own source code; it would also apply to enhanced humans creating the next generation of Singularity technologies.

The Singularity creates speed for two reasons: First, positive feedback – intelligence gaining the ability to improve intelligence directly. Second, the shift of thinking from human neurons to more readily expandable and enormously faster substrates. A brain-computer interface would probably offer a limited but real version of both capabilities; the external brainpower would be both fast and programmable, although still yoked to an ordinary human brain. A true Artificial Intelligence, or a human scanned completely into a sufficiently advanced computer, would have total self-access. At this point one begins to deal with superintelligence as the successor to current scientific research, the global economy, and in fact the entire human condition; rather than a superintelligence plugging into the current system as an improved component. At this point human nature sometimes creates an "Us Vs. Them" view of the situation – the instinct that people who are different are therefore on a different side – but if humans and superintelligences are playing on the same team, it would be straightforward for the most advanced mind at any given time to offer a helping hand to anyone lagging behind; there is no technological reason why humans alive at the time of the Singularity could not participate in it directly. In our view this is the chief benefit of the Singularity to existing humans; not technologies handed down from above but a chance to become smarter and participate directly in creating the future.

9

One idea that is often discussed along with the Singularity is the proposal that, in human history up until now, it has taken less and less time for major changes to occur. Life first arose around three and half billion years ago; it was only eight hundred and fifty million years ago that multi-celled life arose; only sixty-five million years since the dinosaurs died out; only five million years since the hominid family split off within the primate order; and less than a hundred thousand years since the rise of Homo sapiens sapiens in its modern form. Agriculture was invented ten thousand years ago; Socrates lived two and half thousand years ago; the printing press was invented five hundred years ago; the computer was invented around sixty years ago. You can't set a speed limit on the future by looking at the pace of past changes, even if it sounds reasonable at the time; history shows that this method produces very poor predictions. From an evolutionary perspective it is absurd to expect major changes to happen in a handful of centuries, but today's changes occur on a cultural timescale, which bypasses evolution's speed limits. We should be wary of confident predictions that transhumanity will still be limited by the need to seek venture capital from humans or that Artificial Intelligences will be slowed to the rate of their human assistants (both of which I have heard firmly asserted on more than one occasion).

We can't see in advance the technological pathway the Singularity will follow, since if we were that smart ourselves we'd already have done it. But it's possible to toss out broad scenarios, such as "A smarter-than-human AI absorbs all unused computing power on the then-existent Internet in a matter of hours; uses this computing power and smarter-than-human design ability to crack the protein folding problem for artificial proteins in a few more hours; emails separate rush orders to a dozen online peptide synthesis labs, and in two days receives via FedEx a set of proteins which, mixed together, self-assemble into an acoustically controlled nanodevice which can build more advanced nanotechnology." This is not a smarter-than-human solution; it is a human imagining how to throw a magnified, sped-up version of human design abilities at the problem. There are admittedly initial difficulties facing a superfast mind in a world of slow human technology. Even humans, though, could probably solve those difficulties, given hundreds of years to think about it. And we have no way of knowing that a smarter mind can't find even better ways.

If the Singularity involves not just a few smarter-than-usual researchers plugging into standard human organizations, but the transition of intelligent life on Earth to a smarter and rapidly improving civilization with an enormously higher standard of living, then it makes sense to regard the quest to create smarter minds as a means of directly solving such contemporary problems as cancer, AIDS, world hunger, poverty, et cetera. And not just the huge visible problems; the huge silent problems are also important. If modern-day society tends to drain the life force from its inhabitants, that's a problem. Aging and slowly losing neurons and vitality is a problem. In some ways the basic nature of our current world just doesn't seem very pleasant, due to cumulative minor annoyances almost as much as major disasters. This may usually be considered a philosophical problem, but becoming smarter is something that can actually address philosophical problems.

The transformation of civilization into a genuinely nice place to live could occur, not in some unthinkably distant million-year future, but within our own lifetimes. The next leap forward for civilization will happen not because of the slow accumulation of ordinary human technological ingenuity over centuries, but because at some point in the next few decades we will gain the

10

technology to build smarter minds that build still smarter minds. We can create that future and we can be part of it.

If there's a Singularity effort that has a strong vision of this future and supports projects that explicitly focus on transhuman technologies such as brain-computer interfaces and self-improving Artificial Intelligence, then humanity may succeed in making the transition to this future a few years earlier, saving millions of people who would have otherwise died. Around the world, the planetary death rate is around fifty-five million people per year (UN statistics) - 150,000 lives per day, 6,000 lives per hour. These deaths are not just premature but perhaps actually unnecessary. At the very least, the amount of lost lifespan is far more than modern statistics would suggest.

There are also dangers for the human species if we can't make the breakthrough to superintelligence reasonably soon. Albert Einstein once said: "The problems that exist in the world today cannot be solved by the level of thinking that created them." We agree with the sentiment, although Einstein may not have had this particular solution in mind. In pointing out that dangers exist it is not our intent to predict a dystopian future; so far, the doomsayers have repeatedly been proven wrong. Humanity has faced the future squarely, rather than running in the other direction as the doomsayers wished, and has thereby succeeded in avoiding the oft-predicted disasters and continuing to higher standards of living. We avoided disaster by inventing technologies which enable us to cope with complex futures. Better, more sustainable farming technologies have enabled us to support the increased populations produced by modern medicine. The printing press, telegraph, telephone, and now the Internet enable humanity to apply its combined wisdom to problem-solving. If we'd been forced to move into the future without these technologies, disaster probably would have resulted. The technology humanity needs to cope with the coming decades may be the technology of smarter-than-human intelligence. If we have to face challenges like basement laboratories creating lethal viruses or nanotechnological arms races with just our human intelligence, we may be in trouble.

Finally, there is the integrity of the Singularity itself to safeguard. This is not necessarily the most difficult part of the challenge, compared to the problem of creating smarter-than-human intelligence in the first place, but it needs to be considered.It is possible that the integrity of the Singularity needs no safeguarding; that any human from Gandhi to Stalin, if enhanced sufficiently far beyond human intelligence, would end up being wiser and more moral than anyone alive today; that the same holds true for all minds-in-general from enhanced chimpanzees to arbitrarily constructed Artificial Intelligences. But this is not something we know in advance. Since we don't know how many moral errors persist in our own civilization, safeguarding the integrity of the Singularity – in our view – consists more of ensuring the will and ability to grow wiser with increased intelligence than of trying to find perfect candidates for human intelligence enhancement. An analogous problem exists for Artificial Intelligence, where the task is not enforcing servitude on the AI or coming up with a perfect moral code to "hardwire", but rather transferring over the features of human cognition that let us conceive of a morality improving over time (see the section on Friendly Artificial Intelligence for more information).

Safeguarding the integrity of the Singularity is another reason for facing the challenge of the Singularity squarely and deliberately. It may be that human intelligence enhancement will turn

11

out well regardless, but there is still no point in taking unnecessary risks by driving the projects underground. If human intelligence enhancement is banned by the FDA, for example, this just means that the first experiments will take place outside the US, slightly later than they otherwise would have; increasing the possible risks, delaying the possible benefits. If human intelligence enhancement is banned by the UN this means the experiments will take place offshore, out of the public eye, and perhaps sponsored by groups that we would prefer not be involved – although there is a significant chance it would turn out well regardless. In the case of Artificial Intelligence there are certain specific things that must be done to place the AI in the same moral "frame of reference" as humanity – to ensure the AI absorbs our virtues, corrects any inadvertently absorbed faults, and goes on to develop along much the same path as a recursively self-improving human altruist. Friendly Artificial Intelligence is not necessarily more difficult than the problem of AI itself, but it does need to be handled along with the creation of Artificial Intelligence. In both cases, we can best safeguard the integrity of the Singularity by confronting the Singularity intentionally and with full awareness of the responsibilities involved.

What does it mean to confront the Singularity? Despite the enormity of the Singularity, sparking the Singularity – creating the first smarter-than-human intelligence – is a problem of science and technology. The Singularity is something that we can actually go out and do, not a philosophical way of describing something that inevitably happens to humanity. It takes the sweep of human progress and a whole technological economy to create the potential for the Singularity, just as it takes the entire framework of science to create the potential for a cancer cure, but it also takes a deliberate effort to run the last mile and fulfill that potential. If someone asks you if you're interested in donating to AIDS research, you might reply that you believe that cancer research is relatively underfunded and that you are donating there instead; you would probably not say that by working as a stockbroker you support the world economy in general and thereby contribute as much to humanity's progress toward an AIDS cure as anyone. In that sense, sparking the Singularity is no different from any other grand challenge – someone has to do it.

At this moment in time, there is a tiny handful of people who realize what's going on and are trying to do something about it. It is not quite true that if you don't do it, no one will, but the pool of other people who will do it if you don't is smaller than you might think. If you're fortunate enough to be one of the few people who currently know what the Singularity is and would like to see it happen – even if you learned about the Singularity just now – we need your help because there aren't many people like you. This is the one place where your efforts can make the greatest possible difference – not just because of the tremendous stakes, though that would be far more than enough in itself, but because so few people are currently involved.

The Singularity Institute exists to carry out the mission of the Singularity-aware – to accelerate the arrival of the Singularity in order to hasten its human benefits; to close the window of vulnerability that exists while humanity cannot increase its intelligence along with its technology; and to protect the integrity of the Singularity by ensuring that those projects which finally implement the Singularity are carried out in full awareness of the implications and without distraction from the responsibilities involved. That's our dream. Whether it actually happens depends on whether enough people take the Singularity seriously enough to do something about it – whether humanity can scrape up the tiny fraction of its resources needed to face the future deliberately and firmly.

12

We can do better. The future doesn't have to be the dystopia promised by doomsayers. The future doesn't even have to be the flashy yet unimaginative chrome-and-computer world of traditional futurism. We can become smarter. We can step beyond the millennia-old messes created by human-level intelligence. Humanity can solve its problems – both the huge visible problems everyone talks about and the huge silent problems we've learned to take for granted. If the nature of the world we live in bothers you, there is something rational you can do about it. We can do better with your support.

Don't be a bystander at the Singularity. You can direct your effort at the point of greatest impact – the beginning

http://www.singinst.org/overview/whyworktowardthesingularity/

THE AI IMPACT INITIATIVEAdvanced AI has the potential to impact every aspect of human life. We are in a critical window of opportunity where we have powerful but temporary leverage to influence the outcome. Only a small group of scientists are currently aware of the central issues, and it is essential to get input from a broader range of thinkers.

The AI Impact Initiative will foster an interdisciplinary framework for the safe and beneficial deployment of advanced AI. We will form a multidisciplinary body of experts to bring a broad perspective to the critical issue of advanced AI's impact on humanity. This effort will involve researchers with expertise in many different fields, including computer science, security, cryptography, economics, industrial organization, evolutionary biology, cognitive science, political theory, decision theory, physics, philosophy, ethics, religious thought, etc.

The AI Impact Initiative will host meetings over a period of 3-5 years to analyze the central issues and produce strategic recommendations. An inaugural workshop will be held in 2008 or 2009 to bring together researchers from a variety of disciplines and begin the process of unifying their insights. One of the goals of the Initiative is to produce documents that clearly express the central issues so that a broader group of participants may usefully contribute. The Initiative's longer term goal is to lay the foundation for a new science to study these issues. This will involve creating expository materials, building an international network of scientists and scholars, organizing workshops, and creating a comprehensive report to provide direction for future research and development.

Singularity Summit 2007 Podcast: Rodney BrooksAugust 2nd, 2007 – Tyler EmersonDownload audio (mp3)Dr. Rodney Brooks will be keynoting at the Singularity Summit this September 8th and 9th on “The Singularity: A Period Not An Event”. Tickets for the event can be purchased online.

13

Rodney Brooks is the Panasonic Professor of Robotics at MIT. He is also CTO of iRobot Corp (Nasdaq: IRBT). He received degrees in pure mathematics from the Flinders University of South Australia and the Ph.D. in Computer Science from Stanford University in 1981. He held research positions at Carnegie Mellon University and MIT, and a faculty position at Stanford before joining the faculty of MIT in 1984. He has published papers and books in model-based computer vision, path planning, uncertainty analysis, robot assembly, active vision, autonomous robots, micro-robots, micro-actuators, planetary exploration, representation, artificial life, humanoid robots, and compiler design. Dr. Brooks is a Member of the National Academy of Engineering, a Founding Fellow of the Association for the Advancement of Artificial Intelligence (AAAI), a Fellow of the American Association for the Advancement of Science (AAAS), a Fellow of the Association for Computing Machinery (ACM), and a Corresponding Member of the Australian Academy of Science, and a Foreign Fellow of the Australian Academy of Technological Sciences and Engineering.In podcasts, singularity summit | Permalink | 1 Comment | Trackback | Digg this | Add to del.icio.us

SIAI: Why We Exist and Our Short-Term Research ProgramJuly 31st, 2007 – Tyler EmersonBy Dr. Ben Goertzel and Tyler EmersonWhy SIAI ExistsAs the 21st century progresses, an increasing number of forward-thinking scientists and technologists are coming to the conclusion that this will be the century of AI: the century when human inventions exceed human beings in general intelligence. When exactly this will happen, no one knows for sure; Ray Kurzweil, for example, has estimated 2029. Of course, where the future is concerned, nothing is certain except surprise; but the mere fact that so many knowledgeable people (such as Stephen Hawking, Douglas Hofstadter, Bill Joy, and Martin Rees) take the near advent of advanced AI as a plausible possibility, should serve as a “wake-up call” to anyone seriously concerned about the future of humanity. Read the rest of this entry »In intros, outreach, SIAI, singularity | Permalink | 17 Comments | Trackback | Digg this | Add to del.icio.us

Singularity Summit 2007 Abstracts - Third SetJuly 31st, 2007 – Tyler EmersonThe third set of abstracts for the Singularity Summit, this September 8th and 9th at San Francisco’s beautiful Palace of Fine Arts. Tickets are only $50, and can be purchased here.Innovative Applications of Early Stage AINeil Jacobstein, Teknowledge and Institute for Molecular ManufacturingEarly stage artificial intelligence has already produced a wide range of valuable but narrowly focused knowledge systems applications in industry and government. Many of these applications have performed complex tasks such as planning, monitoring, design, risk assessment, diagnosis, training, process control, classification, and analysis. For example, AAAI’s Innovative Applications of AI Conference has published hundreds of successful applications of AI. The applications are in fields as diverse as biotechnology, space flight, manufacturing, security, paleontology, construction, energy, music, military, intelligence, banking, telecommunications,

14

news media, management, law, emergency services, agriculture, treaty verification, and many other areas. This talk will review the distribution of these applications across tasks and domains, and discuss the patterns that connect these applications: what worked, what didn’t, and what are the key trends. None of these systems exhibited general intelligence, but each documented our ability to codify and distribute human problem solving knowledge, and put it to work. The answer to the question about how far are we from advanced AI depends on the operational definition of “advanced”. It is clear from the knowledge systems produced thus far that even relatively straightforward applications can be valuable. The larger endeavor to produce AI systems that learn and reason at human levels and beyond is promising, and will require both enlightened research sponsorship and appropriate safeguards.The Nature of Self-Improving Artificial IntelligenceStephen M. Omohundro, Self-Aware SystemsCan we predict the behavior of systems that modify themselves? Can we design them to embody our values even after many generations of self-improvement? This talk will present a framework for answering questions like these. It shows that self-improving systems converge on a specific cognitive architecture that arose out of von Neumann’s foundational work on microeconomics. In these systems there is a universal principle which governs the organization of all levels of physical and computational resources. They exhibit four natural drives: 1) efficiency, 2) self-preservation, 3) resource acquisition, and 4) creativity. Unbridled, these lead to both desirable and undesirable behaviors.The efficiency drive leads to algorithm optimization, data compression, atomically precise physical structures, reversible computation, adiabatic physical action, the virtualization of the physical, and governs a system’s choice of memories, theorems, language, and logic. The self-preservation drive leads to defensive strategies such as “energy encryption” for hiding resources and promotes replication and game theoretic modelling. The resource acquisition drive leads to a variety of competitive behaviors and promotes rapid physical expansion and imperialism. The creativity drive leads to the development of new concepts, algorithms, theorems, devices, and processes.The best of these traits could usher in a new era of peace and prosperity; the worst are characteristic of human psychopaths and could bring widespread destruction. How can we ensure that this technology acts in alignment with our highest values? We have leverage both in designing the systems’ initial values and in creating the social context within which they operate. But we must have great clarity in imagining the future we want to create. We need not just a logical understanding of the technology but a deep introspection into what we cherish most. With both logic and inspiration we can work toward building a technology that empowers the human spirit rather than diminishing it.Preparing for Bizarreness: Open Source Physical SecurityChristine L. Peterson, Foresight Nanotech InstituteAttempting to take action now to get ready for a world with strong AI is a highly daunting task. Nevertheless it is worth considering our options, especially any that are useful in the nearer term for other reasons. We can ask: In a world of powerful entities, how can individuals be protected?The open source software experience inspires us to look for ways to transfer the advantages of that process to the physical world. Open source has been particularly speedy at correction of security vulnerabilities – precisely the kind of vulnerabilities we will need to guard against in a world of highly powerful entities of various kinds. We can begin now to extend the principles of

15

open source into the physical world: we can start to make physical security “bottom-up”, decentralized, collaborative, and transparent.In No Tags | Permalink | 1 Comment | Trackback | Digg this | Add to del.icio.us

Vernor Vinge PodcastJuly 31st, 2007 – Tyler EmersonDownload audio (mp3)Cameron Reilly’s latest podcast interview is with Vernor Vinge, one of the best science fiction authors in recent decades. His novels include True Names, A Fire Upon the Deep, A Deepness in the Sky, and Rainbow’s End. He popularized the term “technological singularity” in his 1993 essay “The Coming Technological Singularity“.In podcasts, singularity | Permalink | No Comments | Trackback | Digg this | Add to del.icio.us

Do You Care About Hypothetical Persons?July 30th, 2007 – Michael AnissimovAt the Transvision conference, I had a conversation with a respected transhumanist on the issue of existential risks and humanity’s future. He told me that he did not see existential risk as a big deal because of it threatening hypothetical persons in the future, but only because of threatening the currently living population. This is the first time ever that anyone told me directly that they use a discount rate of infinity when considering as-yet-to-be-born persons.When environmentalists tell us to fight against global warming, and economists warn us about the insolvency of Social Security, an often used motivator is to tell us to think about the world we are handing off to our children. This may refer to one’s own children, but can be abstracted to ‘descendants’ - which includes other people’s children, the sum total continuation of the human and eventually posthuman race.What is confusing is that this motivator seems to work a lot on some people and not at all on certain others. Despite the majority assigning some level of concern to hypothetical persons, at least their immediate children, grandchildren, and even great-grandchildren, a significant minority assigns them nothing. Having no crystal ball of where moral philosophy and consensus will go in the future, it is very difficult for us to consider from our current vantage point whether or not this tendency will be viewed in retrospect as an irrational bit of evolutionary baggage, or genuine moral wisdom.If we do care about hypothetical persons, and want to care more, might we eventually be able to reprogram our minds to magnify this aspect of ourselves as beneficial? Will we begin to care more about the teraperson hypothetical collective in the Whirlpool Galaxy, 25 million years from today, than our next-door neighbor we’ll see when we walk out the door in ten minutes? This certainly seems to be what Nick Bostrom is suggesting in his Astronomical Waste paper:“With very advanced technology, a very large population of people living happy lives could be sustained in the accessible region of the universe. For every year that development of such technologies and colonization of the universe is delayed, there is therefore an opportunity cost: a potential good, lives worth living, is not being realized. Given some plausible assumptions, this cost is extremely large. However, the lesson for utilitarians is not that we ought to maximize the pace of technological development, but rather that we ought to maximize its safety, i.e. the probability that colonization will eventually occur.”Despite Bostrom’s persuasive paper, there are people that simply don’t care, because they only value lives as currently lived. As far as I can tell, there is nothing morally perverse about these

16

people - they simply have a different angle on the moral issue. Is one “right” and the other “wrong”? Maybe, but ironically, it would require a hypothetical future person to give the answer with confidence.In No Tags | Permalink | 17 Comments | Trackback | Digg this | Add to del.icio.us

SIAI Interview Series: Ben Goertzel, Singularity InstituteJuly 30th, 2007 – Tyler EmersonDr. Ben Goertzel is SIAI’s Director of Research. In this interview, he explains the Singularity Institute’s mission and research objectives. You can download the audio version here.In artificial intelligence, interviews, outreach, SIAI, singularity, videos | Permalink | 1 Comment | Trackback | Digg this | Add to del.icio.us

Singularity Summit 2007 Abstracts - Second SetJuly 25th, 2007 – Tyler EmersonThe second set of abstracts for the Singularity Summit, this September 8th and 9th at San Francisco’s beautiful Palace of Fine Arts. Tickets are only $50, and can be purchased here.Metaverse SingularityJamais Cascio, Center for the Responsible NanotechnologyThere are numerous scenarios for how the Singularity might transpire, but implicit in most is the notion that the technologies that trigger the Singularity themselves emerge from earlier generations of systems and tools. One particularly rich potential progenitor is the spectrum of technologies encompassed by the term “Metaverse.” Building upon my work in the recently-published Metaverse Roadmap Overview, I trace how each of the four Metaverse scenarios – Augmented Reality, Lifelogging, Virtual Worlds and Mirror World – lead to very different types of Singularities. I look at the ways in which these different Singularity models might interact, and the implications each have for the likelihood of friendly and unfriendly AI.Nine Years to a Positive Singularity – If We Really, Really TryDr. Ben Goertzel, Singularity Institute for Artificial Intelligence and NovamenteCommon wisdom holds that powerful artificial general intelligence is decades to centuries off. Even techno-futurist Ray Kurzweil projects a date of 2029 for human-level AI via human brain emulation. My contention, however, is that powerful and beneficial AGI could come much sooner – if sufficient attention and resources are devoted to the right approaches. My favored approach involves integrating probabilistic and evolutionary learning, artificial economics, and other cutting-edge computer science techniques in a cognitive architecture informed by cognitive science and systems theory; and then embedding this architecture in virtual agents that interact with humans and each other in online virtual worlds. Among other advantages, I argue that this sort of AGI architecture is intrinsically better suited for stably ethical behavior than more closely human brain based architectures, due to the presence of a coherent and logical goal system. Current prototype work will be discussed, aimed at actualizing this approach via the release of intelligent agents controlled by the Novamente AI Engine in Second Life and other virtual worlds. Of course, it is difficult to place any kind of reliable estimate on the course of development of this kind of technology, given the R&D that remains to be done, and the uncertainties regarding funding and other practical exigencies. But radical success within less than a decade does not seem an outrageously unlikely possibility, in the view of this AGI researcher and entrepreneur.

17

Increased Intelligence, Improved LifePeter Voss, Adaptive AIArtificial General Intelligence (AGI) promises unprecedented advances not only in science and technology, but also in ethics and social systems. However, business – and thus consumers – will be first to experience some of the enormous benefits of this emerging technology. This talk will explore some of these improvements, and try to make a case for how increased intelligence leads to improved morality.In singularity summit | Permalink | No Comments | Trackback | Digg this | Add to del.icio.us

Valuing AIsJuly 22nd, 2007 – Seth BaumHow much is an AI worth?First, let’s distinguish between “instrumental” value and “intrinsic” value. Instrumental value is value to something else (usefulness); intrinsic value is inherent value that requires nothing else for it to be worth something. For example, we could say that clothing has instrumental value because they keep us warm, and our feeling of warmth has intrinsic value because it is worth something even in the absence of anything else. Or, we could say that books have instrumental value because they bring us knowledge and our having knowledge has inherent value. (For more, see value theory on Wikipedia.)Read the rest of this entry »In No Tags | Permalink | 16 Comments | Trackback | Digg this | Add to del.icio.us

Singularity Summit 2007 Abstracts - First SetJuly 21st, 2007 – Tyler EmersonAbstracts from Singularity Summit speakers are starting to arrive. Here are the first three.The Singularity: A Period Not An EventRodney Brooks, MIT Computer Science and AI LaboratoryWhatever writes future history will look back at what we are calling the singularity not as a single event but as a period of time. The singularity period will encompass a time where a collection of technologies were invented, developed, and deployed in fits and starts, driven not by the imperative of the singularity itself, but by the normal economic and sociological pressures of human affairs. A Hollywood treatment of the singularity would have a world just like today’s, plus the singularity, as a singular event. In reality the world will be changing continuously due to rapid growth in technologies that are both related and unrelated to the singularity itself. The future will be embedded in a different world than the one we inhabit. And the AI systems we create will not have the same desires, beliefs, and goals as today-us. Tomorrow-us will be much better equipped for the changes that will take place in our world. This talk will explore how things might unfold and how we will transform ourselves along the way.Waiting for the Great Leap…Forward?Dr. James Hughes, Institute for Ethics and Emerging TechnologiesSentient, self-willed, greater-than-human machine minds are very likely in the next fifty years. But to ensure that they don’t threaten the welfare of the rest of the minds on the planet a number of steps need to be taken. First, given their radically different architecture and origins, developing software capacity for recognizing and relating to, perhaps having empathy for, human sentience should be a design goal, even if machine minds are likely to evolve beyond human perspectives and emotional traits. Second, building on the global networks established to

18

identify and respond to computer viruses, governments and cyber-security firms need to develop detectors for and counter-measures for self-willed machine intelligence that may emerge, evolve, or be accidentally or maliciously released. Those detectors and counter-measures may or may not involve machine minds as well. Third, human beings should aggressively pursue cognitive enhancement and cyber-augmentation in order to give them a competitive chance against machine minds, economically and in th event of conflict. Fourth, since machine intelligence, self-willed or zombie, is likely to displace the need for most human occupations by the middle of the century, industrialized countries will need to renegotiate the relationship between education, work, income, and retirement, extracting a general social wage from robotic productivity to lift all boats, not just those of the shrinking group of workers and owners of capital. Finally, in order to ensure that we do not re-capitulate slavery, we will need to be much clearer about what kinds of minds, organic and machine, have what kinds of responsibilities and are owed which kinds of rights. Machine minds with a capacity to understand and obey the obligations of a democratic polity should be granted the rights to own property, vote and so on. Minds wishing to exercise capacities as dangerous as weapons or motor vehicles, should be licensed to do so, while even more dangerous capacities (AI equivalents of bombs) will need to be restricted to control by, or be integrated into the functioning of, accountable democratic governance. The Road to Singularity: Comedic Complexity, Technological Thresholds, and Bioethical Broad JumpsWendell Wallach, Yale Interdisciplinary Center for BioethicsThe prospect of implementing higher order cognitive faculties in AI presumes that theories about the computational nature of mind are valid, that known technological issues can be solved, that there are no major surprise technological thresholds that will need to be crossed, and that computer scientists and public officials will find ways to navigate a broad array of ethical challenges. While some of these concerns have received considerable attention, others are just beginning to be noted. The ethical challenges, in particular, have not been well addressed. From robots carrying weapons, to moral decision making faculties for AI, to institutional review boards for robotic research, and political resistance to some categories of AI research, the bioethical challenges, if not addressed, could potentially undermine funding and public support for advanced AI systems. Progress in developing moral decision making faculties for computers is one area that engineers and designers can begin to tackle, and which will have a significant impact. The successful development of artificial moral agents (AMAs) is a major step that will help ameliorate other societal concerns regarding the development of advanced AI. The pathways for implementing moral decision making faculties in AI include top-down, bottom-up, and hybrid approaches. In addition, AMAs may require supra-rational faculties, such as social skills, emotions, consciousness, and a theory of mind.In singularity summit | Permalink | 3 Comments | Trackback | Digg this | Add to del.icio.us

Skepticism about Powerful AI and SIAI’s MissionJuly 11th, 2007 – Tyler EmersonSome skepticism about powerful AI and thus SIAI’s mission is reasonable, but it is important to have a sense of proportion. Even if you are skeptical about powerful AI to the point where you only give a 5% chance for its occurrence, it is important to balance your forecast with the expected utility residing within that 5% chance to realize an astronomically large social good. There are strong preexisting arguments in support of an astronomical amount of human value resulting from realizing powerful AI safely. You need to have wide confidence intervals in both

19

directions given the current uncertainty residing within this new area of knowledge. You cannot just have it in the direction that favors your particular forecast. That 5% forecast also needs to account for the poor track record humans have as forecasters. Anyone who thinks their technical assessment of the issues is sound enough to justify a 5% (or 90%) forecast is either someone we should talk with, or someone showing sizable ignorance. However, this misses the point: Even if it were possible to know that a 5% assessment were sound, the amount of social good that could be achieved by realizing powerful AI argues that the amount of effort presently marshaled in this direction is a rare form of human irrationality.Unfortunately, very few people think like this. There are some remarkable people who already do, though, and I hope there are more soon. A few include Peter Thiel, Aubrey de Grey, and Barney Pell. I recommend exploring carefully how they think about these issues.

2006 CONFERENCE ARCHIVES:

The Singularity Summit at Stanford, the first academic symposium focused on the singularity scenario, brought together 1,300 people and 10 speakers to explore the future of human and machine cognition, including Ray Kurzweil, Dr. Douglas R. Hofstadter, and Dr. Sebastian Thrun. Press coverage included a feature article in the San Francisco Chronicle.

Smarter than thou?Stanford conference ponders a brave new world with machines more powerful than their creatorsTom Abate, Chronicle Staff WriterFriday, May 12, 2006

Is technology poised to develop machines that can outsmart their human creators? And what will happen to mere mortals if such superintelligent machines arise? These will be among the questions pondered when experts in artificial intelligence, brain research and other futuristic fields gather at Stanford University on Saturday for what is being called the Singularity Summit. [TECH TALK: Chronicle innovation reporter Tom Abate interviews author/inventor Ray Kurzweil on whether technology is poised to make machines that can outsmart their human creators?] Borrowing a term from physics, singularity suggests a horizon beyond which we can't see. It describes the point at which some form of intelligence spawned by technology gains the ability

20

to rapidly improve its own programming -- becoming so powerful that we cannot predict what it might do. At that point, its capabilities could exceed even the power of our imaginations. "This could be very, very good if we get it right, and very, very bad if we get it wrong,'' said Eliezer Yudkowsky, a research fellow with the Singularity Institute for Artificial Intelligence, a nonprofit group in Palo Alto that is co-sponsoring the event. The speakers' lineup will include inventor and author Ray Kurzweil, whose recent book, "The Singularity Is Near," argues that a fusion of machine and biological intelligence is not only imminent but beneficial. "It's not going to be an invasion of intelligent machines coming over the horizon,'' Kurzweil said recently. "We're going to merge with this technology. ... We're going to put these intelligent devices inside our bodies and our brains to make us live longer and healthier." More-skeptical speakers will include Douglas Hofstadter, a cognitive scientist at Indiana University who is probably best known for his Pulitzer Prize-winning book, "Gödel, Escher, Bach." "I don't think it's inconceivable that some kind of singularity entity could eventually have superior intelligence to humans, but I'd be very surprised if anything remotely like this happened in the next 100 to 200 years,'' Hofstadter said, adding that if and when superintelligent machines arise, the question will be, "whether we become animals in the zoo, or go extinct or just coexist (with it) like ants.'' Organizers say more than 2,000 people have already signed up to hear these heady topics discussed. They suggest that anyone who is not already a confirmed registrant for the free event arrive early Saturday to wait in line for cancellations at Stanford's Memorial Auditorium. Other speakers will include: -- Max More, chairman of the Extropy Institute, a nonprofit group that espouses lengthening the human lifespan and making other "improvements" to human physiology and character through technology; -- Nick Bostrom, director of the Future of Humanity Institute at Oxford University, which also advocates "human enhancements" while simultaneously pondering the risks that a global catastrophe -- whether self-inflicted through thermonuclear war or naturally occurring as in an asteroid strike -- could wipe out the human species; -- Environmentalist Bill McKibben, author of "Enough: Staying Human in an Engineered Age," who is expected to argue against such technological improvements. In a way, the daylong summit is shaping up as the Bay Area coming-out party for the tech-inspired philosophy called transhumanism. In a nutshell, transhumanism holds that genetics, nanotechnology and robotics are converging, creating the potential for "human enhancements." Saturday's event is supported by the Stanford Transhumanist Association, a student group that embraces the view that humankind is poised to take evolution into its own hands. "We should view ourselves as a species in transition," said Michael Jin, a 20-year-old sophomore and founding member of the group. Jin and fellow sophomore Yonah Berwaldt have been pleased at how quickly the event drew a close-to-capacity crowd based largely on word of mouth and blog posts. "There's a large audience for this sort of thing on the Internet,'' said Jin, who says it is possible and desirable to engineer away psychological flaws such as selfishness. "This is a troubling aspect of human nature and something we could actually fix,'' he said.

21

Although little known outside technological circles, transhumanism inspires intense opposition from ethical watchdog groups that dispute the notion that such technological tweaking would represent progress. "As soon as you take issue, you're quickly labeled a Luddite,'' said Jennifer Lahl, national director of the Center for Bioethics and Culture Network in Oakland. "But transhumanism begs the question: What needs to be improved upon, who gets to decide and where does it end?" Richard Hayes, executive director of Oakland's Center for Genetics and Society, likened modern transhumanists to the early 20th-century futurists who were fellow travelers with the fascist movements of that era. "The transhumanists are fundamentally elitists," Hayes said. "Once they start enhancing themselves toward post-human status they will have little concern with the rest of humanity." Yudkowsky, the artificial intelligence expert with the Singularity Institute, said those fears stem from Hollywood images such as the part-human, part-machine Borg of the "Star Trek" series whose collective consciousness is akin to a form of telepathy. "How do they (scriptwriters) know that telepaths aren't nice people?'' he said. Nevertheless, the Singularity Institute sees itself as a watchdog, working inside this movement to ensure that if and when smarter-than-human machines arise that they would behave like benign genies to help solve such problems as global warming. "Humanity seems to have two choices in the long run: superintelligence or extinction," Singularity Institute Executive Director Tyler Emerson said. Todd Davies, a lecturer at Stanford and associate director of the university's cross-disciplinary Symbolic Systems Program, said he knew the Singularity Summit would be controversial and tried to ensure some diversity of views on the agenda. "I'm not at all convinced the singularity is near or that it will be a good thing,'' Davies said, adding that, "having the summit is a way to get these ideas on the table.

http://www.sfgate.com/cgi-bin/article.cgi?file=/c/a/2006/05/12/BUG9IIMG1V197.DTL&type=printable

2006 Audio and Video presentations:

http://www.singinst.org/media/

22