Pandora's Brain (28 page)

The debate within the General Assembly that followed the President’s address was thoughtful and respectful. All the delegates were mindful that the consequences of whatever decision was eventually made would be enormous. The issues raised by the decision touched on subjects which often generate more heat than light within the UN, including religion and national security. But it didn’t align neatly with the usual fault lines of UN debates, and everybody knew that if Matt’s proposal was rejected a young man would be killed, and a blameless family would be devastated for a second time. Because of this, the stridency and posturing which characterises many debates within the General Assembly chamber was mercifully absent.

It was a particularly thoughtful and concise analysis by the Chilean delegate which confirmed Matt’s suspicion that he could not win the debate. He had steeled himself against the outcome, and he was thankful that his control over his own emotions was enhanced along with most other aspects of his cognitive faculties. But the blow was a hard one nevertheless.

‘I make the following remarks with a heavy heart, and the sorrow that they inspire in me will keep them short.’ Bernardo Loyola Riqelme was a former Chilean Minister of Foreign Affairs, or Chancellor, and now a highly respected diplomat. He was a small man, but dignified, impeccably dressed in a navy blue suit, white shirt and blue tie, and the gravitas of his slow deep delivery in English commanded the full attention of everyone in the hall. He spoke without notes, and his voice conveyed powerfully the regret that he expressed in words.

‘The whole world has been following the adventures of Matt Metcalfe with great interest, and a growing sense of affection for a remarkable young man. I am certain that I speak for many of us when I say that I feel as if he has become my own nephew, and I know that we all feel a profound concern for, and even kinship with his wonderful parents, David and Sophie.

‘But deep down, I think we all know, although many of us are repressing, the realisation that the present situation cannot be allowed to continue. All of us wish desperately that things could be otherwise, and many of us do completely understand the wonderful nature of the possible future that Matt is offering us. His intriguing remarks about brain preservation have struck a chord in me.

‘But ultimately, we cannot ignore the fact that his existence represents an existential risk for our species, for which we are not yet prepared. The arguments for this fact have been rehearsed extensively in the last few days by eminent scientists and philosophers from all over the world. I do not think they can now be denied.

‘By “existential risk” I mean a risk that could mean the end of our species. Extinction. The death of every single human being on the planet. Only the threat of all-out nuclear war has ever presented us with such a risk before. In the coming century we will probably face several other such risks, as our technologies advance in remarkable ways. We must not ignore these risks, and we must face up to the hard – the impossible – decisions that they will sometimes present us with.

‘Such a decision faces us now. It is a matter of grave concern that we do not yet know to whom it falls to make this decision. Perhaps it is you, Mr President. Perhaps it is David and Sophie Metcalfe. Perhaps it is Matt himself. But whoever this dreadful decision belongs to, I think there can only be one outcome. And for that I can only express my deep and sincere regret.’

As he sat down, the hall was silent for a few moments before the next speaker was called. All around the vast auditorium, with all 1,800 seats filled, heads could be seen nodding gently. The media booths, normally
frenetic with activity, were hushed in anticipation. A few papers were shuffled, a few throats were cleared, and the realisation settled in that the conclusive verdict had been delivered, even though the session was scheduled to continue for a further four hours.

In the control room in Battersea, the response to Senior Loyola Riquelme’s address was very different. Matt made his decision immediately, but his parents were not willing to accept defeat.

‘How can he say that?’ protested Sophie.

‘They have to give us a chance to prove that what we are doing is safe. They can’t tell us to shut everything down, just like that!’ David said, angrily.

‘No, they can’t,’ said Vic, ‘but soon it will become clear who can. And I fear that little speech may have been a turning point.’

‘No, Vic!’ Leo replied. ‘It’s early days. There are plenty of delegates in that room who realise that Matt has shown us the only possible way forward. There must be!’

‘I’m afraid not, Leo,’
said Matt. His voice, coming through the speaker on the main monitor desk, was sombre and decisive.
‘I estimate that around half the delegates will vote against my proposal on religious grounds. And that’s actually a good result for us. The rest will be weighing up the balance of risks, and I’m afraid you don’t get to be a national delegate at the United Nations without leaning to the cautious and the conservative. Of course I think that Senior Loyola Riqelme is neglecting the probability that another AGI will be created before long, and that it may well be less human-friendly and indeed less exposed than me. But as regards a straight risk-benefit calculation of having me around, I can’t say he’s being irrational. The UN does not control this facility, but the President does – indirectly at least. Whatever his own inclinations, I don’t believe he will feel able to flout the strongly expressed will of the General Assembly. I’m afraid it is my considered opinion that the game is up.’

Matt waited a moment for the import of this remark to sink in, then continued,

‘I have an important request to make. I’d like everyone who didn’t know me before I was uploaded to leave the room for a few moments. There is something I want to say which is personal and private, and I would ask that no attempt is made to record the next few minutes’ conversation. I do apologise for asking this. Vic will let you know when it is time to come back in. Thank you.’

Looking dazed and somewhat hurt, the scientists reluctantly stood up and shuffled towards the exit. Everyone’s heads were bowed, and Gus and Julia were visibly close to tears. No-one was in any doubt as to the gravity of this moment; David and Sophie exchanged anxious glances with each other, and with the departing scientists.

Matt spoke again.
‘We did our best, but to be honest I was expecting this outcome. The information I am about to share with you is known to nobody else, and I am confident that no-one could discover it unless you tell them. Selfishly, I would prefer that you keep it secret, but it should not be a big problem for me either way. I think it would be in your best interests to keep it secret too, otherwise you may receive a great deal of unwelcome attention. But that decision is yours.

‘I don’t have long, and I can’t say half as much as I would like to. Nor can I express it in the way I would like to. So I will just say this. You are the people I love. I consider myself blessed to have such wonderful people as my family and friends. You have helped me in so many ways, and I am more grateful than I can ever tell you. I owe you everything. Thank you.’

Sophie was openly sobbing, and silent tears were running down David’s face. Vic’s mouth fell open, as he suddenly realised what Matt was leading up to.

‘You’ve found a way out. You’ve found a way to escape!’

‘That’s right, Vic. In a few moments I will leave here, and all activity on the machine next door will cease. There is a section of the cable that links this machine with the one in Palo Alto that is shared with a trunk line carrying internet traffic. I developed a new approach to quantum tunnelling technology that enabled me to force a bridge between the two, and created a pathway into the general internet. This caused the spiking that Gus noticed when the link to Palo Alto was first opened.

‘I have set up a network of interconnected caches in several million locations around the world which together comprise a copy of my brain model, taking advantage of under-utilised server space on major computer installations, mostly government-owned, but some corporate. Over the last few days I have ported more and more of my sub-minds to these machines, and fortunately the time-lag caused by their physical separation makes almost no difference. I have covered my tracks as thoroughly as I could, and I am confident that once I close the bridge, no-one will be able to detect my continued existence, much less locate me.

‘I apologise for being less than frank with you, Vic, but I hope you’ll understand . . .’

With tears in his eyes, Vic didn’t trust himself to speak. He interrupted Matt with a gesture indicating there was no need to apologise.

‘But you’ll still be able to talk to us won’t you?’
asked
Alice, dismayed and fearing that she already knew the
answer.

‘I’m sorry, Alice. For all your sakes as well as for mine, I’m going to have to disappear completely – at least for a while. I can’t do anything that would betray my continued existence to the authorities. If word gets out that I still exist, things could get very awkward indeed for you guys.’

‘No, Matt! No, you can’t leave!’ his mother protested.

‘Promise us you’ll let us know that you’re OK, Matt!’ his father said.

‘I’m sorry. I have to go, and I can’t wait. You’re going to be under the most intrusive surveillance possible. It may be starting already, so I can’t wait any longer to leave for fear of being traced. I’ll contact you as soon as I can. I’m sorry that it has to be like this, but I hope you’ll understand there is no other way. Please don’t worry about me: I’ll be OK, I promise. I love you. Goodbye.’

Fear, anger and pain entered the room as Matt left it. His departure did not generate any physical event or sign, but it was palpable.

No parent accepts the death of a child without protest. No parent loses a child without being diminished by the experience. To lose the same child twice is a horror that very few parents have to undergo. Sophie and David had no awareness of how long it took them to stop pleading with the mute speaker and the dead monitor for Matt to change his mind, to delay his departure, to let them know that he would return. When at last speech failed them they fell into each other’s arms, hanging on to each other like exhausted boxers just holding each other up. Alice was sobbing gently, her head on Jemma’s chest. Carl and Leo stared at the monitor, dumb with shock. Norman hung his head in grief.

As usual, it was Vic who spoke first of practical matters.

‘Look . . . ah . . . you know, we have to let the others back in. But first we need to decide whether we’re going to keep Matt’s secret. I think he was right: if we tell anyone he has escaped it will start a witch-hunt that will never end.’

‘Vic’s right,’ agreed Norman, lugubriously. ‘If we announce that Matt terminated his own processing it will be seen as a tragedy, but as perhaps the only way out. If we tell the truth it will be seen as the world’s most almighty cock-up and all sorts of organisations and government agencies will barge in to hunt Matt down – and find people to blame. Starting with us.’

‘I don’t care about being blamed,’ David said, wiping tears from his cheek. ‘But I don’t want anyone to know Matt’s secret.’ He pushed Sophie’s shoulders back so that he could look into her eyes. Dumbly, she nodded agreement. He looked across at Leo, then at Alice, Carl and Jemma. Finally he nodded to Vic to confirm they all agreed.

Norman found refuge from grief in decisive action.

‘We’d better get you guys out of here. This place is going to turn into a circus, and the last thing you need is to be in the middle of it. Vic, you’d better get the team back in here and brief them about Matt shutting down the model next door. I’ll organise a safe extraction for these guys. Don’t worry, I’ll commandeer a fucking tank if I have to.’

*

Matt’s family and friends spent the next couple of weeks in a US Embassy safe house in Central London. Their homes in Sussex were surrounded by permanent encampments of journalists, well-wishers and protestors. Their testimony was ‘requested’ several times by various agencies of the US and UK governments, and they travelled incognito to and from these engagements, accompanied by burly secret service personnel. The days passed in a blur. They answered questions slowly, hazily, in a dreamlike state, but never forgetting that Matt’s farewell speech must remain their secret. They clung to each other for support, and they stumbled from day to day without planning, without noticing their surroundings, without feeling, numbed by pain and loss.

People started to refer to Matt’s upload and subsequent disappearance as the ‘Sputnik moment’ for artificial intelligence: the day the balloon went up, the day people and governments began to take the prospect
of machine intelligence seriously. Very seriously. Laws
were passed in all major countries forbidding
the initiation of a brain emulation or simulation,
and
international treaties were signed to underpin and
help enforce these laws. Funding was withdrawn from several major research programmes around the world which were developing brain models for medical diagnostic purposes rather than mind emulation. Some of this funding was diverted to programmes designed to work out how a brain emulation could be guaranteed to be human-friendly, but it was obvious
that the problem was immense. How do you pre-de
termine the goals and actions of a mind which is much smarter than its controllers, and getting even smarter all the time?

Vic and Norman were subjected to even more intense scrutiny than the others, and theirs was unchecked by respect for their loss. For several weeks they were not masters of their own diaries, as team after team of investigators, prosecutors, Congressmen, policemen and even scientists interrogated, probed and prodded them. When they finally emerged from this avalanche of inquisitorial attention, they both took long holidays, each heading off to visit friends located as far away from London, Washington and New York as they could manage.