Avogadro Corp: The Singularity Is Closer Than It Appears (14 page)

“
But?”

Gene pulled a fourth set of papers from his accordion folder. “It turns out that the same behavior is happening in another department. The Offshore Data Center project. Their expenditures exhibit some of the same characteristics. Multiple line items just under one million dollars, budgets coming in at just under one penny less than their limit.”

“
But that’s not under Gary?”

“
No, it’s not. Which makes it more likely that someone has hacked Avogadro’s procurement system.”

“
But to what purpose?” Maggie asked. She scanned the papers again. “Why would someone risk their job and even jail to order servers, satellite communication systems, and hire contractors? It makes no sense.”

“
I agree,” Gene said. “I was hoping that you would be able to shed some light on this, and that maybe there would be an easy answer.” He began to pick up his papers and put them back into the accordion folder. “I’m going to keep investigating this. Please don’t discuss it with anyone.”

Maggie nodded.

Gene stood up. “Thanks for your time, Maggie.” He let himself out, leaving Maggie full of questions.

* * *

From the time Mike arrived at the airport in Madison for the flight home, and periodically since he arrived back in Portland, he had tried to reach David by phone. Frustratingly, David had been off the grid in New Mexico. Mike knew that David always went to Christine’s family ranch for the holidays, and he knew that the ranch was off the grid, so he couldn’t claim any legitimate reason for feeling even more suspicious. Yet here he was, feeling manipulated by a software algorithm.

David had sent Mike a copy of his itinerary by email weeks earlier, so he had David’s flight information. Without even realizing what he was doing, he found himself standing in his living room, holding a printed copy of the itinerary, watching the clock. He had his jacket on, alternating between sitting on the couch tapping his toe and getting up to pace the living room. Finally, with more than an hour left before the flight would arrive, he drove to the airport.

Mike drove with furrowed brows through a light drizzle, with the streets threatening to ice over. His thoughts were bordering on obsessive. What had David done to ELOPe? What was ELOPe doing? Why was he locked out of the servers? He swore as his front tires spun crossing the light rail tracks, and the car fishtailed. He fought the wheel and recovered halfway through the turn onto Airport Way.

He sped into the parking garage, circling up the ramp, and parked near the skybridge. He walked across the covered bridge at a high speed, and stalked through the airport. He got to the security gate, and looked at his phone. Still nearly an hour until David’s flight would arrive.

He checked the flight monitors once, then forced himself to sit down and stay there, trying to squelch his nervous energy. He watched whole families disgorge through the security exit; suitcases, car-seats, and exhausted children in tow. He smiled as he watched a young woman welcome a man home with a single flower and a long embrace. He missed the bittersweet tang of young love.

Then he saw David and Christine coming through the gate. He walked over and waved. David and Christine were delighted to see him. Christine gave him a big hug, and a bigger smile. Standing in the middle of the stream of exiting passengers, Mike launched into an immediate explanation in a hushed voice. David and Christine leaned closer to listen, and their smiles vanished.

“
My dad was absolutely fine. My mother never sent any emails at all. I’m telling you, I’m convinced that ELOPe originated those emails. What I don’t understand is why.”

“
Did you get my email before you left?” David asked. “The one about the override I put in?”

Mike gave him a blank look. “No, but it sounds like you should start explaining.”

Before David could say anything, a TSA agent asked them to move out of the way of people.

“
Let’s get to your car,” David said. “Then I’ll tell you.”

They rushed to the parking garage, made their way to Mike’s Jetta, and threw the bags in the trunk. Mike gripped the wheel tightly, while David sat next to him, and Christine sat in the middle of the rear seat, leaning forward.

“
I put in an override,” David said once they got out of the garage. “I didn’t just use ELOPe to send an email to Gary. I have ELOPe filtering every company email.” David slammed the dashboard with his fist, startling Mike, who swerved sideways, skidding on the icy street, before regaining his lane.

“
I sent you an email, the night you came over for dinner. I told you all this, and that I needed your help to do a live-patch on the server.”

“
I never got it,” Mike said. ”I’ve been through all my emails from you.” He sighed. “This explains so much. Didn’t it occur to you that ELOPe would fabricate that email to get rid of me?”

“
No, it didn’t.” David shook his head. He raised his voice defensively. “Look, I didn’t think that you would fly off to Wisconsin based only on an email.”

“
Hey, I had just heard that my father had a heart attack. I was freaking out. And it’s your damn fault I had to go through that.” Mike was yelling now, and David leaned away in his seat.

“
OK, chill out guys. If the two of you are fighting, you are not going to be thinking straight.” Christine said, also yelling to get their attention. “Look, what happened when you got back Mike? Did you look into ELOPe?” She spoke calmly, placatingly.

“
I tried, but I couldn’t get access to either the source code or the system logs,” Mike said, resigned. “I assumed David had locked down access so that no one would find what he had done. And I didn’t want to say anything to any of the rest of the team, because I didn’t want to raise suspicions. I was still trying to cover for you, David.” Mike glared at him.

“
I’m sorry, Mike,” David said gently. “I shouldn’t have said that about you and your dad. I’m really sorry you flew halfway across the country, but I’m glad your father was fine.”

Mike hesitated a minute, then nodded slightly, accepting the apology. “Well, did you do anything to lock down the system?” he asked.

“
No. In retrospect, it sounds like a great idea to have locked down access to ELOPe, but I didn’t do it.”

“
Shit, then somehow ELOPe has removed my access to the servers and the code.”

“
Ugh, guys, that seems impossible.” Christine said. “Even if you are right, Mike, and ELOPe is somehow originating emails on its own, it seems preposterous to think that ELOPe could social engineer you to leave town. And how is ELOPe going to get your access rights removed? If all it can do is send emails, you can’t send an email to revoke someone’s access. I assume you guys have some kind of internal web application that handles access control. I think you’ve somehow become paranoid about David’s deceit being discovered, and now your imagination is running away with you.”

“
No, Christine,” Mike said. “I’ve thought about this for days now, and it is possible. Let’s say ELOPe didn’t want to be turned off. It knows that I can turn it off. Now it has to figure out how to ensure I won’t do it. If it analyzes enough emails, it could figure out that people don’t do work when they leave town. If it can figure that out, then it can also determine that people leave town for family medical emergencies. If it had enough emails about medical emergencies, it could figure out that messages about family emergencies usually come from family members. My own email history would show who my parents are, and their email addresses, and that I’ve flown to visit them before. If it put all those things together in one long chain of deductions, it could figure out to fabricate an email from my mother saying that my father is sick. I know it sounds farfetched, but this is all within the design parameters of ELOPe.”

“
Are you saying that this thing is reasoning and thinking like a human being?” Christine asked, shaking her head. “Because no matter how smart you guys are, I’m having a hard time believing that some code you wrote is suddenly developing a mind of its own.”

“
It’s not thinking,” David said. “ELOPe is just analyzing emails, figuring out what language will optimize the success of the primary goal I entered, which was to maximize success of the ELOPe project. It’s a straightforward process; goal, analysis, language optimization, in response to inputs. It can chain goals together. It is not independent thought, but it can have the appearance of independent thought.”

Mike raised his hand up. “Look, here’s an analogy I thought of while I was waiting for your plane. Imagine that you’ve got all the pieces of all the jigsaw puzzles in the world. Now imagine you have a computer that is patient enough to try every possible combination of every possible puzzle. Given enough time, it could make any arbitrary picture it wanted out of those pieces. And that’s what emails are to ELOPe—puzzle pieces. It looks at the millions of emails in its library of emails, figures out all the components of them, and then figures out new ways to piece them together.”

“
ELOPe, the computer system that ran away with itself.” David laughed nervously. “Well, we got the name right.”

“
So is it an artificial intelligence? Is it thinking for itself, or isn’t it?” Christine spoke softly, half to herself.

“
I don’t know, hon,” David said. “I don’t see how it could be capable of free-form thinking, which is what most people would think of as an AI. But it is pretty sophisticated when it comes to goal analysis and synthesis. We couldn’t hardwire goals into ELOPe and have it meet the design objectives. We had to let it discover people’s goals. So we gave it the ability to contextually determine goals by parsing emails.”

“
Then it tries to make sense of those goals in terms of other goals it understands,” Mike added. “We implemented two approaches to learning about new goals. First, it can see whether goals might be similar based on language analysis. For example, a ‘break from work’ is semantically similar to a ‘vacation’, and a simple dictionary lookup can figure that out. Second, it can guess where one goal might be an extension of another. If it thinks one goal might be an extension of another, it will predict what people’s responses will be, and then test to see if the predictions match actual historical responses. For example, if I simply said I wanted to have fun today, then ELOPe might be able to extrapolate that activities such as playing a game, miniature golf, or going to see a band are fun.”

David nodded. “So when I added code to create an overriding goal to maximize the success of the ELOPe program, it’s hard to know what it might consider. The more emails it analyzes, the broader the definition of ‘success’ it might have. Up until the last couple of weeks, it had never had such a large base of emails to analyze, nor such a large number of servers to do the analysis on.”

“
From what you’re saying, the more emails it analyzes, not only do the possibilities for what constitutes success get broader, but the system would also discover more methods to accomplish those goals,” Christine said. “What it really sounds like you’ve built is an expert system for social engineering. You know what I mean by social engineering?”

Mike nodded his head yes, but David had a puzzled look on his face, and shook his head.

“
Social engineering is the name given to techniques for tricking people into giving you information or making changes to information systems,” Christine said. “Social engineering was popularized by hackers in the nineteen eighties. And by hackers, I don’t mean the good guy hackers like Richard Stallman. I’m thinking of folks like the Kevins.”

Mike nodded his head again, but David looked even more puzzled, and turned around to look at his wife.

“
Honey, how can you be married to me, and not know this stuff? You know I was a total online geek as a kid, yes?”

“
What can I say?” David sighed. “Please go ahead.”

“
Okay, look. The eighties and nineties were the heyday of hacking. Folks like Kevin Mitnick and Kevin Poulsen were able to get access to all kinds of computer systems, phone company records, credit card company records. I think it was Kevin Poulsen who said that it was easier to trick someone into giving you a password than to brute force hack it. The classic example would be someone who was trying to get access to a company’s internal phone system. She might call the front desk of the company, and tell them, ‘Hi, I’m your AT&T rep. I’m stuck on a pole down the street troubleshooting your system. I need you to punch a few buttons on your end.’”

“
And?” David asked.

“
And the buttons the hacker would ask the operator to press might be a key sequence that would forward all incoming calls to an outside line. Then the hacker could impersonate an employee of the company from their home phone, so they could do even more social engineering. The point is, simply by knowing the lingo, giving plausible reasons, knowing what motivates people, a hacker can gain information or get people to do things by cleverly manipulating the human tendency to trust other humans. Since you’ve built a system that learns lingo, language nuances, and motivations, and can evaluate what will be most effective to the receiver, it is, by definition, an expert system for social engineering.”