Authors: Fiona Pearse
‘What? That’s good, right?’
I ran the command again ‘Back up to forty-five per cent. Not
good.’ I licked panic off my lips and opened the log file looking for error messages.
‘Another skipped packet,’ I read. ‘Oh.’ I jumped and slapped my forehead. ‘The memory’s
not too high, Boris. Every time there’s a dropped packet, there’s a fluctuation
in memory.’ I felt sick thinking of the time I had wasted.
‘But why?’
Boris pressed.
‘Because METX requested we process the packets of data in sequence...
So, when there’s a missing packet, the feed halts processing and shoots a message
off to the Exchange server looking for it. Then the feed just sits and waits for
it to arrive. But in the meantime, newer packets are still arriving and they have
to be held on a queue, each one taking up memory.’ I shrugged helplessly. ‘Then
the missing packet arrives, all the packets get processed and all the memory is
released.’
‘Then why
is your feed skipping packets
in the first place? That’s the real question.’
‘No.’ I said firmly. ‘The question is correct. Why is my feed
not performing fast enough? But, it’s got nothing to do with memory. The memory
fluctuation is a symptom not the cause. I’ve been chasing a red herring.’
A bell tolled and the US market opened. I stared through my fingers
at the terminal imagining the New York trading beginning.
Phones
starting to jump.
Orders and trades entering the system.
And then while I was still holding my breath, the delays got worse.
‘Oh, fuck,’ I said through my hands.
‘Fuck, fuck,’ Boris echoed next to me. ‘
Orla
– what are you doing to me?’
A new warning flashed on the screen.
Tradeq
currently experiencing delays
‘Whoa, what’s going on there?’ Boris grabbed the keyboard and
punched in the
Tradeq
keys.
We watched the prices updating with a drag.
‘Boris,’ I braced myself. ‘I think
Tradeq
and METX are running on the same server.’
‘
Orla
.
Are
you telling me your feed is actually compromising the server it’s running on?’
‘It’s possible.’ I pulled the keyboard back and checked the server
statistics. The memory was high. I checked again.
Still high.
‘It’s not going down.’
‘What’s not going down?’
‘Memory.
It’s staying high. Performance
is going to keep getting worse.’
‘Jesus,
Orla
, you’ve got to find the
bug right now. Felix is going to kill us both.’
A new round of irate messages began popping up obscuring the
slow pricing and, overwhelmed with nerves, I began to giggle.
‘It’s not funny. What are you laughing at?’ Boris threw his hands
at the screen.
‘I’m sorry,’ I choked, watching the flashing messages fighting
each other for space.
Felix was suddenly behind us making me jump.
‘I have traders yelling down the phone threatening to come up
here,’ he said, pointing towards his office. ‘Two feeds are down. What the
fuck
is going on?’
‘I think METX is compromising the server it’s running on,’ I
said, quickly sobering up. ‘I’ll move
Tradeq
to its backup
server.’
‘Do it now,’ Felix said.
With Felix and Boris standing behind me I restarted
Tradeq
on its backup, safely out of the way. The three of us
stared at the screen until the
Tradeq
prices resorted
to normal.
‘Why wasn’t this done before?’ Felix demanded.
‘I was trying to find the problem. I didn’t think the delays
would get any worse.’ I squirmed, knowing the answer wasn’t good enough.
‘I take it METX is a software problem. What about stopping it
and restarting it?’
I hesitated. ‘On start-up it sends a message to the Exchange
server requesting all its market data from the beginning of day. That would flood
the terminals with duplicates from this morning.’
‘Is the start time configurable?’ Boris asked.
‘Yes, 8 a.m. is the start-up time.’ I followed his train of thought.
‘If I just change it to 2 p.m. we’ll only get a small amount of duplicates.’
‘Okay.’ Boris said. ‘Start it on its backup server, wait until
the duplicates are processed and then switch clients to the backup.’
‘Do it now,’ Felix barked.
When I restarted METX on the backup the delays disappeared. When
I was sure it was processing data correctly I re-directed traders to the backup
providing them with timely data. The three of us watched as updates began to appear
on the screen. I checked the performance.
Memory 14%
CPU 12%
‘This looks okay.’ Adrenalin began to drain from my body like
blood. ‘I’ll keep an eye on it. I can restart the feed again if delays build up.’
‘I don’t want any more delays today,’ Felix said.
‘What the hell is going on?’ Boris asked, wiping his forehead
when Felix was gone.
‘I think it’s dropping so many packets that it doesn’t have time
to recover before the next packet is dropped,’ I said limply. ‘So the memory builds
until it starts to use up memory that’s meant for other feeds, slowing down everything
around it.’
‘Shit me,’ Boris said, wiping his brow again.
‘I’ll keep an eye on it, Boris,’ I assured him. ‘At least it’s
isolated now so it can’t affect any other feed.’
‘Yeah, well, once the opening activity in the US is over things
might quieten down. Alright, let me know if you need any help.’
I shook my head at my code. The figures were dancing in front
of my eyes. I turned back to the server stats again and watched the memory usage
rise and fall along with the butterflies in my chest.
Why aren’t you fast enough?
I thought, feeling queasy. I realized I
needed to eat.
In the canteen I felt too sick to eat. I made my way to the juice
bar where groups of brittle girls and sales boys posed like fruit flies. I avoided
eye contact in case my realness was a shock to them and filled a paper cup with
orange and mango juice.
On the way back I stopped in to the Think Tank. The door whispered
closed behind me and suddenly the pace of life was set by the fish traversing the
wall to wall tanks. The only sound was from bubbles rising and the hum of a disillusioned
grad playing with a yo-yo in the corner. Sam was in his usual spot, sitting against
a giant
lego
block, leaning over a notepad on his knee.
He was one of the few people who used the room for its actual purpose – somewhere
to concentrate, away from the noise.
He looked up to see me coming over. ‘So, how are you enjoying
your first live problem?’
I sat next to him and clicked my heels three times. ‘If today
gets any worse I'm going to have an out of body experience.’
‘That’s what happens when you let women write software.’
‘So you’re enjoying this?’
‘Anything come up yet?’ He relaxed his pen on the page.
‘No. Definitely looking like there’s an obscure bug in my code,
but I’ve been staring at it for so long it’s become meaningless. Could we step through
it together?’
‘Sure,’ he nodded.
‘Thanks. So, how’s it going?’ I looked at his notepad.
‘Good.
Should need a review soon.’
He
held across the pages of arrows and squares.
‘Looks complex.’
I reached over to turn
the pages in his hand.
The grad in the corner swore at his trick and tried walking the
dog again.
‘I better go back,’ I said. ‘See you at my desk in five?’
‘Yeah I’ll come round. Want me to get you something?
A coffee?’
‘No, thanks.
I’ve just been to the canteen.’
I stood up with a dramatic sigh and went back out to the floor.
At my desk I opened my files for Sam’s inspection.
‘You know the story so far?’ I asked as he came over.
‘Yeah, I’ve been reading your updates.’ He pulled up the spare
chair and we leaned our heads into the fluorescent screens. The code I had opened
up was spread out over three monitors. It curved from one end of the desk to the
other. Sam squinted, leaning in and I allowed his hand take the mouse from mine.
Trouble-shooting was like going through a maze. Following each
turn and remembering the way back to the starting point if you reached a dead end.
The trick was to block out assumptions, try to forget everything you knew, and look
at things freshly each time.
‘It’s got to be something basic you do for every single packet.’
Sam snatched up the diagram we had drawn together. We had exhausted all the possibilities
and were back at the start again.
‘The only thing I do for every packet is generate an ISIN. I
add one to each Order before it’s sent to Desktop... Let’s look at the ISIN Generator
code.’ I took the mouse, forcing myself into another burst of energy and opened
more files on the screen.
‘It looks fairly clean.’ Sam blinked in the florescent light.
I noticed his eyes were watery. ‘What’s this search?’ he highlighted a line of code.
‘That’s just the search function to retrieve an ISIN. If there
isn’t an appropriate one already in the list, a new one gets generated... Oh, that’s
odd... That should be a
binary
search,
not the standard
find
.’
‘Is it on a map or vector?’
‘A sorted vector.’
We both stopped talking. It was clear what the problem was. I
was using the slowest possible search algorithm trying to find one entry in a huge
list, and it was happening for every single order.
‘No wonder orders aren’t being processed fast
enough,’
I squawked. ‘The list of ISINs can grow up to tens of thousands
of entries throughout the day!’
Sam slumped back in his chair, causing it to moan. ‘At least
it should be a simple fix,’ he said lifelessly. ‘Run your performance tests again
using a binary search. I bet even on a small amount of data you’ll see a difference.’
‘I can’t believe I made such a stupid mistake.’
‘Who did your code review?’
‘Boris.’
‘He wouldn’t know the difference.’
I thought back to my code review. I had left early, flying home
that night for a long weekend. I remember I had emailed Boris some quick notes about
what to look for. ‘I’m an idiot.’
‘Don’t worry about it,’ Sam said. ‘It’s a mistake. It could happen
to anyone.’
I changed my code and re-tested. Sure enough, the speed of the
new search was noticeably faster. I could have kicked myself. The whole time I was
looking at memory.
Looking for complex logic errors.
I
took for granted the most used piece of code was doing what it should. I moved the
fix onto the live server for the next start-up: twelve a.m.; the Asian market.
The rest of the day was a blur of answering complaints about
the crisis and recovering the missed data. As I updated each complaint with the
same explanation, over and over again, I knew this was going to look really bad.
I had made a junior programming error – and worse, I hadn’t caught it.
Sam came over, carrying his coat and rucksack, ready to leave.
‘I hope you learned your lesson, young lady?’
‘That’s harsh. At least the worst is over.’
‘The shit hasn’t hit the fan yet,’ he warned.
Felix was behind us. ‘Is everything in place for tonight?’
‘Yes. I’ve uploaded a fix.’
‘Why didn’t you catch this before it was rolled out?’
‘As I said before, without a high volume of data, the problem
simply didn’t appear. I couldn’t reproduce it in a test environment.’
‘Then how do you know it’s fixed?’
‘Because I can see the processing speed is faster with the fix
in.’
‘Have you done a code review?’ Felix looked at Sam.
‘Yeah, I found the bug with
Orla
. It’s
a simple fix.’
‘Okay,’ Felix said and walked away.
‘Don’t worry about it,’ Sam said again, looking at my crumpling
face.
With the familiar feeling of drained tiredness mixed with hyper
brain activity I slowly made my way off the R&D floor, no longer impressed by
the ceiling-high display of market activity and bold company slogans for
CouperDaye
. All of which were there to remind me; this wasn’t
just a job, this was a way of life.
A tube beneath the city took me home. Where the shiny jagged
edges ended and the old markets began, I disappeared down winding streets towards
my one-bed lux, cut in to an old building.
When I opened the front door I realised I had remembered to get
my broadband line set up for work but had forgotten to get the heating switched
on. The open-plan living room was empty apart from a mound of boxes in the middle
of the floor. I opened a few to find my fleece and a rug. With my back against a
box, I settled on the floor and picked up my laptop to prepare for the 12 a.m. market.
Once everything was in place I went over to the bay window. It
was the first thing I had noticed when I’d walked in to the flat three months before.
I had instantly known that I had to live here. The frames were white and chipped
and they rattled when I shook them. I cupped my face against the glass and squinted
down at the communal garden. Street lights curled like crooked hands over picnic
tables. I felt a little cold air coming in. ‘Now what happens?’ I asked, leaving
my breath on the glass.
Boris leaned against my desk while I tidied around him, preparing
for the weekend.
‘
But that’s your job
,
’
I imitated Felix’s gruff voice.
‘Ooh, that’s harsh,’ he
said, keeping an eye on the end of our wing for anyone from management coming around
the corner. ‘Look, he’s going to get some heat from the US boys over this. You’ll
probably get a bit of a bollocking.’