The I.T. Girl (5 page)

‘Fuck!’ Boris shouted from his cube. Heads popped up like
meerkats
and like a cub mother I immediately checked my feed.

‘Shit,’ I whispered as I blinked at my screens and saw what Boris
was swearing about. My feed was updating prices with a visible delay. My fingers
became moist as I scanned the log file looking for clues to the problem. My testing
showed no performance issues. The only difference now was the code was on a live
server rather than in my test environment.

Number of packets being processed: 70,829

Number of packets missing: 10

My stomach lurched at the number of missing packets. That meant
we could be missing some orders or trades.

‘
Orla
– what the
fuck?’
Boris leaned over my cube.

‘I don’t know,’ I whispered at the screen.

‘Felix is going to have your arse. What is it?
A network issue?
A software issue?
Did
the Exchange say there might be any performance issues this morning?’

‘No.’

‘You did follow test procedure, didn’t you?’

‘Yes of course, Boris,’ I snapped.

‘Okay, fine. First thing to do is check your log files. The old
log files will tell you what you need to know.’

‘We’re missing packets.’ I pointed at the screen.

Boris leaned in. ‘There’s the answer,’ he laughed with relief.
‘Get Gary on the line. Tell him his network is dropping packets of our data,’ he
insisted with a pointed finger.

I got straight on to Networks giving them the details and asking
for an analysis of the network route. Then I shot an email to the Exchange asking
them to check their line. Perhaps there was a delay on their side.

Complaints from the trading floor began to pop up on my screen.

THIS NEEDS TO BE FIXED NOW!!!

HOW HARD IS IT TO GET A DATA FEED TO WORK PROPERLY????

I tried to answer the complaints and update the bug report, offering
a dash of possibilities while taking a closer look at the log file, checking for
error messages or any pattern correlating with the missing data. There was none.

‘Talk to me!’ Boris was back.

‘Networks are on the case. In the meantime, look at this,’ I
showed him the figures.

Memory 40%

CPU 10%

 
‘These are statistics
on the live server.’ I explained. ‘That memory usage looks too high. You wouldn’t
expect more than about fifteen per cent.’

‘Does that mean there could be a memory leak or something?’

‘Possibly.’
I could feel panic rising.

 
‘Can you recreate this
in your test environment?’

‘I’ll try it now.’

‘Keep me informed.’

I started my feed in my test environment and fed it sample market
data. Then I ran performance analysis, hoping it would detect a memory leak or any
sign of corrupted memory. But the results came back normal.

‘Can I help?’ Sam came round behind me.

‘I’m running performance analysis but nothing’s come up as a
red flag. Without a large amount of data to run the test with I can’t reproduce
the same memory behaviour.’

‘What are you seeing on the live server?’

‘Memory is high for some reason.’ I stared at the figures, leaning
into the screen. I could feel the heat from the monitor on my face. ‘But the number
of packets being processed is reasonable. The server shouldn’t be under any strain.
I don’t understand.’

‘Some kind of build-up then.’

He had a point. ‘Maybe there’s a non-virtual destructor somewhere.’

Another email popped up on my screen.

Felix Stern to
Orla
Hanlon

See me now

I jumped back. ‘Is he going to give me a hard time?’

‘He’s not going to give you flowers and a trophy. He’s just got
to make sure everything’s being done to fix the problem.’

‘Bloody hell,’ I said standing up.

‘Give me a shout later if you need me.’

‘Thanks, Sam.’ I tapped his arm as we both left my cube.

I kept my eyes down and fiddled with my hair as I walked around
to the meeting rooms. By now, everyone knew METX was having problems and everyone
could guess I was on my way to talk to Felix Stern. I had never actually met Felix.
My only impression of him was from the bowling competition. I had just joined the
company and entered for the fun. I came out of the match in an early round after
one embarrassing throw where the ball went right off the green. I then spent the
rest of the afternoon wandering around the other matches, trying to remember people’s
names. I’d got an ice cream for the final and accidentally stood next to Sam. I
remember we hadn’t spoken yet. He didn’t usually join in the social events and he
was always so serious in work, as if he wanted the distance.

‘Sam, how’d you do?’ I finally broke the ice.

‘Out in the second round,’ he said evenly, with his eyes on the
match. His face was flushed from the unexpected heat and his hair, usually neat,
was damp with dark bits sticking to the side of his face.
‘Didn’t
do as badly as you though.’

‘Sounds like we did the same.’
I looked
back at the lawn.

The match ended and the winner raised his arms to punch the air
with victory. The crowd made space for him as he began a lap. He made strong-man
poses along the way, red-faced and panting. Muscles bulged through his tee-shirt.

‘What an idiot,’ I said. ‘Who
is
that guy?’

‘That’s Felix Stern.’ Sam laughed. ‘He’s your manager.’

I knocked on Felix’s door and went in. ‘Hi, Felix.’ I sat in
the easy chair opposite his desk.

‘Hi.’ Felix leaned forward. His head was small compared to the
rest of him and he had a youthful face aged only by a worried frown, like a knot
over his eyes.

‘So why isn’t METX working? Let’s make this quick.’

I took a deep breath. ‘Most of these
type
of delays are network issues. They’re taking a look and in the meantime I’m running
tests on my code.’

‘Look at this list of complaints.’ He dropped a collection of
printouts on the desk. ‘What are you doing to make this list go to zero?’

I flicked through the pages. I already knew about the complaints.

‘I’m trying to have the issue investigated from all angles,’
I said clutching the paper. ‘The server seems to be under a strain but I’m not sure
if-’

‘Status,’ Felix read from my online update. ‘Server under strain...
Not sure if this is usual... Delays probably due to network problem... Alternatively
a resource overload due to volume could be causing the delays... Analysis of difference
between test and production environments underway to assess significance on above-mentioned...
’ His eyes flicked up and down the screen as if he was searching for more information.

I sat forward to explain. ‘A delay can be caused by a number
of issues. It might be a memory leak eating up resources. It might be the quality
of the physical line. Or it might just be the feed’s volume is too large for this
particular server.’

‘Why was this not caught during test?’

‘The test server is different to the live server.
So that’s why this could be a server issue.
Also, METX didn’t
provide us with live data for our testing so we had to make do with sample data.
So this could be a problem with the live data that simply didn’t exist before.’

‘Sample data? Why wasn’t the feed stopped from going live?’

‘Sorry?’ I stared at Felix, my brain racing. ‘I don’t understand.’

‘The Exchange refused to provide proper data for testing. What
did you or Exchange group do about it?’

‘I don’t think they did anything.’

‘You
don’t think
they
did anything? Is that a joke?’

‘No. I just didn’t talk to them about this specifically.’

‘What? What are you telling me now?’ His voice filled the room.
‘Your project had an inadequate test environment and you didn’t tell anyone about
it?’

‘It was properly tested with all the available data,’ I pleaded.
‘A lot of feeds don’t have a proper test environment.’ I thought of other projects
I knew in the same situation. ‘The business analysts used to handle the test side
of things,’ I added quietly.

‘But that’s
your
job
now. To push for what
your
feed needs.’ He poked the desk
with a finger.

I glanced around the room, looking for answers. There was a large
window behind Felix. Light came through the half-open blinds in slices. Neat technical
folders decorated the walls on either side of the room and on the floor in the corner
a gym bag lay open with two metallic weights sticking out. My eyes went to the high-energy
muesli bars on his desk. Next to them was a golf caddie pencil holder. It was empty
but it had a photo leaning against it that I could only see from behind. Felix took
a heavy breath and a logo expanded across his shirt.
‘
CPR - The Kick of Life
’.

‘What about analysis of the server before the feed was uploaded?
Did you do anything about that?’

‘I had no reason to think there’d be a problem there,’ I admitted.
‘The live environment is allocated by Feed Support.’

‘But you’re supposed to check these things.’ He poked the desk
again. ‘What have networks said? Could it be a bandwidth problem?
A hardware problem?’

‘I don’t know,’ I squirmed in my seat. ‘They’re analysing the
router configuration at our end and I expect they’ll also do a data trace to see
if there’s a delay at a certain point. But it takes a while.’

‘You’ve got to keep on them.’

‘I will.’

‘Keep on their ass. What about the Exchange? Have you spoken
to them?’

‘I sent them an email to check on their side.’

‘Ring them.’ He gave me a hard stare. ‘You’d better come up with
a solution real quick. This is an important feed and it was given to you.’

‘I understand. I’ll keep you informed,’ I said.

He went back to his keyboard and I took that as my cue to leave.

An email was waiting for me from the Exchange when I got back
to my desk. There was no obvious problem on their side. They hadn’t received any
other complaints this morning.

My phone rang. ‘Come in come in,’ I breathed, cradling the receiver.

‘You in the trenches, soldier?’ Gary asked.

‘Front line, Gary.
What’s the word?’

‘Right,’ he sighed. I could imagine him flicking through the
crumpled pages of his notebook. ‘I’ve been monitoring the data coming in from our
router, yeah? From 8 a.m. London time onwards. I’m not seeing any delays.’

‘What about from the Exchange to their routers?’

‘It’s possible, but the overall turnaround is ninety milliseconds.
We don’t expect faster than that.’

‘So, the packets from the Exchange are coming straight in to
our live servers, consistent, no break?’

‘Yep.’

‘What about the dropped packets?’

‘The line isn’t dropping packets. Your feed is just rejecting
them.’

‘Oh.’

‘The thing is, with the type of connection you have, if your
feed doesn’t lift packets off the line quickly enough, it will just skip them, to
alleviate congestion. That’s what I’m seeing here – Packet skipped due to congestion.
I’m seeing another one now.’

I swallowed hard. ‘So it looks like we can assume there’s no
connectivity problem?’

‘That’s right.’

‘What about server performance?’

‘Between... what are they... your test server is T-A-4 and the
METX live server is L-E-35. All libraries are the same. You’ve got all the same
thread-handling.
Same scheduling and architecture.
There’s
nothing at all to suggest a performance difference.’

‘Thanks a mill’ for checking, Gary.’

‘No worries, soldier.’

‘Over and out,’ I replied in our usual goodbye.

‘Fuck.’ I banged down the phone. With options closing in around
me it was looking like an obscure mistake in my own code was the most likely blame.
I ran the performance tests again, this time replacing the sample data with real
data captured from the live feed. It was difficult to work with the larger volume
but it meant the tests would be more accurate.

Every negative result felt like a little reprieve but it was
replaced with a fresh batch of nerves; I still had no idea what the problem was.
Desperate, I began taking individual modules of code and testing them as isolated
cases. I had to find something that was causing memory to bloat and slow down the
feed. I noticed people coming back from lunch and realised no one had invited me.
This was the first time I had been involved in a live problem. It felt like a landslide.

Complaints and queries about progress were still popping up on
my screen but I was running out of answers. Ignoring them I began trawling over
my code. I mentally stepped through each algorithm calculating what it was doing
and what could possibly go wrong.

Boris was back at my cube. ‘We’ve got to get this fixed before
New York opens and we’re flooded with US activity.’

‘New York opens in half an hour Boris. We’re not going to solve
the problem, test it and roll it out in that time.’

‘Well, do you have any move on? Any update?’

‘It looks like my feed has some major memory issue slowing everything
down.’ I dropped my head into my hands. ‘But my performance analysis isn’t showing
any memory leaks.’

‘Let’s have a look,’ Boris said pulling up a chair.

I showed him my test results, explaining what each module was
doing and how the results showed normal behaviour. We stepped through the large
algorithms together, just in case I had missed something.

‘How high is the memory usage now?’ Boris asked, out of ideas.

I ran the command to check memory on the live server.

‘It’s down to five per cent.’ I narrowed my eyes.