The time that Sales braved the white hot heat of the data centre to save the day

On Call Welcome to On Call, The Register's regular foray into the increasingly unreliable memories of those who have to pick up the phone when everything is on fire.

Today's hot and steamy tale comes from "Chandan", who was keen to make sure we knew he worked within the hallowed halls of "Sales", where we assume expense accounts are bountiful and hair coiffed just so.

Unlike many of his ilk, Chandan was not averse to getting his hands dirty: "I'm pretty technical," he told us, "and have been known to spin up the odd storage array."

Chandan's story takes us back to a snowy UK winter a decade or so ago. He had just arrived home after a hair-raising drive back from his place of employment, "a small-ish reseller," when the phone rang.

It was the service desk manager: "I don't suppose you fancy going into the office do you?"

With the roads now treacherous, and a family evening beckoning, Chandan was unsurprisingly a tad reluctant. There was a long, long pause on the other end of the phone before the manager said "There's an issue in the data centre and our on-call engineer is three hours away".

Chandan explained: "These were the early days of outsourcing to "someone else's data centre" before the hyperscale vendors took over."

"Every reseller and his dog was selling rack space in their own facility, in our case a glassed-off room in one corner of the office containing about 10 racks..."

Like so many, the company had also opted to throw a belated Christmas do for employees: "pretty much everyone was there but me," recalled Chandan. Naturally, he tried the gambit: "I work in sales," which should have shut down the conversation, but no, the phone rang again: "We really need you to go into the office."

It transpired that worried remote workers were watching the temperatures of hardware in the data centre approach melting point. "There had been a power outage," explained Chandan. Not to worry though: "The servers had all been protected by the room UPS."

Hurrah for UPS!

"But someone had forgotten to set the air conditioning units to restart after the power came back online."

Down with infrastructure muppetry!

Chandan ventured out into the freezing conditions and headed to the office before the energy-inefficient kit could start its journey through the Earth's crust.

"I arrived to find the temperature in the 'data centre' significantly north of 50 degrees, with the roar of fans almost deafening," he remembered. "I crawled in on my hands and knees and flicked the aircon switch, which instantly clicked off again."

The temperature had reached the point where the in-room air conditioner simply threw up its hands in despair and refused to turn on.

After a fruitless few more flicks of the switch, "I realised I needed a new plan."

"I propped open the doors of the data centre, turned the office aircon to max and opened all of the doors through the building to allow enough freezing air to enter from the outside world to normalise things a little."

Hurrah for an unseasonably cold winter!

"Eventually," Chandan said, "the data centre aircon stayed on and disaster was averted."

And his reward for risking, life, limb and company car to save the company's data centre from meltdown?

"'I owe you a beer' from the service desk manager," recalled Chandan... "which never materialised."

Ever been foolish enough to let it be known you "know a thing or two about tech" when you would really rather be wining and dining customers? Or been forced to unbutton the top button of a cardigan as the data centre heated up? Of course you have, and you should email On Call to share the pain. ®