How I Transported a Robot, Nearly Went Gray, and Bled All Over the Server Room

This is the story of one of the most memorable incidents in my engineering career. For obvious reasons, I've changed the names, locations, and some identifying details so that the client and other participants cannot be identified.

This is what a tape library (ours was smaller) and a library robot (ours looked just like this) look like. The guy in the photo is not included.

Part 1. The Box

I remember it was the last days of November. Already thinking about the end of the workday, I was planning my evening when I was suddenly informed that in the fine Siberian city of N, our client's tape library had broken down. A replacement part had been shipped immediately via a freight company, but three days later it was still in transit. The freight company gave vague explanations and hemmed and hawed on the phone, while the client was getting seriously nervous. The forecast was uncertain, so it was decided to deliver another spare part ourselves, by plane. A warehouse clerk handed me a bulky box weighing about ten kilograms, covered in barcodes and stickers, and cheerfully slapped me on the shoulder with the words: "Whatever you do, don't check it as luggage — they'll crush it."

The box certainly worried me, but I was no less worried about the bag of powdered milk that the project manager had thrust into my hands at the last moment. "They're having some kind of dairy issues there right now... because of bad weather or something... the local guys asked us to bring them 2 kg. It's no trouble for you, right?" he said. From the expression on his face and the characteristic gesture of his palm, as if covering my mouth, it was clear how much he wanted me to be agreeable, or even mute.

An Important Digression

This story happened several years ago, when I was an engineer and flew around the country constantly. Sometimes I'd wake up in yet another hotel and couldn't figure out which city I was in. Four to five business trips a month left their mark on my lifestyle. And the embodiment of that lifestyle was my backpack, always ready for the next trip.

Part 2. The Librarian

I climbed out of the taxi, and unexpectedly a police officer stopped me outside the airport terminal. "Your documents!" he barked, and a feeling of deep satisfaction spread across his face for some reason. A nasty, biting wind blew, making it hard for me to sort through my pockets and bags. I managed to fish out my documents and handed them to the officer. In theory, a young man traveling with nothing but a laptop, a large box, and a bag of white powder shouldn't raise any suspicion, so I tried to look genuinely confused. "What's in the box?" he asked indifferently. Apparently, his professional instinct had already determined that the bag contained nothing other than powdered milk, so the bag no longer interested him. The box, however, still eluded his powers of deduction. "It's a robot," I said.

At that moment, I had the sensation that his indifferent face had been swapped out, because it lit up from within with genuine curiosity, as though somewhere in the depths of his cap with its badge, an additional module had connected with the function of human curiosity. I realized I had been too hasty with my answer and needed to somehow fix the situation quickly. "It's a robot... for a library," I added, even less confidently, continuing to observe the activation of the remaining modules of sarcasm, gloating, and greed. "A robot for a library, you say... A robot librarian, is it?"

Part 3. Security Screening

I was making my way to the boarding area, replaying how I'd just dealt with the badge-capped officer. "Got through it," I thought, "but I could have missed my plane."

"Bags on the belt, remove jackets, belts, and shoes, take out your phones..." a woman in uniform commanded authoritatively and soullessly. Having loaded my belongings onto the belt, I passed through the metal detector without a peep. It was obvious that the box with the robot was bothering the security screener, who kept sending it back and forth through the X-ray machine. "And what is this?" she finally managed, when all the comparative images in her head had been exhausted. I was clutching the bag of milk, thinking through my strategy to avoid a repeat of the badge-cap episode. But suddenly her colleague sitting next to her gave the monitor a quick glance and said: "Oh, that's just a robot for an L200 library. Third one this week alone — they're having a season of breakdowns. Go on through, young man!" My astonishment was boundless, but at the same time I was enormously grateful to the providence that smiled upon me, as well as to the technical literacy of my fellow citizens working at airports.

Part 4. The Estate

A rusty Volga screeched its way from the airport to the hotel. Out of about thirty dubious-looking cab drivers waiting for the night flight outside the local airport, this one seemed the most restrained. In his entire hunched, smoke-cured appearance there was a certain decency and even nobility. However, a cab driver's appearance is often deceiving, and he spent the entire ride puzzled as to why I lived in Moscow but still didn't know how Alla Borisovna Pugacheva was doing. "You live in the same city as her, and you don't know?!" my driver wouldn't let it go. He disliked me for my ignorance but at the same time didn't want to embarrass himself in front of a visitor from the capital. We pulled up to the hotel doors in style, with screeching brakes and a sharp pirouette that should have been illegal for a vehicle of such venerable age. The car door seemed to open by itself, or perhaps just fell off. Upon hearing my question, "Could I get a receipt?" the driver waved his hands and forgot how to speak entirely.

I walked inside, looking around at the tastelessly decorated walls and columns, all with some kind of gray coating, like dust. The building was of old construction, possibly a former estate. At the reception desk, a girl was indifferently painting her nails. Confident in my rights as a customer, I decisively took out my passport and slapped it on the counter. The madam reluctantly tore herself away from her nails, cast a contemptuous glance at my passport, and muttered: "We don't extend credit." "Excuse me...?" Interjections and other signs of extreme indignation raced through my mind. "I have a room reserved here," I tried to remain polite. "This is a casino! The hotel entrance is on the other side of the courtyard." Who would have known that both establishments had the same name? I also managed to think that the whole time the taxi driver had been driving me here, he probably assumed I'd flown in from Moscow to this unwelcoming winter land to gamble away the contents of my box at the card tables and roulette.

Part 5. To Sleep or Not to Sleep?

The hotel room greeted me with magnificent floor-length curtains and the musty smell of dampness. I sat on the edge of the bed, which was set into a wall niche, gently placed the precious box with the robot in front of me, and unexpectedly said hello to a spider in the corner. Work at the client's site was scheduled for 9 AM; it was 6 AM local time. My head was still buzzing with the engines of the Il-62 and other delights of the hours-long flight. I mentally weighed the prospects of sleeping for a couple of hours and waking up refreshed, and made the wise decision to simply not go to bed at all. Physiologically, I was leaning toward a late dinner — or even a midnight glass of kefir — so I straightened the company's green tape on the box and headed for the shower.

Part 6. Semyon

When I arrived at the client's office at 9 AM with the robot and the powdered milk, I was met at the security desk by an IT department representative who said his name was allegedly Semyon. Semyon talked nonstop about everything around him, periodically wiping his mouth with his hand. From his appearance and simple manners, it was clear that in the department, the role of greeter fell to him more often than others, as the least technically skilled specialist but a very sociable person. Without a doubt, Semyon inspired trust in people, and, as it later turned out, possessed a magnificent, almost irrepressible appetite.

The floor was bustling with activity and smelled of paint. Gesticulating wildly, Semyon explained that this was a new building and they were currently setting up a second site. And then my mobile phone rang. It was the project manager. "Have you made it to the client yet? Listen, help them out there. We're setting up a cluster for them right now. Our Alyosha has been there for a week already. Something's not working. It's no trouble for you, right?"

Part 7. The Cluster

Accompanied by Semyon, I entered the machine room. Boxes of various sizes bearing the names Dell, Cisco, and EMC were scattered everywhere. Many had already been opened, and cables, rack-mounting rails, piles of documentation, and electricians lay all around. The electricians weren't just lying around — they were rummaging with wires under the raised floor and quietly bickering. In the center of this creative chaos stood the senior department specialist, Nikolai. He was holding a staff meeting. Hiding from the meeting behind a pile of boxes, our Alexei was squinting at his laptop screen and secretly having a bread roll for breakfast. Semyon also wanted a roll, but he had to attend the meeting, so he just huffed.

From Nikolai's first words, I understood that time to launch the system was running critically short — less than 24 hours. With the appearance of the second site, management had tasked the team with reorganizing one of the most important information systems. The database — the core of this information system — had to become clustered. It would now be stored on two storage systems with synchronous data replication between them. Cluster software would monitor the entire system's state and could automatically respond to a failure should anything go wrong at the primary site. The preparatory work was nearly finished, but a lot of time had been lost finding and fixing bad fiber optic links between the sites, which had been laid by a subcontractor. There was still the matter of finishing the server installation, updating the operating systems and firmware, and testing everything. The production launch was scheduled for that night — the only time when critical financial systems could be taken down.

Part 8. From Sunrise to Sunset

I barely remember how that day went. I only remember that my colleagues immediately told me the tape library would have to wait given the circumstances, and right now my hands were needed elsewhere.

All day we mounted servers, tightened bolts, ran cables, and connected adapters. During breaks, we apparently even had lunch and went for a walk in the forest. Then I installed operating systems, applied patches, and wrestled with firmware. Sometimes the firmware wrestled back, and to stay awake I recalled how the flight attendant on the plane had masterfully juggled a teapot, lamenting the turbulence, as she served tea to passengers wary of boiling water. Night was falling; through the window, a large orange moon had risen above the forest.

Part 9. Zero Hour

"Zero hour has arrived," Nikolai said briskly, yawning, after a brief conference call on the local phone. "It's arrived, it's arrived," the voice on the other end repeated from the receiver he hadn't yet put down, followed by giggling. These were the application specialists, who were also on duty that night and would need to verify everything was working once we finished.

The clock showed midnight. We had reached the most critical moment: we needed to switch data replication to the other storage system, wait for synchronization to complete, and bring up the cluster at the second site. Alexei was in full command of this process — a specialist in storage systems well known in our circles. Some people also knew about his irrational habits... Smacking his lips, he copied volume identifiers into a script, cross-referencing them against the list the client had provided. Half an hour later, when this painstaking work was done, Alexei touched the Enter key to make sure it was real, then pressed it. Everyone tensely waited for something, and the collective silence was broken only by the crackling of fluorescent lights. I was sinking toward the raised floor, seeing dreams while wide awake, though I don't remember what they were about.

Part 10. Untitled

I came to because synchronization had finished, but for some reason everyone was unhappy. The cluster services wouldn't start. I was summoned from dreamland to look at the logs. Through the fog in my consciousness, it began to dawn on me that the problem was with the disk subsystem. A logical volume group was importing with errors and the data was inaccessible. The causes were not yet clear, but the prospect of rolling back to the old configuration — Plan B — was beginning to take shape. The system had to be operational by 7 AM, so there was no time for deliberation. At that point, I couldn't even begin to imagine that the tragic events of that night were only just beginning to unfold.

I meditated for another five minutes and proposed that we roll back to the old system. Reluctantly, my colleagues agreed, glumly contemplating the morning report to management. We switched the replication direction and issued a disk group import command on the old server. I had already calmed down and was dreaming of making it back to the hotel bed with a sense of duty fulfilled. And then I heard the exclamation: "It won't start here either." And everything inside us sank...

Part 11. Akela Has Missed

The fog completely cleared from my head. They say the human body mobilizes in extreme situations. I was thinking furiously about what could be done. On the storage side, everything was fine. Having confirmed this, my colleague Alexei waved his hand in a gesture of helplessness and went to watch a movie on his laptop. I've already mentioned his irrational behavior. But technically, there was nothing to criticize him for.

The cause of the problems was becoming quite clear. Additional volumes had previously been added to the disk group by the client, but they had forgotten to add them to the list of disks being replicated. When the cluster started, the disk group was imported, but metadata had only been updated on some of the disks, while the new ones remained in read-only mode. Now the disk group configuration was completely inconsistent, and the data was inaccessible from either the new or the old storage system.

When you're dealing with financial data of a massive organization, losing it is tantamount to catastrophe. I immediately recalled a similar story from early in my engineering career when I was doing a repair job at a client's site — a serious organization that had gone legit in the '90s. They were discussing how their admin had accidentally deleted something. "And what happened to him?" I asked casually. "Well, he's been sitting down in our basement for three days now. Waiting for the CEO to decide what to do with him," the shaved-headed employee with a heavy chain on his bull neck answered in complete seriousness.

Part 12. Hindi

Have you ever felt the hair on your head stand on end, preparing to go gray? As I estimated that we had lost the database, I didn't even want to think about what would happen in the morning.

In situations like ours, the proper thing to do is immediately call the manufacturer's tech support. At this hour, naturally, it was the English-speaking line. I roughly sketched out the key words for myself and dialed the number. Two rings and an English baritone answered. I immediately laid out who we were and what had happened. The baritone patiently listened to my monologue and asked: "What is first name?" I was taken aback but, gathering my thoughts, answered. Then came questions about last name, customer name, city, location, serial number, and so on ad infinitum. Time was mercilessly slipping away. Having finished the questionnaire, the baritone asked me to write down the case number and said to wait for a specialist to call back. The specialist didn't call — five minutes later he emailed me. He wrote that he had accepted my ticket, but his shift was over, so I needed to call the dispatcher again and ask them to assign the ticket to another specialist. It's not clear why some people think only Russians don't like to work. It seems this is a misconception.

And then a miracle happened. Another specialist called me back, but he turned out to be from India. The second global technical support center was there, and they had picked up my ticket themselves. I was as happy to hear from him as if he were my own father, but my joy was short-lived: we couldn't understand each other. It was clear he was speaking English of some kind, but that didn't make it any easier. Sometimes certain sounds in his speech gave me a distant idea of what he might mean. But ultimately, our communication devolved into a written exchange. I strained my memory, recalling the finest moments of Indian cinema, trying to make him feel the depth of our grief. Time passed. India, as expected, sang and wept, but couldn't help us either.

Part 13. Execution at Dawn

At 4 AM, we realized there were no other options left but to restore from backup. All eyes turned to the tape library. Nikolai was even crawling in front of it, wiping the floor with his tie. "How long will it take to unwind the database from the tape?" I asked the local backup specialist. "About three hours, probably," he answered uncertainly, "but we've never actually tested it." We urgently needed to replace the robot in the library, because three hours was all we had, and barely that. Never before had I opened packaging and tightened screws so fast. The robot, covered in blood, was done — it was in place. I had cut my finger in the rush and it was my blood, but there was no time for bandaging. The switch flipped and the library lit up from within, the robot came alive and began calibration. We crowded around the library and peered through the little window, mentally willing it to move and scan the barcodes on the cartridges. It did this slowly, savoring the process, while we pressed our faces against the glass and ground our teeth. "Initialized!" said the backup specialist, referring to the library. And added: "Starting the restore." The robot paused for a second, then darted briskly to the right row of cartridges, confidently grabbed the right one, spun around the guide axis, and headed toward the drive.

Meanwhile, Alexei was in the corner behind the boxes, engrossed in his movie and another bread roll. The movie was playing without sound, but the emotion on my hapless colleague's face was genuine. Engineers often carry a collection of favorite movies on long business trips — movies that are great even without sound. For true art, like Charlie Chaplin films, can be watched that way.

Suddenly, from inside the library came the sound of a dull thud. The robot had carried the cartridge to the drive but couldn't locate the slot, and began banging against its wall, trying to insert the cartridge somewhere. It was a complete fiasco, and despair washed over us. Unfortunately, mechanical parts are extremely sensitive to transportation, so there are cases where a new part is immediately classified as DOA (dead on arrival).

Part 14. Hope

Cold sweat was pouring down my forehead, blurring my vision. I slowly pressed the keys, double-checking each one. We had less than an hour before the start of the business day, and all our attempts seemed futile, when suddenly an idea hit me. "When was the last time you pulled diagnostics from the old server?" I asked my colleagues. "Yesterday evening, why?" answered the good-natured, hungry Semyon. "There might be output from a command that shows the disk group configuration before the changes, and from that output we could try to manually recreate it."

And now I was doing exactly that — manually running commands that re-partitioned the disks into a service area and a data area. "Have you ever done this before?" asked Nikolai, emerging from somewhere. "No, I only read an article about it. It's black magic, but we have to try," I answered. Under any other circumstances, they would have cut my hands off if they knew I had just destroyed the entire disk group. But right now, this was our only chance, so there were no objections. All of us in that room were simultaneously the condemned and the conspirators. Hope never dies in a person.

I finished the partitioning and typed the command that, according to the documentation, should recreate the disk group without initializing the space — that is, without wiping the existing data. "Act without fear; if you fear, don't act," I recalled Confucius, and executed the command. To my surprise and joy, it completed without a single error. I displayed the configuration — all logical volumes were in their places, with normal status. All that was left was to verify the data was there. We had 15 minutes to spare. A phone in the room had been ringing for about 40 minutes, but no one was paying attention. Whoever it was, we had nothing to tell them yet. "Let's start the cluster," I proposed, and was surprised by my own voice.

In the library, the robot was furiously bashing against the drive, apparently hoping to wear it down sooner or later. Alexei was stroking his laptop, moved by the film's ending. Semyon was patiently bored, pondering whether he had chosen the right profession, and between thoughts, dreaming of a chocolate bar. Someone else was snoring behind the racks, occasionally twitching in their sleep. The remaining cast of characters stared intently at the monitor, which, without exaggeration, reflected our entire lives at that moment. "It's up!" Nikolai exhaled and grabbed the ringing phone. It was the application specialists — they were already preparing the noose and calling to say goodbye. "Check the application!" Nikolai barked and brushed off his tie.

Part 15. The Box

I was sitting on the box and smiling happily, imagining how right now from all the apartment buildings in this beautiful city, little streams of people in heavy coats, fur hats, and felt boots were trickling out. They were lining up at ATMs, and those ATMs were serving them just as happily and reliably. And in this whole picture there was some kind of simple, proper human happiness.

Everything was working, and with 2 minutes to spare before the start of the business day, the system was up. My colleagues silently thanked me, shaking my hand and shaking their heads. Everything that had happened that night, we intended to take with us and never tell anyone.

The Indian support engineer was still writing to me, still suggesting various wild ideas. At that moment, the door to the machine room opened and someone's sleepy face poked through. It was the security guard from the front desk. He said: "There's a courier here. Says he brought some kind of robot. A robot librarian, I think..."

A beautiful new workday was beginning.