500 Mile Email

500 Mile Email is a curated list of absurd software bug stories and weekly newsletter. It's maintained by me, Harley Hicks, and is named after one of my favorite stories, "We can't send mail more than 500 miles". Know a great story? Send it my way.

Sign up for the weekly round-up newsletter!

Collateral Damage

A field technician must use all their skills to solve the mystery of a label printer constantly crashing and discovers an unexpected source of interference.

By Ralf Maximus on April 25, 2023

In the 1990s I worked for a large 3-letter shipping company whose brown vehicles & uniforms are recognizable worldwide.

My job was a field technician, in charge of maintaining a wide array of devices from computerized scales, to primitive GPS systems, to package tracking software, to call center systems, cash registers... everything.

One of our products was a Windows-based package label printing system for small businesses. It consisted of a cheap PC, an electronic scale, and a label printer. Customers who shipped a lot of stuff weighed & labelled their packages themselves, and our driver would pick it up.

Everyone dug this product. Customers loved it for the convenience and price (we provided the PC + scale + Printer for 'free' along with 24/7 technical support) and Sales loved it because the customers were doing most of the work for us.

So when we learned that a local car dealer was unhappy that their label printer kept crashing I got an emergency call to go investigate.

Why me? I was our office gunslinger: the guy you called when everybody else had tried every trick they knew but failed. Sales was frantic, we'd replaced every component three times but still, the customer reported it quit printing pretty much as soon as our technician left. They were worried we were gonna lose a lucrative contract to a competitor.

When I get there everything is turned off, so I boot the system and test print a few labels: no problem.

I look around: it is a huge, brightly lit, professionally appointed automobile service center. Our label printing setup is in the parts department, where they ship car parts to other dealership locations in the state.

Outside the parts office, inside the main bay are vehicles getting fixed, hoods up. Some are on lifts. Mechanics are all dressed in identical, spotless jumpsuits. It's spotlessly clean.

I turn back to the label printing workstation and verify it's plugged into a massive UPS, a device the size of a small refrigerator because a previous tech wondered if spikey power was the issue. I checked that all the cables were new, sturdy, undamaged, and firmly seated.

On a previous troubleshooting session somebody even thoughtfully grounded the metal table legs with a soldered metal strap.

All around me: the hum of power tools & air impact wrenches.

I call the customer over and ask them to start labelling packages, which they do while I watch. They make it through six labels before Windows blue-screens and everything stops.

I reboot the PC and it comes back instantly. "Do it again," I ask them.

While they're printing labels I take a walk. The parts department is against an inner concrete wall, and I follow the wall until I come to a locked door.

"What's in here?" I ask somebody.

"HVAC," I'm told, as they unlock the door and flip on some lights. Inside is a narrow room/corridor stretching the length of the building. It is crammed with machinery, some of it powered on and humming.

I begin walking, counting paces until I reckon I'm opposite the shipping office on the other side of the wall. Against the wall is a floor-to-ceiling sized air tank next to a compressor: the centralized air compressor that services all the air-driven tools in the main bay. While I'm standing there, the compressor kicks on and an electric motor the size of my head spins up.

Mystery solved.

I return to the parts department and find my customer waiting for me.

"It crashed again," he says.

"I know why", I tell him. "We need to move your label printing station."

Turns out that whenever that pump started, it generated an EMP strong enough to reboot a Windows PC two feet away, on the other side of a concrete wall. Since the pump only started when the tank sensed low pressure, it only ran a few minutes at a time, multiple times per day.

By moving the table three additional feet farther down the wall, the extra space let the PC run without issue -- problem solved, and I didn't even open my tool kit.

Thanks for sending this in, Ralf Maximus!
Know another great story? Send it my way.

Want more?

Copyright © 2023 Hardly Software, LLC. All rights reserved.