Ask Slashdot: What's Your Worst Damaged Hardware Horror Story? 301
"Everyone has that story," writes Slashdot reader alaskana98:
You know, the one where you spilled a Big Gulp-sized cup of sugary Coke all over your laptop and it somehow still works to this day — although the space bar is permanently glued in place.
Or that time you left your iPhone out in a pouring thunderstorm, stuck it in a bag of rice and after a few days it miraculously turned back on. Yes, we've all been there, maybe cried a little and then went on with life — a little wiser for the wear.
So, fellow Slashdotters, what's your worst tale of hardware horrors?
The original submission has already drawn some interesting tales from long-time Slashdot readers, including two thunderstorm hardware horror stories. And there's also the user who remembers how "In the mid 1980s I blew up a $75,000 laser by not turning the cooling water on before firing it up."
But what's your story? Share your own tale in the comments.
What's your worst damaged hardware horror story?
Or that time you left your iPhone out in a pouring thunderstorm, stuck it in a bag of rice and after a few days it miraculously turned back on. Yes, we've all been there, maybe cried a little and then went on with life — a little wiser for the wear.
So, fellow Slashdotters, what's your worst tale of hardware horrors?
The original submission has already drawn some interesting tales from long-time Slashdot readers, including two thunderstorm hardware horror stories. And there's also the user who remembers how "In the mid 1980s I blew up a $75,000 laser by not turning the cooling water on before firing it up."
But what's your story? Share your own tale in the comments.
What's your worst damaged hardware horror story?
lightning (Score:2)
Lightning strike in the backyard took out our microwave oven and our garage door opener, as well as two neighbors' garage door openers.
Steel processing company: Lab near pickling tanks (Score:5, Interesting)
At one point in the 90s, the mom and pop I worked for then had landed a lucrative contract with a local steel processing company, who had a quality control laboratory placed in very very close proximity to an electrochemical pickling treatment system.
Such systems are used to strip off oxides, greases, and other unwanted contaminants from the surfaces of steel products prior to distilled water cleaning and then other surface treatments (like chem film). They are essentially hydrochloric acid tanks hooked up to high voltage. They naturally, produce *A LOT* of free chlorine radicals in vapor/mist form.
The geniuses that designed their building put the quality control lab literally right next to these systems. Without ducting outside air.
About ever month, we would have to replace all the computer equipment in their labs, because of the corrosive atmosphere being ducted right into that lab. It would transform the chasis of computer enclosures into 100% rust, and would destroy sensitive surface mount components on system boards, along with the traces themselves where it could get under the solder resist.
Imagine being able to take a pinch of motherboard, rub it between your fingers, and it crumbling into oxide powder.
That's what the entire computer fleet in that lab would turn into after a month.
I suggested to their maintenance dept several times on the calls I serviced at the site, that they needed to duct outside air, and have positive pressure ventilation for that lab. I still don't know if they ever did that.
Re:Steel processing company: Lab near pickling tan (Score:5, Interesting)
Long ago when I used to fix computers I had one come in from a garage. Whole thing was coated in black soot, inside and out. Some kind of mixture of exhaust fumes, tyre dust, road dust, engine smootz, oil and grease.
The machine had died and they were in a panic because their entire operation was relying on it to manage customer bookings and print out invoices/receipts on a dot matrix printer with special tractor feed paper. First thought was to just give them a new machine, but it was running DOS and DOS wouldn't run properly on most motherboards of that era, and even if it did they needed a parallel port for the printer and most machines were legacy-free.
We eventually found something old enough to just about work. They enquired about a backup system but there wasn't much available for DOS, certainly nothing simple enough for the grease monkeys to use.
Re: Steel processing company: Lab near pickling ta (Score:3)
Similar story with a small doctor's office that used a special dumb terminal attached to an AS/400 server.
Rather than hang the printer on the AS/400 itself, it was hung on the dumb terminal. Needless to say, the dumb terminal failed, nobody at the practice knew the admin credentials for the AS/400, and they had electronic patient records they desperately needed access to.
I left my employer at the time, but i rained documentation and prices for an IBM twinax to Centronics adapter, and documentation on how t
Re:Steel processing company: Lab near pickling tan (Score:5, Informative)
We eventually found something old enough to just about work. They enquired about a backup system but there wasn't much available for DOS, certainly nothing simple enough for the grease monkeys to use.
You can still buy brand spanking new 386s. They are ISA passive backplane SBCs or PC104 SBCs, but they are real 386s. Just google "386 SBC". The passive backplane types can even still be used with one's ancient expansion cards. (There are PC104 to ISA adapter boards, too.)
Re: Steel processing company: Lab near pickling ta (Score:2)
Yup, chlorine pretty well eats anything.
We supported printers in a an abattoir for a while, noticed one in particular where the chassis was rotting... came to find they chlorine fogged the room periodically to kill any anthrax spores....
The other ones were kept in meat processing rooms at 0C (32F) and after 4 years or so, weâ(TM)re just chucked out as the rust build up inside was something to behold, it was kinda cool to watch an A3 sheet come out in a whoosh of steam due the temperature difference tho
Broke a new CPU (Score:3)
I had scrapped and saved to have enough to build a new computer. Finally had enough and bought the new motherboard, memory, CPU, etc. When I was attaching the heatsink to the CPU, I somehow screwed it up and cracked the CPU. Now I had all the parts for a new computer except for that all important CPU which I could not afford at the moment. I had to wait a couple of weeks to get it.
These days I don't build my own systems. I make enough now to pay people to do it for me. I still pick and choose what goes into it, but I don't bother with the grunt work.
Re: (Score:3)
The very first time I built my own computer from bits the man in the shop said he'd do the actual installation of the CPU in the motherboard to make sure the settings were right. (This was in the days before CPUs needed heatsinks.)
I did all the rest and the computer worked fine for about 6 months, then became unreliable. Having replaced most things I came to the conclusion that the fault lay in the CPU and so ordered a new one. When I went to install it I discovered the problem - he'd set the voltage jum
Re: (Score:3)
Aside from the already-mentioned exposed die, this could have been back in the era when CPUs were often in ceramic packages. In the 386 era, that wouldn't be as much of an issue (no need for a cooler), but ZIF sockets were like hen's teeth. Later on, you might have a ZIF socket for your K6, but if you botched the cooler installation, you could once again crack it. I'm always worried, even with today's metal-packaged monsters, but have managed to avoid breaking a CPU or socket.
Exposed dies seem to be more of
Re: (Score:2)
Re: (Score:2)
That was such a common problem back in the day, they made Athlon heatsink shims - basically a piece of metal slightly thinner than the CPU. It's sole purpose was to prevent the heatsink from tipping to one side which would crack the die and render it useless.
https://en.wikipedia.org/wiki/... [wikipedia.org]
https://www.legitreviews.com/w... [legitreviews.com]
It should be noted that Intel was using the same style of flip-chip packaging, however they were more robust and often could have several chips on the side and corners and still worked. A
hard drive (Score:5, Interesting)
Re:hard drive (Score:4, Interesting)
One of the mainframe guys I used to work with in the late 90's had a photo of one of the platters from the hard disk embedded in the server room wall, this was one of those early IBM beasts with 24 inch platters that had a catastrophic failure.
I suspect that was a very old photograph, because I heard a similar story in the 1960s. The drive was the IBM 1301, a very large disk drive. According to the story, if a head crashed in the outer track, it provided enough torque to snap the arm holding the head and hurl it out of the drive with enough force to beak the class door and embed itself in the wall. We ran out two IBM 1301s close to the wall and facing it, so if something like that happened only the wall would be damaged.
Re: (Score:2)
Re:hard drive (Score:5, Funny)
I heard a story many years ago of a tech who went to investigate a cascading failure in a row of washing machine hard drives. A single platter had failed, which damaged the head. They thought maybe it was just the pack which was faulty, so they put in a new pack, and destroyed that as well. So they thought maybe it was the drive which was failing, so they put the first failed pack in another drive, and destroyed the head on that as well. Their attempts to figure out what was wrong ended up killing three or four disk packs and as many drives.
When the tech turned up, he ran his finger around the inside of the mouth of the drive, and his finger came out covered in red dust.
"What's that?" the customer asked.
"Data," he replied.
Re: (Score:3)
In the late 60's, our research lab bought a huge six-platter disk drive, with the platters mounted vertically, like wheels. It took 2 3-phase motors to get these things going, after which one would drop out. We calculated that if the case and building were to dissolve suddenly, the disks had enough momentum to roll over the Bay Area coastal range and into the ocean. Ultimately, the device was heavily damaged when the super-careful and super-snobby technician left a tool in it and scratched half the surfaces. We used the rest for a couple years more. I think the lawsuit is still making its way through the courts.
This sounds like the Librascope disk at the Stanford AI Project. I left at the end of 1968 but still heard about the calculation. I had not heard about the tool damage--thank you for that.
I wrote a diagnosic program for that disk. It was my second experience with unreliable disks, the first being with the swapping disk on the PDP-1. Les Earnest claimed that the Librascope disk would lose data when the temperature changed by two degrees.
Five laptops (Score:4, Funny)
I sent out an email saying that open drinks are not allowed near company computers.
The 'affront' of being told not to do something traveled up the food chain until the Director of Marketing showed up at my desk wanting to know why I would send out such an "offensive email" and it was obvious she was there for my butt. I explained that in the last calendar year we had to replace over $8000 in laptops from the negligence of her trainers. She turned right around and then sent out an email to the trainers telling them that open drinks are not allowed near a computer.
Re:Five laptops (Score:4, Funny)
Virtually brand new fully loaded MBP. Sitting down at home with a pint of beer doing some work and the inevitable happened. Turned it off as quickly as possible, washed it with de-ionised water, dried it out with a hair dryer.
First power up - nothing. Shit. Left it for a day. Next power up got a chime but no screen. Hmmm. Connected to a TV via HDMI port and desktop appears. Whoopee! But not very portable now as plugged into 75inch TV.
After a couple of hours, there was a flicker or two from the screen and voila, it was back! Worked perfectly ever since, apart from a very faint whiff of hops to start with when it got hot enough for the fans to come on...
Re: (Score:3)
I had a laptop that drank almost as much beer as I did. (That's a bit of an exaggeration.)
The thing was magical, however. Every time it drank a beer, it only killed the keyboard. I replaced the keyboard in it so many times (it went through four or five) that I had the process down to under 15 minutes to replace it - on a laptop that went together like a 3D puzzle. I have no idea why it only killed the keyboard and nothing else, as it sure consumed a lot of beer.
Re: (Score:3, Interesting)
We had 8 field trainers and in one year they destroyed 5 laptops by dumping drinks in them. It is pretty obvious that if you put an open drink next to a computer when a person turns around that the sweep of their arms will tend to push drinks into the computer.
I sent out an email saying that open drinks are not allowed near company computers.
The 'affront' of being told not to do something traveled up the food chain until the Director of Marketing showed up at my desk wanting to know why I would send out such an "offensive email" and it was obvious she was there for my butt. I explained that in the last calendar year we had to replace over $8000 in laptops from the negligence of her trainers. She turned right around and then sent out an email to the trainers telling them that open drinks are not allowed near a computer.
To this day, that's the reason I still use only notebooks that have keyboard water drains (business class thinkpads, among others).
Blew up a prototype (Score:4, Funny)
The fire computer (Score:4, Interesting)
He had an old Gateway 2000 desktop with massive burn marks down one side of the case that was in both fires and the damn thing still ran. We laughed every time he turned it on, like we were waiting for it to spontaneously explode or something.
Re: (Score:3)
Once I did an insurance report for a computer that had been flown in from the United Arab Emirates. The guy checked it in on his flight and when it came out the other end it had gone from a normal computer tower shape to some kind of abstract sculpture, kind of like a Picasso painting in 3D. It was one of those all-metal Lian Li cases so no plastic to shatter, it was just massively deformed.
Managed to get most of his data off the HDD but the rest was a write-off. The insurance company kept bugging us to see
USB Killer (Score:3)
Re:USB Killer (Score:4, Interesting)
Older laptops sometimes had very poor USB implementations that could die if they were shorted out, especially if the short was between 5V and the data pins (which are 3.3V maximum).
I think the spec says that all pins should be 5V tolerant and the supply limited to 500mA, but of course back in the USB 2.0 era a lot of crappy hardware didn't bother. I've seen machines that had no current limit on the 5V line at all, so a short would dump a few amps into the controller chip.
The opposite... (Score:5, Interesting)
My boss had the opposite experience. He was into weird ergonomic keyboards (the company was a bit related to text entry, so it was both personal and professional interest), so he had some interesting ~$200 devices, IIRC his favourite one did not have any labels on the keys, so you HAD to touch type. He was also a big Apple fan and had bought the new Mac Mini of the time (~10 years ago). One day, his fancy keyboard stopped working. He tried another fancy keyboard, nothing, then a third and a fourth, which was all he had there. That's when he involved other people, we tried the keyboards on a windows machine and they were dead. Obviously the USB was killing them, which was quite unusual for a 6-month old Mac Mini, and he was out of luck I guess about the expensive keyboards.
He went to the Apple store (NYC), they told him they would replace the board and he would have it in 2 weeks. In two weeks he went back and they told him they had invalidated the warranty because they found "dust inside". This was a new Mac from a non-smoking, no cat etc office, where we've been using computers for years. They told him they were willing to get him 10% off a new device. So he just buys a new MacBook Pro. He comes back and tells me what happened, my jaw drops as I try to explain it's impossible for them to invalidate the warranty for dust - especially when they are the ones deciding whether to install dust filters or not, which is why it's not in their TOS, but he tells me the Apple genius said so, so that's how it must be. The next day I see him with a new Apple Cinema display, I ask him what happened to his 6 month old nice Dell monitor? Apparently the new Mac could not connect to it (I've forgotten why, but it was some matter of dual link dvi or something?), so he went to the Apple store and they told him it won't work with that monitor, he had to buy a Cinema Display....
So, yeah, I call it the "capitalist USB". Cost lots of money, made some great sales for Apple.
Not damaged but disabled... (Score:5, Interesting)
Re:Not damaged but disabled... (Score:5, Funny)
Yeah, back in the late nineties I was logged into a linux server at a similar NOC some 100km away. After finishing, I shut down my workstation with "shutdown -h now", not realizing I was still in an SSH session to the remote box. Which was running 600 websites. At sunday night. I realized the situation after I noticed the workstation took an awful long time to shut down.
Re:Not damaged but disabled... (Score:4, Insightful)
Done that, more than once. I install the mollyguard package on everything now.
Re:Not damaged but disabled... (Score:5, Interesting)
That was easy to do back in the old days before IPMI. After fat fingering the new IP address on a Linux box, I learned to always set a restart 10 minutes from now, then make the change live. If it worked OK, cancel the restart and CAREFULLY change the configuration files. Have someone else double check that the config files match the current running config.
Serial console was much better, but the lack of power cycle could sometimes be a real problem. I saw a few setups where one machine had a digital I/O board hooked up the the other machine's reset buttons.
Once IPMI got more or less reliable, things got nicer. Then Intel had to fsck it up with the management engine :-(
Re: (Score:3)
That's the kind of thing that the much-hated Intel Management Engine is supposed to solve. Even if the machine is "off" you can still connect to the Intel ME, power it up, view the screen via VNC even in the BIOS etc.
Unfortunately thanks to Intel's competence so can everyone else...
Vacuum Cleaner XP (Score:4, Funny)
Re: (Score:3)
I've never seen one with a regular NEMA outlet on it. Every daisy chain power supply I've ever seen had a C13 and a C14. To connect the monitor, you needed an IEC extension cable.
I miss those days a little bit. Your PC turned on and off with a big switch that gave a heavy thunk, and your monitor turned on and off with it. After that era ended, it took a surprising number of years before monitors got smart enough to simulate that automatic power control function properly. I knew a guy who smoked indoors
Cat peed all over a vintage keyboard (Score:3)
Our cat (now deceased) who had a bad habit of marking profusely did the deed all over the Ensoniq Mirage I was storing for a bandmate in our basement. Didn't find out for a few days...
Man, scrubbing circuit boards and dissembled key contacts is bad enough, but semi-dried catp*ss... phew! The board was 100% afterwards though. They were built like tanks.
Re: (Score:2)
That sounds like a friend's VCR. His cat hacked up a hairball right over the case vents. The cleaning process was no fun. We had to patch up a couple circuit traces that were damaged as well, but it worked fine after that.
Fried two PS1 dev boards (Score:5, Interesting)
I was working at a game studio on a PS1 game in the mid 90's. Now there was no standalone PS1 dev kits. Instead the PS1 dev card was actually two ISA daughterboard cards [twitter.com] -- that housed the PS1 -- and plugged into a PC. (The standalone blue PS1's were the debug kits.)
Now Sony in their infinite wisdom had the same 9-pin D-sub connector as the EGA monitor output AND the gamepad input ports on the back of this PS1 dev card. You can probably see where this is going ...
A co-worker and I were tasked with setting up a few new PCs with the PS1 dev cards. We plugged in the gamepad, hooked the EGA monitor up, fired it up, and nothing. Puzzled we powered the computer off, pulled the card out, swapped in a new one, hooked it back up, hit the power and again nothing.
A few seconds later one of us asked "What's that smell?"
We quickly turned the power off and then realized why the two cards weren't working. We had accidently plugged the gamepad input into the EGA monitor out connector and the EGA cable into the gamepad input connector due to the shitty design of both the OUTPUT and INPUT ports having the same pin count and gender!
We told management that "Uh, two of the dev cards we got aren't working". We also sent an email out saying "Please be extremely careful hooking up the PS1 dev kits."
Last we heard was that it cost the company $20K. We were never sure if that was each or total and we didn't dare find out.
We never did tell anyone what happened about those two "mysterious" PS1 dev cards. ;-)
A few years later when I was doing PS2 game development I noticed that the PS2 dev kit "TOOL" had the gamepad input (9-pin D-sub) and monitor connector (15-pin D-sub) be a different pinout and gender. I always wonder how many PS1 dev cards Sony got back due to their bone headed decision to make it extremely easy to accidently hook up. I guess they must have paid attention and fixed it with the PS2 dev kit.
Re: (Score:3)
A friend blew up his PC and two HDDs full of work. His PSU died and it was a module one, so he went and bought another modular one, plugged everything in... And it turns out the new one used a different pin-out with 12V on the old 5V line and some of the grounds.
I've also seen a few IDE devices where someone has forced the cable in upside down. The cables are supposed to have tabs and one blocked off hole, but some cables didn't have the tab part. With a bit of force the pin that is supposed to be missing o
Re: Fried two PS1 dev boards (Score:2)
Torched multiple boards in avionics computer (Score:3)
About 3 months into my first job as an engineer at McDonnell Douglas I was probing a power supply in an AV-8B Stores Management Computer (the one that controls weapon delivery). I was using a pig-sticker (just a wooden stick with a metal pin in the end that a clip from a multimeter could clip to) to get a measurement. I accidentally tilted the stick so that it shorted a couple of points. A large capacitor caught fire. The fire was being fed by burning metal so was very hot. It was also coming out of a small hole in the top that was focusing it into the shape of a blowtorch flame. The torch burned a hole through several boards and a wiring bundle. The cost of the repair was just over $30K (in 1986 dollars).
Luckily, it wasn't the worst f-up of the week. Another engineer managed to drop a $250K HUD onto a concrete floor (not exactly good for the delicate optics).
Re: (Score:2)
Your tax dollars at work.
Re: (Score:2)
My dad talked about a couple incidents where he worked somewhat similar.
This one was an intentional test. During development of the F-35 they were doing robustness testing of the main battery, by dropping a conductive bar of metal across the battery connections. The bar of metal welded itself to the battery terminals and it was more than a couple days before the battery was cool enough to handle. After detaching the bar the battery charged up and worked just fine, after which they promptly disposed of th
Computer repair (Score:5, Interesting)
There was a customer who brought his computer in for some type of service. Probably a dead hard drive .... and probably because the hard drive was so caked in fuzzy, insulating tobacco tar that it over heated and died prematurely.
Inside? Tobacco bunnies all over the bottom of the case.
Between the drives? Tobacco.
The heat sink? It was one fuzzy blob of tobacco tar. The fan? It spun, though a bit slowly.
The machine was clearly thermal-throttling to oblivion, but the user's only concern was the hard drive. There was only so much (little) extra that vacuuming it out would achieve.
This computer reeked of cancer. When it came in, somehow I got the job of "fixing" it. So I did. It was open for as little time as possible. I wore latex gloves whenever I had to go near it. It made the whole damn lab reek of a rank ash tray. Not wet -- that's a far more ashy smell -- not dry, but tar-ry.
Once I was done, I slid a tight-fitting trash bag down over the top. Partly to help seal in the smell. (Not a single person in the place complained of my suggestion, not the secretary who gave it back to him when he came to pick it up, not even the small business owner that had to deal with my antics. Unfortunately I wasn't present when he came to get it to have been able to give him the stank-eye.) Honestly, partly it was to provide a little bit easier of a disposal option for the customer: this computer was toxic, noxious waste, and it needed to be dropped into the nearest dumpster. Preferably triple-bagged. Whether or not it had all RoHS components, it was toxic waste.
Re: (Score:2)
You're not the only one to notice how smoking and cooling fans are a bad mix. Back before the pandemic, my lifetime-smoker sister came to me thinking that her laptop had a virus on it. We fire it up, and the fan is sounding like a jet taking off. I went through it and found no trace of malware. At this point, I opened it up, and found it was just choked with tobacco fuzz. After cleaning that out, the laptop ran a lot better, for some reason.
I'm just glad she didn't run a heavy-duty gaming rig in a high-airf
Catastropic Failure While Backing Up (Score:3)
Aghast, I opened up the computer, and smelled the acrid stench of the burning money. Several of the capacitors on the power supply for the motherboard had exploded, spraying their electrolyte from their split can lids. Something else was wrong. I pulled the drive from the bay. There were components on the drive bay. A transistor was smoked. I pulled the drive from the caddy.
I built a new computer system. Setting my backup drive aside, I installed the old system drive as my secondary. After a reinstall, I did a drive check on the main system disk. It was corrupt! I tried the backup drive. It was completely dead!
Eventually, I was able to copy folders across, but on my old main drive, track-0 was physically damaged, preventing its reuse. In the end, I just barely kept my data. I didn't lose anything but money and time. My data was safe. So, the moral is: add hardware abstraction in the backup chain.
The $125K CCD camera (Score:5, Interesting)
scariest false alarm (Score:2)
While driving home from college one day at triple-digit speeds, my rear wheels locked and a cloud went up behind me. Putting it in neutral, I managed to coast it to the shoulder of the road and didn't bother to even look under the hood before flagging a ride the rest of the way to town. I just knew I'd blown another engine.
The next day I went back with a tow truck to get the car. When we popped the hood, the first thing we noticed was that all we smelled was coolant - no oil. Then we noticed a puddle of coo
Electrocuted a Cow (Score:5, Interesting)
I grew up on a farm and the weekends of my youth invariably involved working on the farm for my father.
One Sunday I had to hook up a trailer to take waste out to be burned in one of our fields.
The tractor I was driving had a front end loader [deere.com], the bucket wasn't fitted it was just the arms.
So I'm looking backwards over my shoulder reversing in to the trailer to hook it on when I hear a loud bang.
Because I hadn't been paying attention to the front of the tractor I hadn't noticed that the FEL arm had swung under the tension wire supporting the electricity power pole in the middle of the yard (yeah stupid place for it).
I had pulled the whole pole over towards me and the power lines had snapped.
Unfortunately for one of our cows it was scratching its arse against a metal gate where the power lines landed and got blown away.
It was on it's back with its leg twitching in the air. I thought it was well cooked and I was in shit with dad.
However after 10mins it was up and off munching grass as if nothing had happened. Tough animals, cows.
The major damage was that I blew out the power grid for all of East Kilbride [wikipedia.org] on a Sunday afternoon.
The power company guys weren't pleased about their Sunday call out, or the angry phone calls about spoiled Sunday roast dinners, but my father was pleased because they not only reconnected it but moved the pole to a more sensible location where it wasn't in our way anymore.
Re: (Score:3)
OMG this wins.
Banged up, but still working to this day (Score:2)
To make a long story short, I once had to take a hammer to a PowerBook’s casing to literally beat it back into shape after a fall left it in bad shape. I still keep that PowerBook around to this day, since it was the highest-end PowerBook ever produced that could still natively run Classic apps, making it the perfect machine for playing all of those great, classic, Mac games that never made the jump to OS X or Intel (insert a well-deserved joke here about Mac gaming).
Seriously though, I still bust it
Probably going to date myself... (Score:3)
Back in the AMD K6 days, I was one of the early watercooling moders. I took the square bit off of a sewage cleanout plug and trimmed it off with a Dremel to fit, then glued it snugly over the top of the CPU. Added a couple of hose barbs, a heater core out of some late model vehicle, and was off and running. One day I noticed some algae growing in the clear tubing. So... I added some bleach to kill the algae. It was shortly thereafter that I learned that bleach eats aluminum... what the top of the CPU was made out of.
Which polarity is this superconducting magnet? (Score:2)
Frozen UPS batteries (Score:2)
Worked in a place where there were a lot of green types. Did not like the aircon to be on in the server room, so switched it off and opened the windows because, Hey! It's winter! We can save power!. The temp dropped quite low and the water in the lead acid batteries in the APC UPS froze, cracking them. When I walked into the server room it was freezing and smelled of acid fumes.
Re:Frozen UPS batteries (Score:4, Interesting)
It's still not long ago or far enough away to post under a username, but, in some small central European country just before the dot-com boom:
Guy is installing UPS batteries for one of our machine rooms. It's a big block of batteries and he's got them all charged up and now has to measure the voltage over them. So he puts a ladder over the top of the battery rack and goes over the battery terminals measuring. Unfortunately, as he's getting down, he knocks the ladder which shorts the entire battery rack.
Being an actual solid ladder, rather than evaporating, as your normal power cable would, it manages to carry the entire current of the short circuit, but starts glowing red. Guy is lucky as hell because he has the presence of mind to kick the ladder which breaks in half breaking the circuit before the entire battery rack explodes. The photos of the room afterward are a shock.
shutdown -h now (Score:4, Funny)
If you do a shutdown -h now, you make sure you are on YOUR laptop, and not via SSH on the SERVER. And don't do this on Christmas day. Especially not while you are doing a quick 5 minute check if everything is still running during the evening and the server is 20 miles away. And you car is broke. And it is snowing outside as if there is not tomorrow.
Exploded iphone - got it working again! (Score:3)
I had an iPhone 6+, which was a hand-me-down from a friend who I always helped with their IT. My daughter was now using it (without sim card) as a glorified ipad. Its battery life sucked, so I bought a replacement kit on eBay.
This wasn't the first time I'd be replacing a battery. I replaced batteries on my own iPhone 6, my wife's iPhone 7, and an iPad mini. It almost became routine. Open up the housing by unscrewing the two screws next to the charging port, and prying it open using the suction-cup pliers I once received with a screen replacement. So far so good. Unplug the battery from the mainboard, heat up the glue with a blow-dryer, and pry it loose from the chassis.
In the past, I had used a plastic tool for this. But somehow, I lost it. So this time, I used a metal prying tool. Big mistake. Huge mistake.
Somehow, I damaged one or more cells of the Lithium-Ion battery, and it started hissing and burning. I was on the top floor of a three story townhome. Holy shit, what do I do? I grabbed a pair of pliers to hold it by a corner, and as it was burning I ran down 4 flights of stairs towards the front door. Halfway there, the fire alarm started going off. I finally made it, and threw the phone in the grass, and waited for it to finish burning.
Miraculously, none of the other components were damaged. The damaged cells were on the outside end of the battery. I was able to pry the whole thing loose and take it out, and cleaned the phone. Then, I took the new battery and connected it, to see if it would turn on.
It did. I patched the phone back up, and it was used for another year.
PS2 port; not hot swappable (Score:2)
Re: PS2 port; not hot swappable (Score:3)
Since you seem to be knowledgeable: "hot swappable" - this means "flashes up when accessed during operation", right?
Just asking for a friend.
I'll show myself to the door now.
Hot swappable RAID set (Score:2)
I had just received an extremely expensive brand new DEC Windows NT server with 2 CPUs and 128 MB RAM. It was 1995 and the vendor demonstrated this fancy RAID setup which would improve reliability. I asked him "So I can just pull out any of those drives while the system is running??" to which the vendor said "Go ahead!"
Surely, once I pulled that drive out of the bay, the RAID set failed immediately and we had to wait for a new hard drive to be flown in from Germany. That was a tough lesson.
My wife's PC -- bad PSU (Score:2)
Part of the power supply turned red hot, fell off, and was flicked out of the case by the fan and onto a pile of papers behind her computer. Luckily we were home when it happened and went to investigate the noise. I am fairly certain after taking it apart the failed component was a thermal sensor for the power supply ...
Sugar drink on laptop - NP (Score:2)
You just pop out the battery, and rinse it in the shower, and let it dry for a day or so. Assuming you are fast enough and nothing short-cutted first, or you have a ThinkPad or other laptop that has spill resistant keyboard to begin with.
Re: (Score:2)
Re: (Score:2)
If it doesn't have a removable battery, then it is only a disposable object anyway. So dispose of it.
Appatently I fried a $50k camera (Score:2)
Appatently I fried a $50k camera once in early 2000s. The beast had a 30 megapixel chip back then, about the size of my palm. It was one of few custom-made devices for a very specific purpose. Slow as hell, reading out a picture took almost a minute (didn't have to, but the hardware manufacturer was peculiar).
The idea was that camera timing should be very deterministic so you could calculate various events. I had to write a Linux / RTAI driver for it, and somehow it magically broke while in my possession. T
Oooh yeh. (Score:2)
Back in the early 80’s I was working on DTS computer based cash registers, very expensive units that were amongst the first to do barcode scanning. I was away in the country, and had a spare unit in the back of the Datsun Sunny station wagon. When I took off rather quickly, it flew out of the back of the wagon,and hit the road, as the tailgate wasn’t closed properly. Broke all the plastic mounting studs inside. I superglued them up, and put it back in the store when I got back. Luckily, it was n
MITS Calculator (Score:2)
As a child I won a MITS LED calculator kit as a door prize at an Altar 8800 seminar. Assembled it and was quite happy with it, although it ate batteries (6xAA batteries in the back of the case assessable by removing 4 screws). Unfortunately my sister borrowed the calculator, then visited grandma, who then gave her a half dozen wet cucumbers which said sister stashed in her purse with the calculator. The electrolytic corrosion of the PCB traces was total as the 6 AA batteries discharged through the wet calc
Fiber optics is tough! (Score:2)
But the miraculous thing is that the fiber optic core (a millimiter-sized glass strand) survived the fire and carried traffic without any issues for more than a day until the repair equipmen
Bad day at work (Score:2)
Re:Bad day at work (Score:4, Interesting)
Too much Caviar (Score:2)
Not as bad a paying someone to make backups (Score:2)
Our company hired a pretty expensive consultant to make sure our backups were taken care of.
Only to realize the backups themselves were completely messed up.
Double Trouble (Score:3)
The first event happened during a year of major refurbishment to the site. I came in to work one morning to find the door to the "Data Center" pegged open [which was against my instructions, since the room ran on positive pressure to help the aircon], but when I looked inside I could see two of my duty operators clambering over a set of 3480 tape drives. One was positioning a bucket on the top of the tape drive unit to catch an impressive spout of water that was emerging between two ceiling tiles; the other had an A-frame ladder and was trying to remove a tile a few feet distant, in the hope they could see what was happening in the ceiling void.
It turns out that part of the building work had resulted in a *huge* pool of water forming on the roof which had then managed to find a way in to the building, had run down the inside of one of many of the interior partition walls until it reached the ceiling to the computer room. At which point it had just come flooding through. Cost us a pair of double-hopper 3480 tape decks, which somehow we got replaced on insurance, despite it almost certainly being contractor negligence...
Second incident was in the same room a couple of years later... More building work, this time an extension was being built on to the building. The original "end" of the building was demolished, turning the far side of the long wall of the computer room into a temporarily "outside wall". The contractors were digging out the footings at the time. More massive rain, which flooded the construction area enough to stop work. What we didn't know at the time was that when they had been removing the unwanted part of the building adjacent to the computer room, they had managed to dislodge a single brick from the wall... at the exact level of our false floor, which was about 6 inches deep and which housed all the mains and data cables for our kit: 2 mid-range servers, 4x3480 tape units, 2xopen reel tape units, 2x1,200lpm impact printers, multiple racks of Netware and Windows servers, one large comms rack...
I came in to the room to find one of my tekkies with a floor tile lifter and tile, looking in to the floor void. Not expecting any work to be happening in the floor, I asked if we had a problem... "Yup!" Four inches of dirty water, in a void full of 3-phase power. We had a major "Distribution Panel" for our electrical feed with two of what we called our "Hammer, House of Horror" switches - in that it would be "Horror" for us if we ever had to use them in anger. First and only time I used them.
But with metal-framed gear, an aluminum floor, dirty water and a lot of cables...
My personal stories... (Score:2)
1. I left out my 5.25" Apple 2 floppy disks on my friend's backyard wall and its sprinklers came on. Some disks died from the water damage. :(
2. I accidently fried my running quad core PC after letting one of its case fan's cut power cable touch the PSU. :(
3. As a newbie in my brand new custom built 486 DX2/66 PC, Norton Utilities v8.0 (DOS)'s Disk Defragger hosed my HDD's datas due to its VERIFY=ON bug. :(
Worst? (Score:2)
Re: (Score:2)
So did you send the HCF command or were you responsible for the fire suppression system? :-)
I've had a few (Score:3)
I also have had 2 separate power supplied try to burn my face off. Both times trying to troubleshoot a no-start and at some point I put my face around back by the fan to see if it was spinning right when they decided to shoot flames out the back. You would figure I'd have learned the first time. Ah well. Now if I want to check if a fan is spinning and I can't see it I use the end of a small zip tie.
Then there was the time I accidentally shot my monitor, but we don't talk about that one.
Ruby Rod Laser (Score:2)
Back in the 80s I was a technician at a laser company. One of the engineers was installing the ruby rod into a very expensive unit. It as a 5/8" diameter ruby rod. When he tightened down the bracket the rod split the long way. Because he had used a 1/2" bracket. Each ruby rod cost between $10k and $20k in 1980 money, so roughly $30k to $60k in today's money.
We gave him a special award to commemorate the auspicious occasion.
The Pepsi Syndrome (Score:2)
There's nothing quite like the light show you get from spilling your soft drink onto a vacuum-tube audio amplifier. Did that one when I was a wee lad and such amplifiers were the norm.
The $100M bucket wheel excavator (Score:2, Interesting)
I built a monitoring system to keep an eye out for upcoming failures on our bucket wheel excavator.
One of those scoops is the size of a car and the motors powering such machinery are just insane.
Unscheduled maintenance means ships waiting in the harbor, which can be like $300K per vessel per day.
So one morning the alarms trigger for the front wheel bearings and I eagerly report to the chief of maintenance.
I get sent away with the message that the excavator is still running fine.
Two weeks later the front buc
Worst lightening strike ever at a customer. (Score:2)
The next day we got a call, nobody could log in.
Dozens of network switches, computers and some servers were destroyed.
Central storage had to be rebuilt. Typically replace and reinstall. It took days to get everyone up and running again.
We guessed it overloaded the electrical system through ground.
We had some devices still failing weeks after the event took place.
All in all, a great time to upgrade to the la
You know those giant construction cranes? (Score:2)
Of course the crane towered completely over my house, and of course the crane got struck by lightning and the EMP took out all my home I.T. gear. Some of it just died slowly, which was even worse in hindsight actually.
Fun with processor bringup (Score:2)
This one still makes me angry. (Score:2)
We're setting everything up waiting for the union electrician to show up and hook up the large power drops for everything. I've got everything ready to go when the guy shows up. He starts working and half an hour later he say's he's done and then flip
Probably not the goriest rather the most stupid. (Score:3)
I was tasked with doing pre-startup and commissioning activities at a Chinese chemical plant. We were well underway testing control and safety systems. Honeywell was here doing doing some database maintenance on all the operator stations at the same time so I took a break and sat myself in a coffee room. I overheard one of the contractors talking to one of the plant staff complaining about 2 large UPSes with the power cables just kind of running through the room and going underground. The staff were asking what the UPSes were for and the contractor didn't know suggested we turn them off to find out.
It was Thursday morning I hadn't had my morning coffee yet so I didn't jump up and abuse everyone as to what a horrible idea that was. 10min later I walk out into the control room just in time for all operator stations to go dark. That's what the UPSes were for. No amount of dual PSUs can save you when you turn both upstream UPSes off at once. Fortunately due to the database work underway all drives were corrupted.
I say fortunately because Honeywell told us it would take 2 days to rebuild and since it was Thursday I quickly jumped online booked a hotel and a ferry to Hong Kong and decided to have myself a fantastic long weekend.
Probably not the most stupid rather the goriest. (Score:3)
I was doing maintenance work on two large 24V UPS supplies that kept the batteries topped up for about 4 safety systems at a medium sized refinery. I'm sure you can see where this is going.
The UPSes were dual redundant down to each card in the safety system. That is each UPS had a distribution panel with circuit breakers, which spread down to more distribution panels each having a twin fed from the other UPS, which all spread down to individual card power inputs on the safety system which also had a second feed from the other UPS. The procedure I wrote involved visually confirming all circuit breakers were in the correct position.
We took off the first UPS without issue and swapped out the capacitor bank. It did seem like a good day. Powered the UPS up again without issue. After lunch we came back to the second one, called on the radio "UPS12 B is being powered off, you'll get some alarms.". Suddenly an operator shouted back at us "We got alarms, turn it back on! TURN IT BACK ON!"
Turns out one of the circuit breakers was broken internally. The switch was on but it didn't make contact. We knocked out the power to some 30 field I/O cards, together servicing some 400 field instruments. All being safety related it meant that for any fault something goes to the safe state. I felt, sick but the effects I wouldn't know for a few days.
So 2/3rds of the refinery was shut down. Fluid Catalytic Cracker, Hydro Cracker, Alkylation plant, Jet Merox, one of the Crude units, Both of the Vacuum units, and the north ends Gas processing plant. However... The shutdown was not in any way controlled. There were procedures for dealing with a safety function tripping the plant, but there were no procedures to deal with every safety valve in the plant just deciding to close at the same time.
In the Alkylation unit HF acid slumped in the towers rather than getting flushed out. The unit was restarted, flushed out, and shutdown for 4 days for a detailed inspection and x-ray.
In the FCC we were near end of run, that means lots of coke buildup on the walls. Due to the sudden shutdown all the metal in the unit thermally contracted dislodging the coke which fell down into the first stage cyclone separators and blocked flow through the unit. They discovered that the following day while trying to startup.
Summary: Tripping a medium to large refinery for a few hours is a $3million event. Shutting down the biggest money maker in that refinery for 2 weeks to clear blocked cyclones is an order of magnitude worse.
Regardless of how many people say it wasn't my fault, the reality is I was able to write a new procedure that avoided that hidden failure.
Old House Wiring (Score:2)
My parents' 1890's house - no good ground. TrueVision Graphics Adapter - 24-bit video card that cost six months' paper route money, purchased for my 486DX-25.
Dry air. 1" bright blue spark from VGA cable to port. No video signal. Sad boy.
I may have been 14, but a half a year's salary mistake still bit just as hard as it would today.
Re: (Score:3)
I now have a humidity monitor stuck on my rack and a full 42U rack in the office), and a humidifier in the house for the winter times when it drops to 25$-30%. Also, anti-static
Not reading the protocol (Score:2)
somehow my doctor had Douglas Adams ability to count and at the end it fried all the wires on the right side of my body and the hardware no longer functioned properly.
YAG laser power supply (Score:2)
From my PhD advisor: in the early 80s his Nd:YAG laser power supply was acting up. These power supply units were the size of refrigerators and had a lot of handmade circuits. A repair person came to take a look at it. Given the lax safety rules of the time, he came walking in with a nice hot cup of coffee and spent the morning drinking coffee and fixing the unit with the cup on a shelf inside it. When he was wrapping up, he went to grab his coffee and knocked it over, spilling the coffee all over the insid
My story is probably unique... (Score:2)
I tipped over a bottle of oramorph (morphine in liquid form) and it spilt all over my Nintendo 3DS. Despite taking it apart down to the component parts, cleaning every piece multiple times, then putting it all back together, the home, select and start buttons are super-super-sticky (just like oramorph is) and I just don't know what else I can do to fix it...
Re: (Score:2)
Cocaine in the 51/4" floppy (Score:2)
Wafer boxes (Score:2)
Re: Wafer boxes (Score:2)
Horror, no? (Score:2)
I remember when we had to manufacture a special cable with parallel/centronics on one side and 240 Volt on the other side.
During a moratorium on new hardware, it was the only way to get new printers by getting them to flame out.
Movers tried to use a hand-truck on a full rack (Score:2)
In my favorite edition of "Pick the Lowest Bidder and See What Happens!" we contracted to move our classified data center from the Washington Navy Yard down to the Naval Information Warfare Center Atlantic. Some no-name company put in the lowest bid, despite zero experience with large-scale transportation of electronic equipment. I think they might have moved a few individual servers at some point in their past, but that was it.
They showed up at the data center with a bunch of dudes and refrigerator hand-
Sun E10K -- Destroyed (Score:2)
I was working at a big networking company in 2000 and was tasked with configuring a soon-to-be-delivered Sun Enterprise 10000 [wikipedia.org]
for a large customer. For those not familiar, this is a refrigerator-sized computer and fully-loaded went for over $1,000,000. The one we were expecting was about half-full, with 32 UltraSPARC II processors and accompanying RAM, etc. That's what the PO said, anyway. When we opened the back of the panel truck we found that the unit hadn't been secured properly. There were bits of brok
Here's my story (Score:5, Funny)
I once installed Windows Vista on a computer. The hardware ran fine physically, but never recovered psychologically.
DSLAM wiring issues (Score:3)
Plugged in some new 48 volt telco side ADSL equipment the connectors were red and black wires coming off something that looked like a short ATX motherboard power connector. Screwed the wires to the fuse panel, turned it on and BANG, shower of sparks and a dead DSLAM. I call the company and they tell me that red and black are reversed in their country and I plugged it in backwards. Next few installs go OK. then BANG. "You told us you didn't like the colors so we reversed them for you.
After that, the new DSLAMs came with screw down terminals with clearly marked positive and negative instead of the plug they used before. I was also told there was a protection circuit to protect against reverse polarity but we didn't test that.
I should also mention that after the second explosion, I won the argument with the company owner where I had said from the beginning that the professional installers who did the fuse panel should also connect the equipment. I never touched 48v DC again after that.
Re: (Score:2)
My worst story is a structural fire during which the fire crew pointed their hose at the room that held a rack. There was an IBM X-Series 346 that was soaked, but started up after it dried out. Data was readable, we copied it and threw away the server. Tough bird those older IBMs.
Re: (Score:2)