Forgot your password?
typodupeerror
Bug Hardware

Server Room Smells Can Be an Early Warning 154

Posted by timothy
from the ever-smelled-a-starter-motor-burn-out? dept.
Barence writes "As embarrassing as it may seem, an eggy smell in a server room needn't mean broaching the delicate subject of hygiene with a colleague. It can actually be a signal that something is about to go wrong with your server setup, as this consultant discovered after days of assuming questionable personal habits were to blame. The culprit? An expiring UPS device, sending out its own unique warning signal."
This discussion has been archived. No new comments can be posted.

Server Room Smells Can Be an Early Warning

Comments Filter:
  • by Anonymous Coward on Sunday March 21, 2010 @05:05AM (#31556008)
    Amazing how many dying UPS devices must be hidden in my boss's office.
  • But.... (Score:3, Funny)

    by santax (1541065) on Sunday March 21, 2010 @05:10AM (#31556040)
    Does this mean I can use my father-in-law as a UPS?
  • by voodoo cheesecake (1071228) on Sunday March 21, 2010 @05:11AM (#31556042)
    Sulfur Dioxide. Ventilate, replace or recondition battery. If the egg smell is strong and you quit smelling it, that's olifactory fatigue and lethal levels of the gas exist.
  • An alternative to blaming the dog. "Wasn't me, honey, it was the computer."

    • by iggymanz (596061)

      Reminds me of a story.

      Harvey finally was asked to dinner at his girlfriend Wendy Jone's house, to meet her parents for the first time, and hopefully make a good impression.

      Wendy's mom had made sauerkraut and sausage, and Harvey had to pretend to like it more than he did. So Mrs. Jones had given him seconds and even a third large helping.

      Later, as everyone sat and talked in the living room, Harvey could feel his bowels bloating with an enormous volume of gas. But Mrs. Jones was talking about the last days

  • Two comments...

    1. Windows in the server room?
    2. No-one noticed the UPS with all its error lights on?

    • by Anonymous Coward on Sunday March 21, 2010 @06:08AM (#31556262)

      "Windows in the server room?"

      You'd be surprised how many servers still run XP professional, or 2003

    • by osu-neko (2604)

      2. No-one noticed the UPS with all its error lights on?

      It's that SOP for a UPS? ;)

      On a more serious note, is there a more unreliable piece of equipment in any office? I swear I've seen more bad UPS's than good ones, and I'm not talking about ones that have been sitting around for years -- I've seen more UPS's that did not function properly three months out of the box than ones that did. Granted, I'm talking the cheapo kind you can get at Best Buy or the like, but still, they ought to work at least half the time, no?

      • Argh. Risk and cost assessments say that the closer you get to the outlet, the better your equipment should be. It's like people with $1000+ PCs who put in a $20 PSU. You're better off getting a brand name low end device which has been recently decommissioned.

        • $200-300 for a good UPS. Your price range was off, pretty much anything under $200 drops to 15min in 6mos and dies in a year. If you want a battery that isn't fundamentally a fire hazard, you can add another $600-$700.

          Then again the best batteries we have in our data center (admittedly fairly small) are about 12yrs old, work like dogs and only fail with a stinky bloat and/or random diagnostic jabber.

          • My price range wasn't anything - I was comparing with people who buy a $20 /PSU/, and was indicating that that was a /bad/ idea ;-).

            Even my current home APC UPS, which was approaching $200 new and bought second-hand for half that, has been sitting there for a decade. It still gives me more time than I ever need during power problems, i.e. about 15 minutes, as I'm in an area which gets an outage of more than a few seconds no more than twice a decade.

    • by Moraelin (679338) on Sunday March 21, 2010 @07:35AM (#31556582) Journal

      If they're like some of the IT departments I've seen, they might be working by some rule from upper management that they need to justify their existence by writing internal invoices for everything they do. It tends to result in them doing nothing until you tell them to, so they can bill you for it. The UPS could have not only the error lights on, but a binking "RED ALLERT" sign and the accompanying acoustic blare, and verily be on fire and billowing smoke, and nobody would touch it until you fill the proper form requesting them to put it out.

      Because, yes, that's another thing I've noticed that a lot of departments love, IT including: inventing bureaucracy and paperwork to discourage and delay actually having anything to do. You may need to fill in a 5 page form and draw powerpoint diagrams as to why you want the UPS doused and what are the architecture implications of that. And if you're unlucky a few meetings too, to convince some Mordac The Information Services Preventer why he should move his ass and turn that UPS off, and why his suggested workarounds (in which he'd not have to do anything) aren't quite solving the problem.

      • Why does the UPS not have a fail safe that kills it when the battery goes bad to stop a fire?

        • by Reziac (43301) *

          That's a damned good question. Is there a UPS engineer in the house??

        • by LWolenczak (10527)

          You can test lead-acid batteries by measuring the resistance between the terminals. Most upses only look for voltage from the battery. This means that the battery casing can split open, leak acid, before the batter grounds out or goes kaboom, or whatnot. I had one battery in a 64 battery cabinet leak out and ground out the batteries.... The bloody APC Silcon (now discontinued) ups didn't throw an alarm at all, all we noticed that there were no lights all of a sudden when the power flickered. It was app

        • Well, the UPS part was more like satire. But I've seen the attitude I've described about everything else. Hard drive space, database configuration, MQ queues, you name it. There are some departments out there who'll let you drive against the wall without even knowing, because they monitor nothing, do nothing except what you explicitly request so they can bill you (e.g., DBAs who require that the _programmers_ tell _them_ what tuning parameters to set for the database), created a lot of unneeded bureaucracy,

        • by sjames (1099)

          Perhaps it does, but it failed. What it needs is a device that shuts down the UPS if the device that shuts down the UPS when the battery fails fails.

    • by jbengt (874751)

      1. Windows in the server room?
      2. No-one noticed the UPS with all its error lights on?

      Those were my two main thoughts about TFA, exactly.

    • Windows in the server room is a HUGE security risk, and generally a big no-no. You need to have proper ventilation and cooling, but usually the server room is windowless and in the middle of the building, not on the outside walls. What is the point of having "secure" server room if all someone needs to do is break a window to get in?
      • If the server room is in the 4th floor of a 8 floor building, then breaking in may be difficult. Also, the windows probably have security sensors.

      • by xaxa (988988)

        Windows in the server room is a HUGE security risk, and generally a big no-no. You need to have proper ventilation and cooling, but usually the server room is windowless and in the middle of the building, not on the outside walls. What is the point of having "secure" server room if all someone needs to do is break a window to get in?

        0) Windows on the Nth floor aren't a security risk, with a high enough N.

        1) Perhaps they have an old building which has windows in every room. The server room where I work is in a building possibly older than your country (I can't find the exact date, it's close).

        2) Perhaps there isn't an information security issue. There are bars on the windows in the aforementioned server room, but all the interesting data on the servers is available on the web (or on DVD if you ask nicely), so the security is only to avo

        • If your servers don't have any valuable information on them, they don't need their own separate secure room. If all the info on your server can be found on a DVD or the web, theft of said server really wouldn't inconvenience you beyond the monetary cost of replacing the server.
  • This is funny, I routinely smell my servers and my UPS at the fans where the air come out of them to make sure nothing overheats but I never thought about mentioning that to anybody ;-))

    hehe...

    • by Runaway1956 (1322357) on Sunday March 21, 2010 @06:16AM (#31556304) Homepage Journal

      Uhhhmmm - it isn't just computers. If I notice an odd smell when I walk through the plant, I investigate. Our plant makes plastic products, and 2/3 of the time, the odd smell is just overheated plastic. But, the other 1/3 finds a problem of one sort, or another. Overheating oils are bad news, overheating capacitors are more bad news - actually, ANYTHING hot enough to give off an odor is bad news. Three weeks ago, we had a machine that was kicking our asses - the mold wouldn't open either manually, or in automatic. 4 of us went over that machine from one end to the other, multiple times. Ohmeters and voltmeters said that everything was just fine, believe it or not. Finally, I caught a whiff of something funky, opened up a solenoid from which the odor seemed to be coming, and found that half of the windings were burnt and shorting.

      The sense of smell is a valuable tool in troubleshooting and maintenance, unless you ignore it.

      • by TheLink (130905)
        Maybe someone should make a semiautomated "factory" that breeds and trains rats for such scent-detection purposes :).

        Rats have a very good sense of smell... And they're portable, put them in an unsealed cage, and have them transported around the factory. Or pass them air samples from various places.
    • by X0563511 (793323) on Sunday March 21, 2010 @11:17AM (#31557738) Homepage Journal

      Far too many people rely on performance metrics and alarms. You're one of the ones who actually pays some attention :P

      Any time you enter the DC you should take stock:

      1. What do you hear? Perhaps an alarm through all the server noise? Unusually loud fans/ACUs? Anything unusually quiet? Other noises? (I 'predicted' an ACU failure because I heard the fan belt rubbing on something lightly shizz-shizz-shizz-shizz-shizz...)
      2. What do you smell? This article basically points this out. Could be leaking ACU coolant. Batteries dying. Burning server. Overloaded circuit, etc.
      3. What do you see? Yea, stupid I know but - does that corner of the room appear slightly dimmer? Better go check it out, a rack might be down and you haven't noticed yet.
      4. What do you feel? Vibrations through the floor? Could be an ACU about to pop a fan belt or blow a compressor.
      5. What do you feel further? Unusually dry or humid air? Temperatures etc.

      In short, you should be using every sense except taste and direct tactile feel. Anything shorter and you just aren't paying full attention.

      • Re: (Score:3, Insightful)

        by RMingin (985478)

        Sense of touch can be valuable too. You can get sub-audible vibration readings by touching a case, and touch is more sensitive to small amounts of temperature change than other senses. Likewise it can be a really exciting way to check for failing/floating ground.

      • by ls671 (1122017)

        > you should be using every sense except taste and
        > direct tactile feel

        I know a car mechanic who likes to taste a car motor oil. He pretends he can tell if something is wrong with the engine that way.

        My guess is that he does it only to freak people out when they look at him tasting oil but who knows ? ;-)

  • I remember some old AMD slot style CPU's would smell like waffles when they burned out. Good times.
    • by unitron (5733)

      With the right heat sink the old Socket 4 original Pentiums could serve as waffle irons. : - )

  • Can be? (Score:5, Funny)

    by bertok (226922) on Sunday March 21, 2010 @05:50AM (#31556184)

    "Can be an early warning?"

    "CAN be?"

    Like all IT administrators who've actually worked with server hardware, I have a heightened sense of smell, but only specifically for the smell of burning plastic. It's not a mere warning, it's an instant alarm that'll have every IT person in the room sniffing the power supplies.

    We IT people, we're like bloodhounds or something. I can smell burned plastic from across the street. I've been set off by welders at a car mechanic a block away. I've been set off by an invisibly tiny bit of cheese someone dropped into a toaster oven once... three floors down from the server room. Had me in a right panic.

    IT is all fun and games until the servers literally melt into slag. There's no repair CD for that -- and we all know that the backup tapes, while wonderful for backing up, aren't so good at the actual restoring bit. That's why they're called backup tapes, not restore tapes, see?

    • Re: (Score:3, Informative)

      by dreamchaser (49529)

      Amusing post, but if your backup tapes are not reliable to restore then you're doing it wrong. I know you were being a bit tongue in cheek, and yes I've seen many cases where backup tapes were next to useless. In each case one could trace that to user error on the part of an administrator, often the person who setup the backup.

      Yes smell is an important warning tool in the data center. This article isn't even really news, or at least shouldn't be to anyone with more than a little experience.

      • by sjames (1099)

        It's just the nature of backup tapes in general. No backup tape is to be considered 100% reliable until it has been used successfully to restore a system. Even then, since it saw some wear in the process, the certainty of getting one more restore out of it is slightly less than 100%.

        • Oh yes, that I can agree with, but I've found that in many, many cases it is bad methodology that makes backups fail rather than bad tape.

    • by hey! (33014)

      Like all IT administrators who've actually worked with server hardware, I have a heightened sense of smell, but only specifically for the smell of burning plastic. It's not a mere warning, it's an instant alarm that'll have every IT person in the room sniffing the power supplies.

      Great! Now all we need is a way to turn that mutation into some kind of reproductive advantage. Or maybe we should segregate sever admins into a separate breeding population, and after a few dozen generations we'll have a new subspecies: H. sapiens resinanasus.

    • Re:Can be? (Score:4, Funny)

      by drinkypoo (153816) <martin.espinoza@gmail.com> on Sunday March 21, 2010 @10:11AM (#31557324) Homepage Journal

      I have a heightened sense of smell, but only specifically for the smell of burning plastic.[...] I've been set off by an invisibly tiny bit of cheese someone dropped into a toaster oven once...

      Must have been American "cheese", the only kind you can't tell apart from the wrapper.

  • > As the season has changed, for the first time in something like five months, I've opened a window in the server room I've been working in.

    Then the smell was gone, but there was this loud beeping sound. After doing another Google search I found out from a security consultant's blog that it was the break-in alarm on the window.

    Another crazy day at the office.

  • of things about to go wrong in a big way...

    • by umghhh (965931)
      The sound of sirens of coming fire brigade is also a good indicator.
    • I love the smell of capacitor in the morning, but if I smell the interior of an LED then I am out of there.

      I once blew the belly out of a 7413 by running it on 10V. I probably shouldn't have inhaled in that room after that.

    • by jgreco (1542031) on Sunday March 21, 2010 @07:01AM (#31556462)
      Computers run on smoke... when the smoke comes out, they stop running.
      • by CFD339 (795926)

        Not just any smoke. You're referring to the "Mysterious Blue Smoke" (MBS) that imbues all electronic wizardry with its magical properties.

        I've explained to many of my neighbors that often, once you let the MBS out of your gear it can be very difficult and expensive to ever make them operate again. It can be done, but it requires a high level soldering guild member, some odd looking parts -- often identifiable only by those who can read the arcane stripe patterns by which they are known. In addition, in

  • Server Room Smoke can be an Early warning, also.

    As embarrasing as it may seem, a cloud of smoke in the server room needn't mean broaching the delicate subject of kicking the marijuana habit with a colleague. It can actually be a signal that something is about to go wrong with your server setup.

    as this consultant discovered after days of assuming questionable personal habits were to blame. The culprit? A server whose board was in the process of being about to friggin explode!

  • I've never physically been inside a data center, but I'd have thought that the locales would have really good ventilation, that would simply shut close (or rely on gas weight and gravity) if the halon system or equivalent would need turning on. The ventilation is in fact so bad, there can be a gas buildup so severe you need to (according to posters above me) go in with hazmat gear?
    • by jgreco (1542031) on Sunday March 21, 2010 @08:02AM (#31556688)

      A triggered fire suppression system should trip the A/C interlock, shutting down ventilation and outside air (blowing air is stupid when FM-100 or whatever is used).

      Normally, however, air may well be circulated in a fairly tight closed loop. You do not want to inject outside air without a lot of treatment; filtering and humidity are very large concerns. Drawing in extremely moist, hot air from outside and bringing it into your air supply may well be a lot more challenging than simply recycling the existing clean warm air that already has a roughly correct humidity, for example, and then what happens when it's winter and suddenly the outside air is cold and super-dry? You suddenly have a different HVAC challenge.

    • He said 'server room', that's not necessarily a datacenter. It could just be a room with a few servers in. I'd assume it isn't a full scale datacenter. If you read the article, it appears to be a room with a rack of servers. There's a mention of the (lack of) ventilation system too. I know it's slashdot, but why not RTFA before asking questions answered in it?

      • by jgreco (1542031)

        Yes, he said server room, but the problems are similar.

        Regardless of the size of the room, any "computer room" with anything more than a modest amount of power consumption will generate a certain amount of heat. A modern server can easily consume hundreds of watts of power, and sticking two or maybe three on something like a SmartUPS 1400 is likely to put it near the ragged edge anyways. That alone is more than a quarter ton of cooling, and "opening the window" can rapidly become a problem to managing how

  • by Anonymous Coward on Sunday March 21, 2010 @06:32AM (#31556360)

    From the clasic BOFH :)

    "The admin gene," the PFY explains. "The ability to recognise things that users don't. A slight flicker of lighting, a whiff of hot component in the air, a fractional change in the pitch of a cooling fan - all of which the garden variety user misses in the headlong rush to read their email."

    http://www.theregister.co.uk/2008/07/04/bofh_2008_episode_24/

  • Yes, yes! You can smell a capacitor blown, which means a power supply is not long for this world .. ATLs with a stuck tape smell of phenol .. and you can smell 'warm' .. when some piece of equipment is beginning to overheat, it emits an odour which I can't describe otherwise. Transformers burning generate a nasty brown smell as well.

    I just noticed that I also suffer from synasthesia.

  • APC UPS's (Score:5, Informative)

    by jgreco (1542031) on Sunday March 21, 2010 @07:00AM (#31556458)
    APC UPS's have a tendency to cook their batteries as they get near the end of their lifetime. The results can be horrifying... bulging batteries, and if allowed to go on long enough, yes, even "sealed" lead acid batteries will rupture and you'll get the lovely sulfur smell.
    I recently pulled these APC batteries out of an APC Smart-UPS 1400, which had to be disassembled (including the removal/replacement of rivets) in order to get the batteries out.
    http://img221.imageshack.us/img221/171/imageyv.jpg [imageshack.us]
    • by ledow (319597)

      Me too... exactly the same with an APC UPS with the original batteries. I had to bend the hell out of the UPS casing in order to get the thing out, it was so bloated.

      • Re:APC UPS's (Score:5, Interesting)

        by jgreco (1542031) on Sunday March 21, 2010 @07:54AM (#31556648)

        Having worked with various standalone and both older and newer rackmount models, I would have to say that one of the most frustrating bits about APC's designs are that their battery compartments seem to be engineered to assume that batteries never bulge.

        On something like a standalone 1400, any bulging of the batteries tends to increase their width, jamming the batteries against the sides like Comic Book Guy trying to get through a turnstile.

        On the older 3U 1400's, the front of the chassis is cut-to-battery-size, and being constrained spacewise front-to-back, bulging usually happens widthwise across the batteries, forcing them bigger than the opening. This means that even if you remove the chassis top, you can't just get in there with a crowbar and forcibly eject them out the front chassis opening, you still need to unscrew the inner battery compartment partition (and maybe even remove it).

        This newer 2U 1400, though, wow, what a pain... the chassis is an artfully bent sheet of steel. If you cannot get the drawer to slide out of the battery compartment, the inner battery compartment partition is actually riveted to the chassis, so you have to drill out the rivets, and even after you get the partition removed, you only have one option for which battery to remove first, and they're taped down, so you have to use something like a crowbar to get them out, which is just vaguely scary because all the wiring for the battery tray surrounds the batteries in all the points where you need to exert force to break the tape adhesive.

        It's almost like they want it to be frustrating as all hell (maybe so you just replace the whole UPS?) while looking like they've tried to make it new and easy with their wonderful battery tray.

        • by X0563511 (793323)

          I think the idea is to encourage early and frequent battery changes.

        • by sjames (1099)

          That's the downside of the division of labor. If the designer was at all likely to ever have to get a bulging battery out of the compartment, it would be a lot easier.

          If the bean counters who want rivits and stupid plastic posts were at all likely to have to disassemble and reassemble something, you can bet it would have nice screws instead, and the designer would pitch in and make sure they were easily accessible. The total added cost would probably be lost in the noise.

    • by Mashiki (184564)

      All batteries cook when they're at their end of life. Doesn't matter who/what/where, because the system is being told that it's either insufficient charge/capacity/etc, and it goes into a full charge cycle. Which means rather then a normal discharge/charge cycle, you're always in a 105-115% charge. Happens with cars/motorcycles/trucks/etc as well. Had a buddy with a sealed battery(inside the car vented out), that went. The battery ruptured leaking acid through the inside storage panel. Very unpleasant

  • water cooling (Score:5, Insightful)

    by bananaendian (928499) on Sunday March 21, 2010 @08:03AM (#31556694) Homepage Journal

    This is why I prefer to build my new server rooms with individually cooled racks - each rack having its own AC-circulation - as well as using centralized water cooling for its efficiency and reliability. Circulating all your cooling air around the server room is simply a bad idea. When you have 1 kilometer of rack space on a single building floor, one source of contaminant, be it chemical or metal particles, will get into all the enclosures in the hall and cost you everything. And BTW UPS maintenance is something that modern IT management, especially outsourced services, have forgotten. Any veteran admin knows you need to estimate the end-of-life for their electronics AND replace them BEFORE they fail - just like AC-filters - If allow those to fail, they will have already done some damage! There's no "RAID" for burning electronics or blocked cooling air!

    • Any veteran admin

      Whoops, found your problem right there. Who wants to pay for an experienced administrator anymore?

      • Any veteran admin

        Whoops, found your problem right there. Who wants to pay for an experienced administrator anymore?

        These days, the cutoff for "Veteran" admins is the same as "veterans" in JRPGs.

        Max age: 19
        Qualification: Knows which end of a weapon (keyboard) goes into the bad guy(server)
        Had a few small, unimportant battles (Ran a linux server on his home cable connection)

  • Shhh (Score:3, Funny)

    by gadzook33 (740455) on Sunday March 21, 2010 @10:22AM (#31557372)
    Listen...do you smell something?
  • Anyone else notice that old HDDs smell strongly of body odor? It's like bacteria love the warmth and go through X generations until there's a constant "eww" smell to them. It seems to happen more with 10-15k RPM SCSI drives in 1U servers where they don't get as much ventilation, but I've noticed it with IDE and SATA in desktops too.

  • Service Status Last Check Duration Attempt Status Information
    UPS1 OK 03-21-2010 13:01:44 2d 1h 3s 1/3
    UPS2 OK 03-21-2010 13:01:44 5d 4h 1s 1/3 Hey it wasn't ME
    UPS3 OK 03-21-2010 13:01:44 2d 1h 3s 1/3
    UPS4 Gassy 03-21-2010 13:01:44 5d 4h 1s 1/3 ;-)

Never test for an error condition you don't know how to handle. -- Steinbach

Working...