Forgot your password?
typodupeerror
Hardware IT

Managing Servers In the Frigid Cold 122

Posted by samzenpus
from the chill-out dept.
1sockchuck writes "Some data centers are kept as chilly as meat lockers. But IT operations in colder regions face challenges in managing conditions — hence Facebook's to use environmentally controlled trucks to make deliveries to its new data center in Sweden, which is located on the edge of the Arctic Circle. The problem is the temperature change in transporting gear. 'A rapid rate of change (in temperature) can create condensation on the electronics, and that's no good,' said Facebook's Frank Frankovsky."
This discussion has been archived. No new comments can be posted.

Managing Servers In the Frigid Cold

Comments Filter:
  • by onyxruby (118189) <onyxruby@NOspam.comcast.net> on Monday August 06, 2012 @03:47PM (#40898325)

    This isn't anything new, anytime you take something from the extreme cold and bring it inside you risk condensation. This is usually dealt with by simply letting something sit at room temperature for several hours before powering it on.

    In the middle of January if you take a freezing cold delivery and power it on right away and fry your new (XXXXXX) you deserve to void your warranty. There is no excuse for stupidity. Why is this on slashdot as news?

    • by Pieroxy (222434) on Monday August 06, 2012 @03:54PM (#40898411) Homepage

      There is no excuse for stupidity. Why is this on slashdot as news?

      Because most slashdotters aren't from extremely cold areas. What seems like obvious to you isn't even considered by most, just because most aren't subject to the same conditions as you are.

      Let me ask you another question in return. I think I deserve an answer, since I answered yours: Why do you think everybody should know what you think is obvious?

      • by vlm (69642)

        Let me ask you another question in return. I think I deserve an answer, since I answered yours: Why do you think everybody should know what you think is obvious?

        Temp is a bell curve, what we do for a month every year is maybe one day every other year for you.

        • by vlm (69642) on Monday August 06, 2012 @04:12PM (#40898615)

          Oh I thought of another one. The problem has nothing to do with temperature. The problem is when the indoor/outdoor/dew points intersect which happens all the time, not just when its cold.

          One excruciatingly humid summer day I was hauling around a protocol analyzer worth about as much as my car, and it cold soaked in front of the car air conditioner duct cooling itself to 40 degrees or whatever the AC output is, then it was dripping condensed water as I carried it into the customer premises, an un-airconditioned factory floor. So I'm sitting there doing nothing and explaining to the customer how I have to do nothing, until the test set dries off because its too cold (customer VP looks out window at blue sky 110 degree day). Yes that was an unpleasant meeting.

      • by Baloroth (2370816)

        Have you ever seen a glass of ice water that showed condensation? Bingo, you should be able to figure out who powering on freezing cold electronics is a bad idea. I'm pretty sure everyone working at a data center is familiar with the idea of condensation, certainly anyone working in any situation where it would be an actual issue.

        • by Pieroxy (222434)

          I'm pretty sure everyone working at a data center is familiar with the idea of condensation

          Did I ever said otherwise? You actually reinforce my point: Most slashdotters DON'T work in a datacenter.

      • Re: (Score:2, Insightful)

        by onyxruby (118189)

        I can certainly understand that people from warmer climates won't understand how to drive during an ice storm or how to recover from a skid. These are things that come from experience with exposure to a certain climate.

        I will answer your question as I wasn't trolling. I think everyone should know consider this obvious because condensation is elementary physics. When you consider that I am on a technology site with a notable science influence it's the kind of thing I just expect that people would know.

      • Let me ask you another question in return. I think I deserve an answer, since I answered yours: Why do you think everybody should know what you think is obvious?

        3rd grade science class [kidzone.ws]

        • by Pieroxy (222434)

          Ok, my question was badly worded. Everyone knows about condensation. But then why would you expect most of us to think about it before powering a machine? I mean, unless we've tripped on that one once or twice?

          • Ok, my question was badly worded. Everyone knows about condensation. But then why would you expect most of us to think about it before powering a machine? I mean, unless we've tripped on that one once or twice?

            Logic.

            - It's common knowledge that powering on wet electronics is a great way to let out the magic smoke, right? Right.

            - It's also common knowledge that condensation forms on cold/warm items entering a warm/cold environment (respectively), right? Right.

            Therefore, knowing that electronics which are brought from a cold environment into a warm environment are going to form condensation, and that moisture is bad for powered electronics, the logical assumption is that plugging them in whilst still cover

            • by Pieroxy (222434)

              Your view of the world is amazing. I wish you luck with your endeavors.

              • Your view of the world is amazing. I wish you luck with your endeavors.

                Huh, never had it called that before...

                I appreciate your consideration, and return the good wishes.

    • by aepurniet (995777)
      not everyone lives in that type of climate, i havent RTFA, but i assume there are some more challenges that people dont usually think about. seems like it would be interesting if your into data centers.
      • by gl4ss (559668) on Monday August 06, 2012 @04:09PM (#40898575) Homepage Journal

        well, it's not particularly interesting article I'm afraid, the interesting part is that they mix incoming fresh air with already circulated heated air instead of having isolated heat exchanger arrangement, other than that it seems like a fairly traditional datacenter - no ssd devices dipped in epoxy sitting outside in ice or crazy stuff like that.

        "woot heated trucks".. well duh, not everything likes to be frozen. ever had partially frozen milk on school lunch? it sucks and we walked to school. both ways. or bicycled(on ice). or used kicksleds(if there wasn't sand on the route).

    • by vlm (69642) on Monday August 06, 2012 @03:55PM (#40898421)

      Living slightly to the east, yet just as cold in winter, the strategy is to leave the gear sealed in the box while you prep the racks and wiring and gather tools. Its really not that complicated.
      You don't have to wait until the gear reaches room temperature, merely gets above the interior air dew point, which I assure you is very low in the winter.

      • by afidel (530433)

        Why would the dew point be any different during the winter, don't you have humidity control in your datacenter?

        • by vlm (69642)

          It never works that well. I've seen the logs/graphs. Dew point skirts the bottom of minimal acceptable in winter (around 40 F, from memory), and skirts the top of maximum acceptable in summer (around 60 F, from memory). I suppose it depends on your center. I'm thinking specifically of a private couple acre financial services DC in the upper midwest, although the telecom data centers I've been in are about the same.

        • by petermgreen (876956) <plugwash@p10MENCKENlink.net minus author> on Monday August 06, 2012 @04:33PM (#40898819) Homepage

          Even if you do have the ability to artificially raise or lower the humidity that doesn't mean it makes sense to keep it at the same level all year round.

          Most electronics is specified for quite a wide range of relative humidity. Usually 5% to 95% or so.

          In winter you want low relative humidity to reduce the risk of condensation on stuff brought in from outside (yeah you try to seal stuff and let it warm up before unwrapping but mistakes and emergencies happen). It's also cheap to achieve low relative humidity due to the low outside air temperature (for a given absoloute humidity relative humidity goes down as temperature goes up).

          In summer humidity doesn't matter so much since stuff brought in from outside will be warm. It's also likely to be more expensive to achieve low relative humidity since it involves active dehumidification (which is achieved by cooling the air to the point where the water condenses out)

          • by afidel (530433) on Monday August 06, 2012 @05:10PM (#40899217)

            Low RH is bad because you get static buildup. Sure we've got anti-static wax on the floor and all the cages are grounded, but I still don't want to risk frying a computer because I couldn't keep RH in the right range. Also low RH is easy to achieve since CRAC units due it by their nature =) Even in the middle of summer you have to run humidifiers.

          • by Shatrat (855151)

            Just because the equipment says 5%-95% doesn't mean it's a good idea. That same equipment is probably also rated to 50c+ temperature.
            We keep all our facilities around 40% rh. Too high promotes condensation and too low promotes static discharge.
            All decent HVAC units can control temperature by condensing or evaporating moisture.

    • by prakslash (681585) on Monday August 06, 2012 @03:56PM (#40898429)
      Why is this on slashdot as news?

      So you can get a taste of the luxuries you can afford after a 100 billion dollar IPO. Why wait a few hours before powering up you equipment when you can transfer it using expensive, climate-controlled trucks. At Facebook, even lifeless plastic and metal rides in style on the gravy train.
      • by sumdumass (711423)

        I'm not sure how much more expensive the climate controlled trucks are. I think it is probably trivial.

        I used to run a refer unit (yes, refrigeration units in trucks will heat too) and we were always picking up coiled steel and transporting it on the refer unit to control the climate until it reaches it's destination. The difference between out runs and flat beds making the same runs was less then 5 cents per mile to cover the diesel to run the refer units..

    • by Joce640k (829181)

      Have they not heard of an invention called "plastic bags"?

      I hear they do wonders against condensation.

    • by afidel (530433)

      Yep, we never put the tapes from the delivery truck directly into the tape library during the winter for exactly this reason. The tapes get loaded on the truck early in the morning when it's often in the teens outside, taking them from that to the 80 degree datacenter is bad enough, putting them in the 100 degree environment of the tape library without acclimating them would be foolish.

    • by timeOday (582209)

      This is usually dealt with by simply letting something sit at room temperature for several hours before powering it on... There is no excuse for stupidity. Why is this on slashdot as news?

      Maybe because what you said is NOT the same as what Facebook is doing? If simply letting the servers warm up gradually at the destination works fine, why are the spending extra money on heated delivery trucks?

      • Because they dont understand common sense solutions to simple problems, and would rather throw money at the problem?

    • by oodaloop (1229816) on Monday August 06, 2012 @04:28PM (#40898773)
      In the Marine Corps, I had some cold weather training before a deployment to Norway. We were instructed to leave our rifles outside of our tents. Otherwise, they would accumulate condensation inside the barrels, which would then freeze when you walk outside. Lots of fun stuff like that.
    • by imp7 (714746)
      If I had mod points I would be giving them to you. I'm from MN too and it's common knowledge to seasoned sysadmin's here. I find it sad that facebook has to waste money and energy.
    • This isn't anything new, anytime you take something from the extreme cold and bring it inside you risk condensation. This is usually dealt with by simply letting something sit at room temperature for several hours before powering it on.

      True. But what you're forgetting (generously assuming you knew it in the first place) is that condensation isn't the only issue. Servers are made of a variety of materials - all of which expand and contract with temperature at different rates. Extreme cold can actually phy

      • by onyxruby (118189)

        Very true. I had a dell repair tech keep my laptop (unauthorized) for repair in his car over the weekend once in a cold snap in the winter. He froze and shattered the screen. I ended up with a new laptop from Dell when all was said and done.

        Early in my career I worked for Polaris and used to arrange deliveries of computers to places in the Arctic circle. We took a number of precautions keep the equipment from getting destroyed by the extreme cold. We never kept things in a heated container though and I was

        • by AK Marc (707885)

          Early in my career I worked for Polaris and used to arrange deliveries of computers to places in the Arctic circle. We took a number of precautions keep the equipment from getting destroyed by the extreme cold. We never kept things in a heated container though and I was shipping computers to places like Nome Alaska. We never shipped anything in a heated truck though.

          Nome isn't in the Arctic circle. Call us when you go someplace interesting, like going on a Polaris from Faribanks to Anaktuvuk Pass with a server strapped on the back (no, I haven't. Alaska Air has multiple destinations above the arctic circle, and I've been to all of them, on a plane, I try to not go outside when it's cold).

          My favorite was discussing wind chill with regards to electronic equipment, or finding anything rated to work below -40, as most goes to -40, so anything rated for colder requires

          • by onyxruby (118189)

            We sent equipment to anyplace that had a Polaris dealership. Nome was a particular dealer that came to mind as I had to deal with them more than once. We certainly had dealerships in the Arctic circle as we were the effective equivalent of the local car dealership up there (and Canada).

            I'm not arguing the extreme cold is extremely hard on the equipment. Facebook isn't doing the oilfield type of conditions, they are shipping equipment down a highway to a heated data center which is very different from the c

            • by AK Marc (707885)

              We certainly had dealerships in the Arctic circle as we were the effective equivalent of the local car dealership up there (and Canada).

              Odd, a search for Polaris dealerships didn't show any above the arctic circle.

              https://maps.google.co.nz/maps?hl=en&client=opera&channel=suggest&ie=UTF-8&q=polaris++dealers+alaska&fb=1&gl=nz&hq=polaris+dealers&hnear=0x5400df9cc0aec01b:0xbcdb5e27a98adb35,Alaska,+USA&ei=lmwgUNruK4iIrAfB_oGgCw&ved=0CLcBELYD [google.co.nz]

              But then, places like AC sell ATVs without being "dealers", as far as I can tell. Though I looked them up and didn't see any Polaris there. Many would buy them

              • by onyxruby (118189)

                Your not looking very hard. I'll start with a little town called Barrow [google.com] Alaska. I think we can both agree that it is in the Arctic Circle. They also have a Polaris dealership.

                Eskimos Inc Polaris
                PO Box 1273
                Barrow , AK 99723
                907-852-8000

                If you really want you can look things up directly on Polaris's website [polaris.com]. As I said I worked there, I dealt with the dealerships for a couple of years. They also have dealerships in arctic circle in Canada. They have dealerships that operate under everything from Harley Davidson

                • by AK Marc (707885)
                  They didn't show up on a google search, and on the Polaris website, searching for Barrow gets a result in ME, and Barrow , AK gets no results. And yes, I didn't look too hard. I just gave it a little look and didn't see anything popping up on Google and such, and I did go to Polaris's web site, but the flash version required didn't match what I have installed, though the link you gave went to a page that embedded a Google map frame, which worked better. Closest I've been to the arctic circle in Canada is
                  • by onyxruby (118189)

                    Their website is terrible, it always has been. Many dealerships often won't even have "Polaris" in their name which throws things off even more. Small motor dealers often carry multiple lines and it's just the same for Honda or the other manufactures.

                    I just happened to recall the northern Alaska / Canadian dealerships in particular with all the stuff I had to go through compared to a typical place. For many of those dealerships at the time it was literally the first time they had ever seen a computer. Needl

                    • by AK Marc (707885)
                      I replaced a failed tape drive in a warehouse in Alaska, and I pointed out to the shop guy why it failed. There was over one inch of dust collected in the bottom of the case. The silt-like dust/dirt would get into anything and everything.

                      I'll never "get" the Harley. It's slower and more expensive than almost everything else out there, but then, from what I can tell, most of the people who buy Corvettes are old fat white guys who never even hit the speed limit.
    • by Anonymous Coward

      This isn't anything new, anytime you take something from the extreme cold and bring it inside you risk condensation. This is usually dealt with by simply letting something sit at room temperature for several hours before powering it on.

      In the middle of January if you take a freezing cold delivery and power it on right away and fry your new (XXXXXX) you deserve to void your warranty. There is no excuse for stupidity. Why is this on slashdot as news?

      Not comparable. Bicycles and cars are custom made for the Swedish market (i.e. you have to custom make a lot of parts and replace all nuts and bolts), because the humidity fluctuations (and to a lesser degree temperature fluctuations) make cars and bicycles made for "normal" markets corrode really fast. E.g. a bicycle made for any other European market would only survive a few months use in Sweden, it would corrode the fastest during spring and autumn, or mild winters, in Southern Sweden, when it is not ex

      • by Firethorn (177587)

        Also, most electronic circuits freeze to death within a few hour by -5C; Luleå stay below that temperature most parts of the year. Any remotely advanced electronic circuits is dead by -20 - -30C, unless isolated from the cold, common temperatures during winter in N. Sweden.

        Fairbanks, AK here. We get plenty of -30, and we find that most electronics that survive it just fine. Vehicles are plugged in more to keep the lubricants, coolant fluid within their operating range, and keep the battery a bit warmer so it can start stuff. Even servers can get that cold when not operating; just let them get up to operating temperature before plugging them in.

    • by AK Marc (707885)
      Move faster than condensation. The cold places don't have as much moisture inside (cold air heated is dry), so if you plug in cold equipment quickly, it'll warm up before it gets condensed on.

      Also, if it's that cold outside, then use filtered outside air for room colling, keep the room at outside air temp, and you won't have an issue.
    • by grcumb (781340)

      This isn't anything new, anytime you take something from the extreme cold and bring it inside you risk condensation.

      Having worked at a commercial ISP in the Arctic [wired.com] for 3 years, I can tell you that it's a little different from Minnesota. A few hours isn't nearly enough time to let them sit. Our standard was 24-48 hours in the room the equipment was going to occupy before we'd attach cables and power on. Radical heating and cooling also meant that we'd re-seat RAM, NICs and other cards before booting as a matter of course.

      And it's not just cold up there, it's also perfectly arid. Things get dry in southern Canada and the n

    • by CAIMLAS (41445)

      I'm from South Dakota, but just got back last month from a year's internment in California - you know, where most tech companies and many data centers are. It's got a similar climate to Texas, supposedly, where there is also a lot of tech. :P

      It gets down to -20F very normally out here during the winter. With temperatures that low, there is very little moisture in the air indoors, either. So there isn't going to be all that much condensation. The best thing to do is leave the server in the box for several ho

  • by Isbjorn (755227) on Monday August 06, 2012 @03:53PM (#40898387)
    This is nothing--years ago I deployed PCs at Alaskan oilfield installations. Extreme cold makes everything brittle, kept having problems with things like cracked motherboards, just from setting the PC on a desk.
    • by vlm (69642) on Monday August 06, 2012 @04:00PM (#40898469)

      Cold drive bearings don't want to spin up / SMART fail from drive motor overcurrent.
      Happens to cooling fans too. Fan can't spin so equipment overheats.
      I've never knowingly had a voice coil bearing seize up, which is interesting because its probably the lowest power actuator in the system yet probably the highest precision / smallest tolerances.

    • Circuits must be specifically designed and qualified for low temperature operation. Common low-cost ceramic capacitor dielectrics (Z5U) are rated only to +15C and are useless by 0C. Y5P/Y5V are rated to -30C. X5R / X7R will get you to -55C. Aluminum electrolytics are useless at low temperature; tantalum is required.
  • Suck it up, pussies. (Score:2, Interesting)

    by gl4ss (559668)

    Sincerely, Finland.

    Been pretty shitty weather all summer here. Oh and I remember vividly when we were kids and we were coming down from a family trip in the winter, we couldn't play the new games we had bought before the next morning since the 8mhz bugger wouldn't boot until the "computer room"(porch thingy, badly insulated) heated up. And many many times we were playing games with our winter jackets on, maybe our parents were trying to discourage from being such nerds but they failed.

    • by sconeu (64226)

      You're missing the "now get off of my lawn" part of your post.

      • by gl4ss (559668)

        You're missing the "now get off of my lawn" part of your post.

        I don't have a lawn :(.. but during winter, I'm very happy to live in an apartment building and not a house with my own driveway. I hate shoveling ice.

  • by Anonymous Coward

    Why would you need an "environmentally controlled truck"? What about just using some basic insulation? Shipping in cardboard boxes would slow the temperature change near the electronics enough to prevent condensation.

    • I agree. You don't need environmentally controlled trucks for packaged basic electronics.
      • by Firethorn (177587)

        Basic insulation like cardboard isn't going to cover it when you're looking at a 100F temperature difference between your server room and the outside, and you're possibly looking at a week's worth of shipping time.

        A 'temperature controlled' truck doesn't even have to turn on said features until like 24-48 hours before delivery, but it's still useful.

        I unloaded trucks in the wintertime in Nebraska. Nowhere near the arctic circle, but the cold seeped into said vehicles. Even with a hot air blower in the sto

  • Back in the day, I had to go to my data center when it was around 100 degrees out side so I was of course in shorts, t-shirt and sandals. I was there for 18 hours. Temperature inside was like 50 degrees. Yeah, that doesn't seem cold, but after 18 hours I felt like I had hypothermia.

    • And it was uphill both ways to get there, right?
      • It was twice as far uphill on the way, and three times as far on the way back.

        Seriously though, it wasn't so much the temperature, it was getting blasted constantly the by the air conditioners. Wind chill, I guess.

      • by mooingyak (720677)

        this:

        uphill both ways

        and this:

        to get there

        don't quite mesh.

  • by Anonymous Coward

    http://www.lpsind.com/SilicaGelDesiccant.htm

  • by N1zaam (2598947) on Monday August 06, 2012 @04:15PM (#40898643)
    A few years back while doing Tier 2 level support for a major Canadian telco, I started seeing overheating alarms from some Nokia DSLAM's. The odd thing was that it was -40C outside at the time. It turns out the fan's on these DSLAM's froze solid and the devices thought they where overheating and throwing alams left right and centre. We had to put a tarp over them with a heater during the winter to make sure they kept going.
    • Re: (Score:3, Informative)

      by mooingyak (720677)

      The odd thing was that it was -40C

      Random tidbit: at -40, you can just say -40. Kelvin doesn't go negative, and -40F == -40C. Unless you're using some other scale, but that should cover most of the cases.

  • by Alkonaut (604183) on Monday August 06, 2012 @04:39PM (#40898879)

    Luleå has to have one of the most extreme temperature ranges anywhere. Summer temp is quite consistently 15-20C with occasional peaks of 30C and winter temp is zero to 40C below. So the range is nearly 90C (130F)! This of course seasonal variation and not "rapid change" so data centers should not be affected by this. The fastest changes there are probably in winter when the temperature in rare cases can go from -40 (and zero humidity) to zero (and damp) in a day or two. That kind of change, especially the other way round, could mean trouble (condensation in air in/outlets etc.)

    In fact, if google just wanted cold/dry climate, there has to be better locations. Northern sweden is mild, and has quite warm summers. Arctic inland climate further from the gulf stream atlantic would be more logical. Border between Russia and Finland for example. But there are probably logistical reasons (huge cargo airport, good port, good roads, railroads, lots of good technical people, ridiculous backbone connection) that placed the datacenter there.

    • by Yvan256 (722131)

      Canada has similar ranges. We got above 35C a few days ago and it went below -30C last winter.

      Also, from -40C to 30C is a range of 70, not 90.

      • by AK Marc (707885)
        What, is this a contest? Go live in Fairbanks Alaska. +40C to -60C over a year.
        • by mennucc1 (568756)
          slow down, cowboy! According to wikipedia, in Fairbanks the average is +22C for highs in July and -28C for lows in January; the record temperatures in Fairbanks are high +37C and low -54C; but these records were not set in the same year; so I doubt that Fairbanks gets "+40C to -60C over a year". An even harsher climate, BTW, is found in Yakutsk: the average is +25C for highs in July and -41C for lows in January: this beats Fairbanks by 6C. The records are even more impressive: +38C for high , -64C
    • by afidel (530433)

      That's not that bad, here in NE Ohio we've had from 45C to -40C at the extremes.

    • summer storms can drop the temperature 10-30F in 1 hour or less.

    • by pipatron (966506)
      Not to mention good access to electricity, and a geologically stable area.
    • Naah, now you're pushing it. -30 is the usual minimum range for the whole of lappland, maybe -35 when it's really cold. But yeah, the summers are hot, you can swim outside in the rivers if you really want to. As for the reasons for the location of the datacenter, you're forgetting Luleå technical university.
  • Plastic bags. Look into it. You can even reuse them if you're so anal-retentively inclined.

  • Then it would have been well-known in that area already. I have been living in that area earlier and the issue of low temperatures and condensation was not one of the major concerns.

    What people tends to forget is that when the outdoor temperature goes down the relative humidity indoors also drops considerably and that means that the condensation issues aren't that big. And the most sensitive parts are the hard disks, just wait to unpack them from the ESD bag until they have been up to room temperature. Same

"Now this is a totally brain damaged algorithm. Gag me with a smurfette." -- P. Buhr, Computer Science 354

Working...