Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Data Storage Hardware IT

A Closer Look At Immersion Cooling For the Data Center 213

Posted by timothy from the where's-the-vinegar dept.
1sockchuck writes "Want to save money on data center cooling? Tip your racks on their side, fill them with mineral oil, and submerge your servers. Austin startup Green Revoluton Cooling first profiled here) has a video demo of its immersion cooling solution, which it says can handle racks using up to 100kW of power. A photo gallery on the company web site shows some early installations."
This discussion has been archived. No new comments can be posted.

A Closer Look At Immersion Cooling For the Data Center More

A Closer Look At Immersion Cooling For the Data Center

Comments Filter:

  • The thing here is they are commercializing a cooling technique usually reserved for the hobbyist. I don't know about the energy saving claims, but their setup looks fairly organized. Interesting turn for a still niche cooling solution.

  • What about the hard drives? (Score:3)

    by Compaqt ( 1758360 ) on Wednesday April 13, 2011 @05:57AM (#35805652) Homepage

    Well, on the other hand, if they're supposed to be air-tight, I guess they're baby oil-tight, too.

    But there's got to be something or another that doesn't react well with mineral oil, right?

    I guess this means they save on fans, and the power to run fans. That's additional power and heat savings right there.

    OK, I've got it: what about the CD/DVD drives? Or is it all network IPL in data centers? I'm racking my brains trying to think of something this would mess up.

    • Re: (Score:2)

      by slim ( 1652 )

      TFA says the hard drives have to be sprayed with a coating, presumably to make the housing oil-tight as well as airtight.

      Blowing air around is a tremendously inefficient way of cooling, and this replaces that with pumping the oil through a heat exchanger / cooling tower.

      Things with unsealed moving parts obviously are vulnerable, but not everything needs to be submerged. If you really want a DVD drive, have it outside the oil.

      There may be some things that don't react well with mineral oil: avoid having these

      • TFA says the hard drives have to be sprayed with a coating, presumably to make the housing oil-tight as well as airtight.

        This should IMMEDIATELY ring alarm bells. Hard drives are NOT airtight. They have a filtered air hole. They would never get away with such flimsy construction on an airtight product.

        Plus does this system REALLY offer that much advantage over conventional "waterblocks" which keep the cooling fluid seperate from the electronics. I very much doubt it. The major heat generators in a PC are designed to pass out their heat by contact conduction anyway.

        • Re:What about the hard drives? (Score:4)

          by fuzzyfuzzyfungus ( 1223518 ) on Wednesday April 13, 2011 @06:20AM (#35805762) Journal
          I suspect that the main advantage over waterblocks is that your server vendor of choice doesn't have to cooperate, and you don't have to run so many small hoses, and can get away with the big huge tank. Now, if some sort of critical mass were achieved in terms of industry acceptance of waterblocks as a factory option(along with some sort of standardized fluid connector), I suspect that that would swiftly become a superior option(though, quite possibly not displacing fans entirely. A fair number of ancillary systems, too small to justify their own waterblocks, get very unhappy if they aren't getting any airflow, even if the CPUs are taken care of...)

        • The other thing that rang my alarm bells is the idiocy of submerging hard drives at all. You DON'T DO IT. It doesn't help, it only causes problems. They should either do what immersion gaming cases do and use long SATA cables to run the hard drives to a dry compartment on the outside of the case, or network-boot these servers.

          • Re: (Score:2)

            by slim ( 1652 )

            ... or use SSD.

            But if you're going to have HDD storage anywhere, that's going to be generating heat too. Possibly most of it, if you're doing non-CPU-intensive file serving. So the bean counters would probably *like* their fancy new cooling technology to work with HDDs.

            • Even if you want to use SSDs, immersion-cooling them is not worth it. The SSD is then at risk of succumbing to PCB softening like every other component, and for what? Do SSDs even get that hot?

              Network booting is probably the best solution here.

              • Re: (Score:2)

                by 1s44c ( 552956 )

                Network booting is probably the best solution here.

                Using SAN disks would work quite nicely too. Either way you still end up with disks somewhere that will need to be cooled outside the oil bath.

                Plus this setup uses at least 4 times the floor space of vertical racks.

                This is a neat idea, but I don't see it working in practise.

    • Re:What about the hard drives? (Score:5, Informative)

      by gclef ( 96311 ) on Wednesday April 13, 2011 @06:07AM (#35805706)

      Actually, hard drives are *not* supposed to be air tight. They intentionally allow airflow into the HD, but through a filter to keep dust out. If you want a drive that is airtight, it'll cost more.

      http://www.acsdata.com/how-a-hard-drive-works.htm [acsdata.com]

    • I've never tested to see if anything bad happens if you change this yourself; but hard drives are definitely not airtight. Pretty much all of them have one or more visible holes(usually on the top cover, often next to a "Do Not Cover" sticker). There is a fairly serious looking filter on the inside, covering the hole(some sort of fine carbonish powder sealed in what appears to be a teflon pouch); but that is to impede dust, rather than gases.

      Nothing stopping you from PXE booting(or iSCSI or fiber channel

      • Re:What about the hard drives? (Score:4, Interesting)

        by macs4all ( 973270 ) on Wednesday April 13, 2011 @07:15AM (#35806138)

        The only other issue I can imagine might crop up would be discovering the hard way that some polymer used in one of the system's components doesn't handle oil exposure well in the long term. I suspect that most are fine; but if the plasticizer used to soften the insulator coating on some important bundle of wires leaches out over 18 months in a warm oil bath, and the embrittled insulator cracks and shorts the next time you mess with it, the joke would be on you...

        Hard drives aren't the only thing designed with "vent holes".

        Every single electrolytic capacitor has a tiny vent hole (to keep them from acting like a mini fragmentation grenade if they develop an internal short circuit, etc.) Over time, with thermal cycling, the oil might get pumped in and out of the vent holes, thus degrading the electrolyte (guessing), and one fine day...

        And as you say, think of the insulation on the cables...

        • Capacitors do not "breathe" like you describe, otherwise humidity would get in and ruin them. Instead they have "vents" in the sense that part of the casing is weakened to rupture safely if the electrolyte starts to break down and build pressure, rather than have the whole can explode.

          The irony here is that the electrolyte in the "beer can" style caps is a mineral oil not completely unlike what they use as a coolant. The other option is to pay extra for non electrolytic capacitors on your equipment.

          As for t

          • The irony here is that the electrolyte in the "beer can" style caps is a mineral oil not completely unlike what they use as a coolant. The other option is to pay extra for non electrolytic capacitors on your equipment.

            Is there a list of manufacturers using solid caps? My Gigabyte board proclaims loudly on the boot screen JAPANESE solid capacitor... whoops. Looks like it's time to find a new supplier. Having sucked up a lungful of blown cap smoke in the past, I'm glad to see solids...

      • My understanding is that if you cover a hard drive's vent hole, unusual atmospheric pressures could cause the disk head to either crash into the platter or float too high over it.

  • I have to wonder at their claim that it works well with standard OEM gear. Even most cheap consumer shit monitors the speed of at least the CPU fan and tends to freak out if a fan that is supposed to be there is either absent or performing substantially below expected speed(and, given the relatively high stall current of these fans, burning a trace isn't totally out of the question, if the fan or fan controller isn't smart enough to give up after a short time...) Given that high-density servers and blades w

    • Re: (Score:2)

      by slim ( 1652 )

      It seems pretty trivial to replace the fans with mock fans, that either always reports an OK fan speed, or does something with the measured oil temperature.

      True enough about the horizontal mounting and the weight. I don't fancy dealing with a heavy unit dripping with baby oil -- but surely, since they have an installation, they've addressed these practicalities?

      • I imagine that some sort of pneumatic/hydraulic piston system(like the one used on many vertical-open car doors and the like) would be well within the realm of the possible, and offer a zero-apparent-weight slide-up ; but I can't see whether or not there is anything of the sort from the pictures provided.

        The one picture showing "easy serviceability" shows the operator having completely unracked a relatively small server. That doesn't scream 'easy' to me. Pull server up, hold with one hand, unclip a bunch

    • I have to wonder at their claim that it works well with standard OEM gear. Even most cheap consumer shit monitors the speed of at least the CPU fan and tends to freak out if a fan that is supposed to be there is either absent or performing substantially below expected speed

      Enter the BIOS (hit DEL during Power On Self Test), go into the Power or PC Health (depending on what BIOS you have). Alter the value of CPU FAN to "Not Monitored" or "Ignored". Hit F10 (or whatever yor key is) to save settings and reboot.

      SpeedFan (etc) will still give you a speed readout if a fan is connected, but your BIOS won't complain if one isn't.

      This procedure should be similar for UEFI based systems.

  • Paging Dr. Freeze (Score:5, Informative)

    by upside ( 574799 ) on Wednesday April 13, 2011 @06:06AM (#35805694) Journal

    1999 [slashdot.org] Have I been reading Slashdot that long?

  • The setup looks nice as it is, but having 42" racks laying on their backs never gives the same rack density than the same racks standing upright.
    • All you need is very low ceilings, allowing multiple tiers of racks in what would ordinarily be a 1-story room. You can then hire malnourished Dickensian urchins to scuttle, bent over in a permanent half-crouch, among the dripping coolant tanks, swapping drives and cards... Just remember to flog them frequently, to remind them of their place, or you'll find one cooking his fish-and-chips in the hot-coolant line and clogging the heat exchanger with crumbs.
    • Given that you can lay two of these racks back to back and then run them end to end, and that you can remove most of the regular AC equipment from your room, the amount of stuff you can get in your datacenter is the same.

    • Re: (Score:2)

      by Enry ( 630 )

      Normal racks are 15-20kW. If you can fit 100kW into a rack that uses up the floor space of 3 normal racks, you're still ahead of the curve in terms of power usage.

      Then again, I'm getting 4*14*12=672 cores in a single rack using less than 20kW. Unless they're using 150W CPUs, I have no idea how they need to cool 1/10 of a megawatt in a single rack.

    • Re: (Score:2)

      by LWATCDR ( 28044 )

      Why not just use water cooling. Water is a lot more efficient at removing heat than oil. Just install water blocks in place of the heat sinks. You wouldn't need to worry about sealing the contacts and pins which you would with oil. and if done correctly maintenance would be too bad.
      I believe that Cray used immersion cooling on their Cray-2 line using Fluorinert.

      • Have you tried fitting a water block in a blade lately? How about 2000 of them? :)

        • Re: (Score:2)

          by LWATCDR ( 28044 )

          But they are not using blades they are using 1Us. At least that is what it looks like. I find it odd that they are putting the HDs in the oil and in the server. To be as green as possible I would think that having the server boot from the network and having a SAN would be a better choice. And it would beat the daylights out of having to pull an HD out of an oil soaked server.
          How about you used heat tubes from the copper block on the chip connect to an easy to fill water block/ heat exchanger for a blade sys

  • Four years of college.... (Score:5, Funny)

    by Ahimoth ( 2029382 ) on Wednesday April 13, 2011 @06:20AM (#35805760)
    and I still get to go home smelling like fries!

  • Aha (Score:2)

    by jovius ( 974690 )

    So this is what happens when you drool too much.

  • When I experimented with mineral oil based cooling, the main issue I had were water droplets condensing on the surface of the cold mineral oil and then promptly sinking... towards the motherboard sitting in the bottom of the old aquarium I was using as a case. Of course there were solutions to this problem but it was a quick and dirty (you can take that dirty word quite literally) test back when I was a student, so we gave up on the idea pretty quickly.

    I wonder how they have managed to solved the condensati

    • Re: (Score:3)

      by Phs2501 ( 559902 )

      I wonder how they have managed to solved the condensation problem.

      They run their oil at 40C. If the dew point in your server room is that high you have other problems...

    • Re: (Score:2)

      by RingDev ( 879105 )

      If you had condensation (water) that was dense enough to sink in your oil, you were using the wrong type of oil.

      Mineral oil is way more dense than water. And condensation that occurs should sit at the top of the pool and never create an issue (and it should be pretty easy to skim off).

      -Rick

  • Okay, oil is more efficient than air. But the problems with this are plainly obvious when it comes to anything that falls in the area of maintenance and upgrades.

    I wonder what gasses can be used instead of oil? Something that wouldn't likely leave a residue? Substituting a gas for a liquid might reduce some efficiency, but you are still containing the unit completely and entirely. A lot of efficiency can be added merely through the act of containment. There must be some sort of gas that can be pumped i

    • I wonder what gasses can be used instead of oil?

      Refrigerated air might be best. Cool it before it passes through the system, so it can remove more heat. Of course you either pay for the energy to pre-cool the air, or you locate your data centre somewhere very cold. If thats not good enough you could look at supercharging. Compress the air around the system. That increases density and heat capacity.

  • Wait till the tsunami wipes out the generators.

    • Re: (Score:2)

      by Ksevio ( 865461 )
      Then at least when the nuclear servers meltdown, it won't be onto the units below.

    • If the tsunami wipes out the generators, there will be no power to run the data center themselves.

      But on the other hand, what if the heat exchange pump breaks down, and the liquid solution ends up boiling.....

  • I don't see how this can be good for use w/ a server. It does nothing to increase cooling within the room itself. The server is going to emit the same amount of heat regardless of whether it's air cooled, or mineral oil cooled. The mineral oil will transfer the heat from the components faster, but it will not transfer out of the mineral oil into the air as fast. On a hobbyists computer, it will get shut down, or the load will decrease to almost nothing daily and allow the built up heat to dissipate. On a s

    • Well, I just went back and watched the video in TFA. They are pumping the oil out to a radiator, or are cooling it somehow. With the added cost of needing to pump mineral oil and cool it, I'm not sure where the savings in electricity is coming from. And all the other problem are still present.

      • Giant data warehouses, in addition to all the fans for their individual servers, also have enormous air conditioners trying to keep the ambient temperature from hitting 40C. All the hot air that is blow OUT of servers has to go somewhere, and if that somewhere is an enclosed warehouse, things would get hot pretty quickly if the room was not originally designed as a data center. I recall hearing something about "chicken coop" style data centers which have windows to allow for air flow and breezes outside,

  • it's hard to tell but they might have a special adapter for plugs. i have read that cables will wick the oil in the shielding braid (usb cat5 etc) which can cause a mess.

    also there are a lot of server rooms out there that aren't that organized. i'd imagine in a working installation everything would end up oily, i'd be also a bit wary of installing hard drives in these things. i thought they had the pressure hole for a reason and if your coating failed you could have a massive drive failure.

    that said i'm su

  • So the oil is not electrically conductive (a good thing right?). What happens when it seeps in between connectors, i.e. into the ram slots or PCIe slots? You start getting really odd random problems, or? How do they address this problem? Also how do you clean the system if you need to service it (i.e. replace bad ram/cards/etc.). If you don't the oils going to get into the slots for memory/cards/etc. when you start swapping components out.

    • Re: (Score:2)

      by slim ( 1652 )

      My guess about the contacts is that once things are plugged together, they're touching and oil won't break that contact.

      I seriously suspect that the expectation is that a system won't ever be repaired -- if it breaks, it's binned. This is likely to be justifiable based on some sums to do with MTBF, depreciation etc. Of course that kind of thing only really works if your operation scales to hundreds or thousands of machines -- or, I suppose, if an insurance company takes on the spread of that risk.

    • Re: (Score:2)

      by blueg3 ( 192743 )

      Air is also not electrically conductive and seeps between things much more efficiently than oil.

    • I had the same problem with air getting in between my contacts.

  • Not a new idea... (Score:3)

    by zevans ( 101778 ) <zacktesting@@@googlemail...com> on Wednesday April 13, 2011 @07:53AM (#35806398)

    Cray-2 used Fluorinert. In 1985. Related jokes and memes abounded until... dunno. Certainly they were still part of HPC culture when I started my career in 1994.

    • Re: (Score:2)

      by ajlitt ( 19055 )

      Everything old is new again. In other news, someone just discovered that distributing DC within a rack uses less power.

  • Mineral Oil is not exactly green (Score:3)

    by rabun_bike ( 905430 ) on Wednesday April 13, 2011 @08:10AM (#35806558)
    A mineral oil or liquid petroleum is a liquid by-product of the distillation of petroleum to produce gasoline and other petroleum based products from crude oil. And it isn't exactly non-toxic nor non-flammable (see link below). Not to mention all the heavy metals still found in many servers. Inevitably some of those metals will be picked up by the circulating oil so disposal might become an issue as well. Don't get me wrong, I like new ideas that save energy but touting it is "totally green" is skipping a few steps.

    http://www.jtbaker.com/msds/englishhtml/m7700.htm [jtbaker.com]

  • Wow... awful idea for 99% of datacenters... Especially those that have ceilings greater than 6ft high.

    Let's see... in all of the pictures the submerged rack is placed on some sort of black grid. I'd bet that if you put this rack on a normal datacenter tile floor and 1 drop of oil got on those tiles, you'd have a nice slip n fall lawsuit on your hands. Besides, the thought of having to stock paper towels and a hazardous spill cleanup kit next to every rack doesn't excite me...

    How many vendors actually supp

  • It seems like a lot of people confuse the ability to cram this many servers into a "rack" with an energy efficient, "green" data center.

    The thing is, even though it's about 5x the power density of a "normal" data center, all you're saving is space that more conventional servers would have taken, and maybe gaining a little efficiency in power at the cost of having to maintain all those mineral oil baths. You still have to supply those servers with network connections, potentially also external storage, p

Slashdot Top Deals

The sooner all the animals are extinct, the sooner we'll find their money. - Ed Bluestone

Close