Immersion Cooling Drives Server Power Densities To Insane New Heights (datacenterfrontier.com) 80
1sockchuck writes: By immersing IT equipment in liquid coolant, a new data center is reaching extreme power densities of 250 kW per enclosure. At 40 megawatts, the data center is also taking immersion cooling to an entirely new scale, building on a much smaller proof-of-concept from a Hong Kong skyscraper. The facility is being built by Bitcoin specialist BitFury and reflects how the harsh economics of industrial mining have prompted cryptocurrency firms to focus on data center design to cut costs and boost power. But this type of radical energy efficiency may soon be key to America's effort to build an exascale computer and the increasingly extreme data-crunching requirements for cloud and analytics.
The sad part, evil pays the highest rent. (Score:1)
All this exascale capability will be used by the highest bidders to crack encryption to enable snooping without a warrant and model new nuclear weapons technologies...
Re: (Score:1)
They really won't care about that.
What they will care about is "which stock is going to make our company the most money this nanosecond?"
Re: (Score:3, Informative)
Which sucker trying to front run suckers can I front run this nanosecond.
Re: (Score:1)
I always read this. China builds a new computer, its for "cracking encryption" and modeling nuke poops. The US builds a CPU farm, and it is for the Illuminati to crack AES-256. Realistically, the computer winds up for other things. Better hurricane models (the current prediction technology had the worst hurricane on record cause -zero- deaths in Mexico.)
The computers that do the number crunching are not vector based machines anyway. They are made for MIPS, not MFLOPS, for the most part, although FP is
Re:The sad part, evil pays the highest rent. (Score:5, Informative)
Better hurricane models (the current prediction technology had the worst hurricane on record cause -zero- deaths in Mexico.)
That was caused by sensational journalism rather than bad models. Although Patricia was a record storm out at sea, models showed that it would lose energy as it passed over cooler waters close to the coast. The models correctly predicted that the winds would drop from 200mph to about 165 by the time it came ashore.
The next big use for large machines is HFT. A microsecond or two on a fast pipe can mean millions in stock gains.
The glory days of HFT are in the past. Speed is no longer an advantage when everyone is doing it. Besides, HFT needs fast pipes, but doesn't really need a lot of computation.
Re: (Score:2)
>> HFT needs fast pipes, but doesn't really need a lot of computation.
HFT needs short pipes, not really fast pipes.
Re: (Score:1)
Better hurricane models (the current prediction technology had the worst hurricane on record cause -zero- deaths in Mexico.)
There is a reason for that. Hurricane Patricia hit in a sparsely populated area and the Mexican government did a good job of evacuating people from areas that might get hit. The storm weakened considerably after hitting the mountain range that's near the coast. Patricia had quickly increased in power while off the coast but it didn't have a lot of time to get a strong storm surge going (the storm surges are what usually cause most of the damage). The sea floor drops significantly off the coast too, so t
mirror! (Score:1)
The website is slashdotted. Here's a mirror:
1sockchuck writes:
By immersing IT equipment in liquid coolant, a new data center is reaching extreme power densities of 250 kW per enclosure. At 40 megawatts, the data center is also taking immersion cooling to an entirely new scale, building on a much smaller proof-of-concept from a Hong Kong skyscraper. The facility is being built by Bitcoin specialist BitFury and reflects how the harsh economics of industrial mining have prompted cryptocurrency firms to fo
Re: (Score:1)
Sounds like even with all that power, their servers still suck. Not the best PR.
The eighties called... (Score:2, Informative)
...they want their cooling back: https://en.wikipedia.org/wiki/Cray-2
Re:The eighties called... (Score:5, Interesting)
...they want their cooling back: Cray-2 [wikipedia.org]
I was actually one of the admins for a Cray-2 (and other systems) at NASA LaRC from 1988-1992. It was pretty cool (no pun intended). The chassis was Plexiglas (or something else clear) and you could see the 3D circuit boards immersed in the Fluorinert [wikipedia.org] - which was wicked expensive back then. I always wanted to put some plastic fish inside the system... The system was moved to the Virginia Air and Space Center (VASC) for a while after being decommissioned sometime later.
Despite the summary, this is somewhat new... (Score:5, Informative)
The difference with this approach is two-phase cooling, where they're actually boiling the heat transfer fluid. That can remove heat a lot more quickly, as long as you can keep a few issues under control:
1) Getting a working fluid with an appropriate boiling point and otherwise acceptable physical parameters (non-flammable, doesn't dissolve your circuitry, etc). 3M has already stepped up to the plate on that.
2) Recondensing the vapor fast enough. This is a lot easier than cooling the circuits directly.
3) Preventing the hot chips from forming a vapor barrier, which insulates the chips from the coolant. The Leidenfrost effect [wikipedia.org] is an example of this, but you can lose efficiency long before you reach the droplets-skittering-around level, especially if there are lots of nooks and crannies where bubbles can get stuck. Presumably the designers have handled this as well.
If they go with a transparent enclosure and some gratuitous lighting, this could become the new mad-scientist/Big Scary Computer visual trope. Let's face it, lab coats, blinking lights and reel-to-reel tape drives are really tired...
Re: (Score:2)
3) Preventing the hot chips from forming a vapor barrier, which insulates the chips from the coolant. The Leidenfrost effect [wikipedia.org] is an example of this, but you can lose efficiency long before you reach the droplets-skittering-around level, especially if there are lots of nooks and crannies where bubbles can get stuck. Presumably the designers have handled this as well.
I don't know if they took additional steps but in their schematic they show the chips being vertical, thus minimizing the horizontal surfaces needed for the Leidenfrost effect and making it easier for the vapor to move up.
Re: (Score:2)
Re:Despite the summary, this is somewhat new... (Score:4, Interesting)
Two phase cooling eh? I don't know if that's a good idea.
I would contend that it's usually not a good idea to cool something using a liquid when you let that liquid boil when in contact with what you wish to cool. This is especially true for things you wish to keep evenly cool. I understand that you do gain a lot of heat transfer capability by vaporizing the liquid, but you loose the ability to easily keep heat evenly flowing from a surface when you let vapor bubbles form on it. Perhaps you could deal with that issue using conductive materials to spread the heat out (you are going to need some of that anyway) but it might be cheaper to implement a single phase solution. Also, presumably they are suggesting a "closed loop" system for this liquid, where the vapor would need to be recycled by compressing and condensing it back into liquid. This puts the ambient temperature as the lowest you can get the liquid, without some other multi-phase process (and associated expense).
I would think that it would be better to stay a liquid at all times and pump the liquid though a heat exchanger to be cooled using conventional refrigeration methods. You avoid vapor bubbles causing hot spots, only need to come up a suitable liquid based on it's non-reactive nature that will stay liquid and not have to worry about it having the necessary phase change pressure/temperature for your application. Plus, water chillers are already standard fare at current data centers and in industrial cooling equipment. Just pump liquid though the whole thing and push the thermodynamically expensive processes that involve phase changes off onto existing efficient equipment designs which exist. In short, avoid inventing the wheel...
Re:Despite the summary, this is somewhat new... (Score:4, Informative)
I would think that it would be better to stay a liquid at all times and pump the liquid though a heat exchanger to be cooled using conventional refrigeration methods.
The thing is, you typically move immensely more heat via phase changes than by simply raising and lowering a liquid's temperature. For water, heating one mole (about 18g, or 18 ml, or 1.2 tablespoons) of liquid from the freezing point to the boiling point takes about 7.5 kJ; converting that same amount of water from liquid at the boiling point to gas at the boiling point takes over 40 kJ. (Standard pressure, etc, etc.)
That confers a huge advantage in two-phase systems. Yes, you have to deal with bubbles and vapor barriers, but you also get free vigorous agitation, reducing the risks of boundary layers and poor mixing that complicate all-liquid systems.
Re: (Score:2)
The ideal system will re-condense the vapor quickly, perhaps before it is more than a few cm from where it boiled. The vapor movement accelerates the fluid near the "point of contact" and gets much more heat transfer due to the fresh fluid brought in, the vapor transfers its heat to the cooler fluid until it is vapor no more.
If those vapor bubbles live long enough to get near other boiling spots, then they're starting to have some negative effects. If you end up with a large reservoir of vapor at the top
Re: (Score:2)
Hot spots are worse in conventional systems.. Unless you carefully design the boards you will have areas with no flow that will overheat. Any hot spot in this system will immediately vaporize the liquid and refresh with cooler fluid. Also simplicity- no pumps or hoses to worry about and you could easily have a failsafe that shuts everything down if the fluid evaporates past a certain level.
read the article (Score:2)
They're not compressing it, they're simply condensing it on a cooling coil. And I assume they're going to need some form of refrigeration for that, so I'm not sure why they talk about getting rid of chillers.
Also, the chip surfaces are shown to be vertical, so the bubbles will rise along the surface of the chip, likely creating a convection current in the process.
Re: (Score:2)
I was looking at the 3M site about the fluid they use. I have no idea how expensive this stuff is, but I have a feeling it's going to be pricy stuff. This means that you won't want it to be disappearing so you will need a closed system. This will imply some kind of air tight enclosure topped with a serious heat exchanger. suppose you could get rid of the chillers if you could afford enough of the Novac liquid and didn't care that it evaporated away... But what's the point of the heat exchangers then?
Re: (Score:1)
Re: (Score:2)
I was not going to say anything but that is the most asinine way of thinking. So where did you get your degree, from a cracker jack box?
No, it was the close cover before striking school of engineering...
And with that, we are done.... I might choose to read the rest of your rant, some day when I have time to waste, just not today.
Re: (Score:2)
Re: (Score:1)
Re:Destroying our world (Score:4, Interesting)
Re: (Score:2, Informative)
Also, bitcoin transactions require barely any electricity at all, it's bitcoin mining that wastes valuable resources that could be more productively used on nearly anything else.
Except bitcoin mining is inherently tied with transactions: https://www.bitcoinmining.com.... [www.bitcoinmining.com] bitcoin mining is how the transaction system works, verifying each transaction. The system blows through cheap energy to make fake cash. The whole system results in a massive waste of energy for an inefficient currency.
Re: (Score:3)
The whole system results in a massive waste of energy for an inefficient currency.
Cite? Transactions with traditional currencies aren't free, you know. The bookkeeping, auditing, forensics and prosecutions needed to keep normal transactions sufficiently secure do have a significant real-world cost. Not so much in energy, I'd expect, but much more in person-hours. How does bitcoin compare, all costs considered? I don't know. If you know of some research that tries to answer this question with data, I'd very much like to see it.
Really? (Score:5, Funny)
"INSANE" new heights?
Has Slashdot been sold to the Gawker network now?
Re:Really? (Score:5, Funny)
No don't understand. This is revolutionary. The article has such amazing facts such as:
The Novec liquid inside a BitFury cooling enclosure actively boils as it changes phase, removing heat from bitcoin mining hardware.
How have we not had stuff that ACTIVELY boils as it changes phase before! It's INSANE!
Re: (Score:2)
Not only that, but it's ENCLOSED! That's so much more efficient than a caseless design.
Re: (Score:2)
Re: (Score:3, Insightful)
The inherent value from some mineral from the ground is exactly zero too.
It is all about scarcity and the value we place on things.
Re: (Score:3)
Most minerals have intrinsic utility. Gold is useful as both a conductor and as a reflective coating. Silver is useful in various chemical compounds. Platinum and iridium are very useful catalysts.
A better comparison would have been to fiat currency, which is equally useless except as a means of economic exchange.
Re: (Score:2)
Gold is useful as both a conductor and as a reflective coating. Silver is useful in various chemical compounds. Platinum and iridium are very useful catalysts.
Those things are only useful for people who need/want them. There is no intrinsic value in anything.
Re: (Score:1)
I said nothing that belies my own desire to fleece the market...
Re: (Score:3)
Those things are only useful for people who need/want them. There is no intrinsic value in anything.
That they're useful for people who need/want them is the definition of intrinsic value. If you're dependent on other people to need/want it like fiat currency or bitcoins, then it has no intrinsic value. The reason gold is so successful is because it doesn't decay and is easily portable so you can store it under your bed for fifty years or bring it anywhere to whoever needs it which makes it easy to pass along. Of course it's theoretically possible that gold once had value but today we're only trading it ar
Re: (Score:2)
Most minerals have intrinsic utility. Gold is useful as both a conductor and as a reflective coating. Silver is useful in various chemical compounds. Platinum and iridium are very useful catalysts.
And aluminum, once far more valuable than gold, is also extraordinarily useful due to its combination of light weight, strength, flexibility and corrosion resistance. Today, gold is worth approximately 28,000 times as much as aluminum. Why? Because aluminum is no longer rare, not because gold is 28,000 times as useful, industrially. Should we discover a massive quantity of gold on earth, or tow a multi-million ton asteroid of the metal into earth orbit and start sending chunks down, or find a cost-effective
Re: (Score:3, Insightful)
The inherent value from some mineral from the ground is exactly zero too.
Gold mining is a filthy business. It causes extensive erosion, silted creeks and rivers, and mercury poisoning. Bitcoin mining is far better for the environment, and most of the electricity used comes from clean and cheap hydro-power.
Re: (Score:3)
You can make bitcoin as scarce as you want. Until I can cut ceramic with it I think diamonds are more valuable.
Now you are right in case of pretty things for prettyness, but minerals have properties that make them useful for a lot of different applications. For instance gold is a noble element and thus has great corrosion resistance. Platinum while great for making nice looking rings is also a great catalyst for chemical reactions. While bitcoin's scarcity and value is based on what someone thinks about it,
Re: (Score:2)
By the same argument, couldn't you say that the value of bitcoin comes from humans' continuous demand for currency combined with the utility of bitcoin as a currency? Ignoring its volatility for a moment, bitcoin is a useful medium of exchange in the same way that cash is, and the demand for this utility gives bitcoin value in the same way that the demand for the utility of gold gives it value.
We need a medium of exchange in the same way we need catalysts and conductors, and that medium needs to have variou
Re: (Score:2)
Except that the utility of bitcoin is limited. There is literally one use for it, currency. If we stopped wanting bling tomorrow we'd still be using gold and platinum. If we stopped refining oil tomorrow we'd still be using gold and platinum. If we shut down chemical plants tomorrow we'd still be using gold and platinum.
In each case their prices may fluctuate a bit as supply and demand re-adjust but the wide utility of most minerals mean that their worth and value is constrained.
If we stopped needed a crypt
Re: (Score:2)
Fair enough, but that's still only a difference of degree rather than a fundamental difference. Anything, whether it has only one use or many, only has value while it remains useful. If the luddites won and we gave up technology, going back to simple lives on farms, we wouldn't have any use for platinum either. Gold maybe for fillings, but only for as long as we care about our teeth.
Just because we don't expect an end of a material's usefulness anywhere in the near future, doesn't mean that it has an intrin
Re: (Score:2)
Back and forth it's never constrained due to speculation. Every commodity in the world is subject to wild fluctuation if some idiot investor decides to play with the market. However the underpinning utility of minerals provides a floor to how low the price will go. The world wide resources industry is in the pits at the moment thanks to a global crunch on growth, but don't bank on resources falling any lower unless something major comes and upheaves the market. Gold won't ever be worthless baring some major
Re: (Score:2)
None of which gives gold an intrinsic value. Like you say, if there's a fundamental change in how we live, gold may very well become worthless.
Just because it's unlikely, doesn't mean we should treat it as though it's impossible.
Floors are temporary, and caused only by constraints on supply. Production processes continually get cheaper, so the only constraint in the long run is the supply of the element and the energy
Re: (Score:2)
Actually the way I look at history the things which have had the worst upheavals are things which existed due to speculation. Like the price of tulips, even really really pretty tulips can't be forged into tools. Silver is valuable now as it was in the middle ages, gold more so
It's all about resilience. Take oil for instance. What would cause oil to drop to the price of water? We can rule out finding a never ending supply of it (by the way given the cost of missions like ARRM I don't think gold will do what
Re: (Score:2)
The middle ages is recent - I'm thinking about the last 50,000 or 500,000 years of history, not the last 500.
Off the top of my head, we use oil as a fuel and as an ingredient to make various products (plastics, cosmetics, etc). It will no longer be used as a fuel when we improve battery tech to the point that we can use solar/wind/nuclear/future-tech instead, and our lungs will be grateful. Plastics I don't
Re: (Score:2)
The middle ages is recent - I'm thinking about the last 50,000 or 500,000 years of history, not the last 500.
I don't think any economist could care less about what happens beyond maybe their life time and those of their kids. In the economic world thinking in terms of 200 years is truly farsighted.
Off the top of my head, we use oil as a fuel and as an ingredient to make various products (plastics, cosmetics, etc). It will no longer be used as a fuel when we improve battery tech to the point that we can use solar/wind/nuclear/future-tech instead, and our lungs will be grateful. Plastics I don't know, but you're right in pointing out that the supply is finite, so we'll have to find some alternative. Maybe we can distill the necessary ingredients from plants, as we try to do now with biodiesel, or maybe we just stop using plastic at some point in our development.
There's a lot more to oil than that too. Fuel for every single powered device, sometimes to create electrical power, chemicals of all sorts, medical applications, cleaners, fundamental components of manufacturing, dissolving, greasing, degreasing, I'm wearing a t-shirt right now that once started it's life in a polyxylen
Re: (Score:2)
No, actually, I'm perfectly fine with the way you put it there. I just take issue with people saying gold intrinsically has value, as they usually then use that as the reason to conclude we should use gold a
Re: (Score:1)
Right? They complain about "the harsh economics of industrial mining" when it's really just that there aren't any more suckers for the bottom of the scheme.
I can't believe they're building out more hardware this late in the game. If it was general compute or even GPGPU that'd be one thing; they could try to pivot to providing cloud-compute power once the bottom fell out of Bitcoins. But they're all custom ASICs; they're boned.
Re: (Score:2)
Re: (Score:2, Interesting)
Bitcoin mining has reached a point where the lone individual won't really be able to 'win' anymore. Those already wealthy, with access to resources, will find the bulk of the rest of the keys. If Bitcoin is considered money, then these organizations have literally found a way to make money.
On top of that, since corporate taxation is not especially good even with real money, and ta
Re: (Score:2)
Lead wins from both. Just like gold its value is much less likely to drop to zero, but unlike gold you can use it to trade without even losing it. Beat that.
Re: (Score:1)
Higher performance assumes higher energy use (Score:2)
I remember reading an article about Moore's Law and some rough calculations that at some point we would have to have the energy of the Sun moving through our computers to keep up with performance. In some sort of Matrix-style universe maybe the Milky-way is just some of super advanced alien data center. :-)
Re: (Score:3)
It's not really a law, but more of a set of guidelines... Moore or less.... He said so himself.
Re: (Score:3)
Re: (Score:2)
Energy costs would be zero.
No, energy cost would be the cost of the solar panel, amortized over the number of computations the unit performed during the lifetime of the panel.
If batteries get cheap enough, they could be incorporated to allow processing to continue at night
In which case you need to include the cost of the batteries in your estimate of energy cost.
It's possible that your idea could be very cost-efficient, but definitely not zero.
Re: (Score:2)
I did it (Score:2)
I spilled a glass of wine on my laptop. It didn't go any faster. In fact, it died.
Found a nice video by 3M (Score:1)
It is originally fire suppressant liquid (Score:1)