Microsoft's Underwater Data Centre Resurfaces After Two Years

Two years ago, Microsoft sank a data centre off the coast of Orkney in a wild experiment. That data centre has now been retrieved from the ocean floor, and Microsoft researchers are assessing how it has performed, and what they can learn from it about energy efficiency. From a report: Their first conclusion is that the cylinder packed with servers had a lower failure rate than a conventional data centre. When the container was hauled off the seabed around half a mile offshore after being placed there in May 2018, just eight out of the 855 servers on board had failed. That compares very well with a conventional data centre. "Our failure rate in the water is one-eighth of what we see on land," says Ben Cutler, who has led what Microsoft calls Project Natick. The team is speculating that the greater reliability may be connected to the fact that there were no humans on board, and that nitrogen rather than oxygen was pumped into the capsule.

Sounds all wet to me

[Anonymous Coward]

It takes Aquaman to replace a hard drive.

Keep it on land?

[hawguy]

Wouldn't it be cheaper and easier to keep it on land and pump in cooling water from a nearby ocean? They can still seal it and keep humans out, but is there any real advantage to actually putting it under water?

Re:

[green1]

is there any real advantage to actually putting it under water?

You don't get nearly the same PR bump for "data centre with no people" as you do for "data centre underwater!"

Re:Keep it on land?

[cayenne8]

Reminds me of the old adage of "how do you make a MS Windows server truly secure"?

"You unplug it, pull the harddrives out, put it into a metal cage and bury it in the ground 10 ft deep encased in concrete."

I guess just putting it underwater adds an extra level of security?

;)

Re:Keep it on land?

[bhcompy]

Yea, coastal dry land is ridiculously expensive. Submerged land is comparatively less expensive.

Re:

[hawguy]

Yea, coastal dry land is ridiculously expensive. Submerged land is comparatively less expensive.

They don't have to be literally on the coast, you can get rural costal land for $10 - $20K/acre. Or they can put the containers on the back side of a coastal hotel and sell the waste heat to the hotel for heating and/or hot water preheating.

Re:

[SuperKendall]

They don't have to be literally on the coast, you can get rural costal land for $10 - $20K/acre.

Have you ever tried piping sea water for any length? It is WAY more corrosive than even just plain water, which is already pretty expensive to pipe long distances. Also, in the piping you would lose a ton of the coolness of whatever you were transferring.

And then what do you do with the spent sea water? You can't just dump that salty water anywhere as it would kill vegetation. Do you pipe it all the way back,

Re:

[hsmith]

Uh, you don't need seawater at all, unsure why you are hung up on running seawater, anywhere.

Re:

[SuperKendall]

Uh, you don't need seawater at all, unsure why you are hung up on running seawater, anywhere.

Possibly you forgot to read the post I was responding to, talking about rural coastal land in order to be close to sea water? It was exactly this, as a helpful refresher for you:

They don't have to be literally on the coast, you can get rural costal land for $10 - $20K/acre

And as an aside, why would you waste valuable fresh water on data center cooling when you could use cheap abundant already cooled sea-water? The

Re:

[dgatwood]

And as an aside, why would you waste valuable fresh water on data center cooling when you could use cheap abundant already cooled sea-water?

For the same reason that you don't want to pump sea water a long distance. It is highly corrosive. :-)

Re:

[fox171171]

For the same reason that you don't want to pump sea water a long distance. It is highly corrosive. :-)

Plastic lined pipe solves that easily.

Re:

[dgatwood]

Not if you you're trying to get efficient heat transfer through the walls of the pipes.

Re:

[pak9rabid]

And the expense to get people in there when needed?

Re:

[Clouseau2]

Why can't they just dump into a deep swimming pool somewhere in rural Texas or another place with cheap land?

Re:

[SirSlud]

If you wanted to test what they were testing, in what world is building a plumbing system and using pumps and powering all that to surround box with water simpler than just .. uh, dropping it in the water?

Re:

[hawguy]

If you wanted to test what they were testing, in what world is building a plumbing system and using pumps and powering all that to surround box with water simpler than just .. uh, dropping it in the water?

It took them an entire day to pull it out of the water and open it -- salvage ship time is not cheap. Is running power and data to an underwater datacenter really cheaper than running a pipe?

Re:Keep it on land?

[williamyf]

It took them an entire day to pull it out of the water and open it -- salvage ship time is not cheap. Is running power and data to an underwater datacenter really cheaper than running a pipe?

Yes it is cheaper to run power and data to the submerged DC, rather to have it on land, in terms of energy and CO2 pumped to the atmosphere (refrigerating is pretty expensive in DCs).

Google had a paper mill in a nordic country near the sea converted into a DC, and pumped cold water from the ocean to cool it (right what you advocate). Check their numbers and you will see.

Also, underwater transoceanic Tbps optic fibers (including the power to feed the regenerating amplifiers) is a well understood science, so feeding the submerged DC a few Km from the coast power and data is relatively simple.

Since after this experiment they saw that failure rates are lower than in a normal datacenter, maybe next time, they will leave it submerged for longer, maybe 4 years, and if that proves successfull as well, they'll leave it submerged for the intended ussefull life of the servers...

Re:

[hawguy]

It took them an entire day to pull it out of the water and open it -- salvage ship time is not cheap. Is running power and data to an underwater datacenter really cheaper than running a pipe?

Yes it is cheaper to run power and data to the submerged DC, rather to have it on land, in terms of energy and CO2 pumped to the atmosphere (refrigerating is pretty expensive in DCs).

Google had a paper mill in a nordic country near the sea converted into a DC, and pumped cold water from the ocean to cool it (right what you advocate). Check their numbers and you will see.

Check their numbers against what? How do you know how that compares with this undersea experiment. Did Microsoft publish numbers for the cost of a pressure vessel, towing it out to sea with umbilical power/data, then recovering it and unpacking it? I'd like to see what the full lifecycle costs are for this -- especially in case of hardware or software failure. "Uhh we just installed new firmware in the TOR switches and now we lost connectivity to half the racks, someone needs to go press the reset button so

Re:

[SirSlud]

Why do you think you have more information to make a better decision than they did, for what purpose, with what money, given what goals, full stop. It's kind of hilarious to me why so people .. even some pretty bright engineers .. can spend their entire days knowing how the devil is in the details of what they do, and then get home and play armchair technical director. It's a character flaw to skip immediately to "oh I probably know better" mode.

Re:

[markxz]

This was a test installation to test the concept of an underwater datacentre. A larger scale impemtation would be designed to allow the more economic instalation and retreval of the submerged modules.

Re:Keep it on land?

[denzacar]

Wouldn't it be cheaper and easier to keep it on land and pump in cooling water from a nearby ocean?

No.
Energy to run the pumps, cost of purchase and maintenance of the pumps, cost of land lease, idiots with blow torches...

but is there any real advantage to actually putting it under water?

No cooling costs, very low land lease costs, safety from idiots with blow torches...
Also, in theory, you could just drop them off of a container ship along the coasts, where needed.
All them dockside properties getting converted into upscale business/tourist zones could simply ring Microsoft and have all their serverlicious needs satisfied without wasting any of that pricey seafront property on storing big and noisy server containers.

There ARE advantages. In theory, there's a market too. Is it worth it... time will tell.
The way oceans keep going there's a chance it's the wave of the future.

Re:

[Darinbob]

Yes you can, but it's a much more boring experiment that way.

Re:

[ArhcAngel]

Radioactive squid!

Re:

[Tailhook]

Wouldn't it be cheaper and easier

Putting "pump" and "cheap" together in the same sentence... No. Just no. There is literally never anything easy or cheap about pumping untreated water. Everything leaks. Everything breaks down. Everything gets plugged up. Everything corrodes. Electricity is expensive and has to be made redundant. On and on.

Read some NRC Event Notification Reports some time. Sites that use natural water sources for cooling all suffer the same dreary collection of problems. The intakes are forever getting plugged

Of course, with a "conventional" data center

[93 Escort Wagon]

The server failure rate may be significantly higher; but it's much easier to get at a failed server to fix/replace it.

Of course, this is only one data point anyway. The lower failure rate could just be a fluke.

Re:

[Spinlock_1977]

I believe you speak with the tongue of a naugties server-hugger who dreamed client-server and quaked in the presence of the Mighty Mainframe High Priests.

The most modern architectures are largely serverless or containerized, so nobody cares if a server spills its bits - another will pick up the load. They're like $99 HP printers these days: If it barfs, you switch to another one.

Re:

[Darinbob]

And these would be modularized. Not just one of these tanks, but one hundred. Then you periodically pull one up, refurbish it, put it back down. It may be that the humans who walk through data centers to replace broken servers are inadvertently reducing the lifetime of other servers; by requiring oxygen in the room, allowing in or creating dust, and human error.

Re:

[Shades72]

Could be considered an oversight in design. You could design the interior to have a walkway of sorts that allows a server to be pulled in/out of the rack, yet keep the walkway and server rack air-tight/sealed from each other. That is, if you (as the owner) fancy the ability to replace/repair servers inside those containers.

Don't think the cost of replacing a complete container will be the issue, but even under water, the usable real estate is limited. There is enough place to expand the amount of containers

Re:

[Rockoon]

I suspect that the definition of 'failed' is one of the many variables that measure differently between data center and sunken container.

When you cannot service failing equipment, you dont just say "oh well guess we are down some equipment" if said equipment still partially works. You re-task that equipment. In this case suppose the local port on one server boards failed, but it can still boot off network share, so even though the system is now diskless, it can still do server-like tasks. In a regular dat

Re:

[93 Escort Wagon]

You're confusing "sunk" with "defenestrate".

Re:

[thegreatbob]

The two aren't mutually exclusive, given an appropriate location.

Radiation?

[Walking The Walk]

I wonder if the lower failure rate has to do with how well water blocks ionizing radiation? I couldn't find any numbers on how deep they sank it, but I think I was taught that a rough estimate is 10cm of water blocks half the ionizing radiation. So at just 1m deep only 1/1000 of the radiation would get through. If your drive errors are coming from cosmic rays or what-have-you, maybe an underwater data center is cheaper than the equivalent in lead shielding?

Re:

[fizzup]

Good question, but water reduces gamma/x/cosmic rays by something like 0.5x per meter. Concrete is about 0.5x per 10cm.

Re:

[larryjoe]

I wonder if the lower failure rate has to do with how well water blocks ionizing radiation?

I can certainly believe that the water helped to shield a significant part of the cosmic radiation. However, those particles generally result in transient errors, i.e., errors that don't cause permanent damage and which are "fixed" with a reboot. My guess is that the 8 out of 855 failure rate refers to permanent failures, which tend to be associated with hardware aging effects but which may also be affected by environmental conditions such as oxygen and heat.

Re:

[kyrsjo]

Also, server grade hardware presumably have ECC, so it won't notice a bitflip here and there from cosmics.

Re:

[larryjoe]

Also, server grade hardware presumably have ECC, so it won't notice a bitflip here and there from cosmics.

That's generally true, but it's all a matter of probability. Depending on the process technology, there is a non-zero probability of multiple bit flips in a single ECC-protected word. So, the vulnerability for experiencing an uncorrectable multi-bit error depends on the scale, time, and resource utilization. 855 units over 2 years could be enough to experience an uncorrectable error.

The underwater location of the server definitely has an impact on the rate of high-energy neutrons, so that's helpful. I r

Re:

[Bruce Perens]

While radiation mostly results in transient errors, it can also cause latch-ups that blow a gate off of the die. For rad-hard equipment in space this is mitigated by silicon-on-insulator fabrication, so that you can't have a latch-up between the substrate and a silicon feature.

Re: Radiation?

[dsgrntlxmply]

I worked in telecom equipment for quite a while. One generation of the principal SoC did not have ECC on a bridge table (TCAM). The failure rate was not high enough to give a statistically significant assessment, but it looked like systems deployed at higher altitude failed more frequently. Third semester undergrad physics had a homework problem on muon decay. Muons should decay between the time that they are created in upper atmosphere cosmic rays interactions and the time that most would reach Earth's

No dust and debris?

[The Faywood Assassin]

I would imagine in a sealed container there would also be less dust to into *everything*.

I agree that a longer experiment would be helpful in getting more reliable failure rates.

Small correction.

[williamyf]

TFS/TFA says "The team is speculating that the greater reliability may be connected to the fact that [...] nitrogen rather than oxygen was pumped into the capsule."

I guess they meant Air rather than oxigen. As any scuba diver (and many science oriented slashdot readers) should know, air is about 21% oxigen, and about 78% nitrogen (the rest is CO2, water vapour and noble gases and other stuff).

If they pump air, even filtered air, they get O2, CO2, water vapor and soe OVCs.

And no one in their right mind would fill a capsule with 100% O2 gas, as that is a bomb waiting to explode as soon as some BadCap (remeber those) explodes ;-)

Re:

[ZombieCatInABox]

No, I'm pretty sure they meant actual 100% nitrogen. And it has the added advantage of making fires within the capsule basically impossible.

Re:

[raymorris]

>> guess they meant Air rather than oxygen....
>> no one in their right mind would fill a capsule with 100% O2 gas, as that is a bomb waiting to explode

GP didn't say "they didn't use nitrogen".
The summary says they filled the capsule with nitrogen *instead of oxygen*.

GP is pointing out the nitrogen wasn't chosen over the competing proposal of filling them with oxygen, nitrogen was chosen I preference to *air*.

The summary should say "nitrogen rather than air", not "nitrogen rather than oxygen".

Re:

[boskone]

I'd love to see that though. I don't want to be close to it, but would love to see how lithium backup batteries do in a 100% oxygen environment

Re:

[bill_mcgonigle]

Surely the contrast was about components oxidizing vs. not.

The important part is the lack of oxygen. It doesn't sound like there is enough cost savings for this to be practical on land.

That's normal

[nospam007]

"The team is speculating that the greater reliability may be connected to the fact that there were no humans on board, and that nitrogen rather than oxygen was pumped into the capsule."

If you fill any server-room with nitrogen, you don't have problems with humans anymore.

Re:

[93 Escort Wagon]

We tried the Zombie Sysadmin thing for a while, but the maintenance and upkeep ended up being higher than expected.

Re:

[williamyf]

"The team is speculating that the greater reliability may be connected to the fact that there were no humans on board, and that nitrogen rather than oxygen was pumped into the capsule."

If you fill any server-room with nitrogen, you don't have problems with humans anymore.

I hope you get moded as funny, but a clarification. There is gear, similar to scuba gear (facemask + Tank, normally at 1atmosphere), that would allow you to stay in a 100% nitrogen gas filled datacenter for hours on end. Firemen sometimes use it, and some other specialized roles do use it too.

In telecoms we had a saying. For a (telco) switch to work flawlessly once commisioned, you only need a dog and a kid. The dog is there to make sure no one touches the switch, and the kid is there to bring food and wate

Comparison?

[lyran74]

It doesn't "compare well with a conventional data centre"... it blows it out of the water!

Awesome

[gavron]

I've ran data centers. This is awesome. As others have pointed it's hard to replace drives or deal with other failures, but given that:

1. COOLING COSTS are way down. Let the sea water do its thing. Sure, it contributes to ocean heat, but immeasurably so.
2. RELIABILITY is way up. 1/8 failure rate of land-based datacenters? Less heat cycles? Less heat per cycle? Awesome.
3. SECURITY is a breeze. You can protect underwater containers easier than a building on a public roadway. Nobody will "drive a truck" through those "fences".

Good on Microsoft for having forward thinking enough to try this, and putting their resources into it. I can't wait for the white paper with data, and I'm not even a MS fan but this is awesome!

Ehud

Re:Awesome

[kyrsjo]

Regarding security, I'm not so sure. While physical theft would be extremely difficult, denial of service would be easy and hard to find who did it. A small boat-anchoring "oops" can take out data and power for a few days, and diver with a explosives could utterly obliterate the thing without anybody seeing it.

Re:

[ebvwfbw]

You've "ran" data centers? How about you've "run" data centers?

Sorry... pet peeve of mine...

Ran, meaning he no longer RUNS data centers.

So what were the failures?

[mykepredko]

I'd be interested in the failure analysis done on the failures they got.

They don't mention intermittent (bit flip) failures which means that they remotely reset the server and kept on going. But, did they get multiples on servers that maybe a human would look at (and call a failure and replace the hardware on a just in case situation)?

Right off the bat you could ask:
1. All N2 atmosphere, does that result in fewer connecton/wiring failures (no oxidation)
2. Less Cosmic Radiation (this has been noted by other

Re:

[currently_awake]

Add vibration and dust. People and trucks driving by will jiggle the hard drives and cause bit errors. Something on the bottom of the ocean only gets earthquakes. Microscopic dust will get into the components and short out the wiring, or cause a head crash on the HDD. Dust plus oxygen means chemical damage to the circuitry. Dust in the power supply and cpu will block cooling and cause it to overheat.

Re:

[currently_awake]

If you include a robot inside the box, it could remove dead servers and send them for repai out an airlock to a robot sub. You could build a reservoir up in the mountains as part of a pumped storage system. The cold location combined with cold sea water would make cooling cheap, you get cheap local power, the power plant pays for the construction, and you could have service tunnels built underneath for repairs.

Re:

[Headw1nd]

Adding an airlock would be a point of failure, or at least a possible source of water or undesirable gasses. I think one of the advantages of the current setup was that it could be completely sealed. The idea of an internal maintenance robot would be cool though.

Marketing's up to their old tricks

[scalptalc]

Puff piece, but good promo on Microsoft's part. I can't believe there is sufficient, efficient enough heat transfer through the walls of the cylinder to properly cool 800+ (silver circle Dells?). Even if they had active cooling on the end of the cylinder where the cabling from the servers terminates. It's in the pics. Maybe a passive fin that exits the cylinder. Nitrogen's thermal conductivity is only slightly better than that of plain ol' air, with helium being much better (https://www.engineersedge.com/he

Re:

[will_die]

that is for the faiure of hardware. the setup saves money because of the saving in heat removal which is far more expensive on land.

Off-shore wind farms

[Pravetz-82]

This will mesh very well with off shore wind farms. There is already power there. There is a place to anchor the containers.
Even if current pylons can't be loaded with submerged DCs, maybe future projects will account for both.

Re:

[Spinlock_1977]

And this, will merge very well with Starlink, providing high speed low latency communications.

Total non-starter

[nothinginparticular]

There is no way this is is ever going to float (sink in this case) for standard data centre applications. In order to maintain these units you'd have to carefully bring them to the surface, transport them to a clean environment, unseal the water tight container, then you can work on them. Then you have to reverse the process to get them back in the water. Any water ingress and the whole data centre is ruined. Compare that with a standard datacentre where everything is designed for hot swap. In order to redu

Nit pick please

[Gonoff]

The article I read says that Orkney produces 100% of its electrical needs from renewables. (specifically wind).

This is an underestimate. When I was back a few years ago, they were producing 125% of their needs from wind. I think they may have passed 140% by now.

This is not an unreliable source. There are very few days where it is windless at sea level and apparently none up the hills where the turbines are.