Three-Mile-High Supercomputer Poses Unique Challenges 80
Nerval's Lobster writes "Building and operating a supercomputer at more than three miles above sea level poses some unique problems, the designers of the recently installed Atacama Large Millimeter/submillimeter Array (ALMA) Correlator discovered. The ALMA computer serves as the brains behind the ALMA astronomical telescope, a partnership between Europe, North American, and South American agencies. It's the largest such project in existence. Based high in the Andes mountains in northern Chile, the telescope includes an array of 66 dish-shaped antennas in two groups. The telescope correlator's 134 million processors continually combine and compare faint celestial signals received by the antennas in the ALMA array, which are separated by up to 16 kilometers, enabling the antennas to work together as a single, enormous telescope, according to Space Daily. The extreme high altitude makes it nearly impossible to maintain on-site support staff for significant lengths of time, with ALMA reporting that human intervention will be kept to an absolute minimum. Data acquired via the array is archived at a lower-altitude support site. The altitude also limited the construction crew's ability to actually build the thing, requiring 20 weeks of human effort just to unpack and install it."
Redundancy! (Score:2)
Simple answer: Have redundancy all over the place so it doesn't matter if a few modules fail. The repair crew can go in once a year and swap them.
Re:Redundancy! (Score:4, Insightful)
The article is about the challenges posed with operating the components at such a high altitude and for people who aren't used to high altitudes, they really can't work effectively up there limiting the pool of tech support personnel you can send up there significantly.
Re: (Score:3)
Clusters don't do load balancing in the sense that a datacenter would and the nodes don't fail over (but the switches probably should in this case). If a node fails, it gets turned off and the cluster i slightly less powerful until it is replaced.
Re: (Score:2)
If the hardware is in the rack, why would you idle it?
Re:Redundancy! (Score:4, Funny)
Great, now I need a new skill:
IT sherpas needed for new datacenter.
Re: (Score:3)
The only IT related things from the actual article are:
Use SSDs.
Use bigger fans.
Seems kind of a waste to not put that in the blathering summary.
Re: (Score:3)
Or if they're having trouble finding a sysadmin, I'm available. My Spanish is decent and I have extensive experience in avoiding physical activity thus reducing the need for oxygen. Email address above!
Altitude Sickness... (Score:2)
Re: (Score:2)
Mauna Loa is about 13,000ft above sea level at the peak. I was the only one that didn't have a problem breathing, but our rental SUV barely made it up... it was choking on the thin oxygen most of the last quarter of the way up. Might be one of the reasons we weren't supposed to go up the mountain according to our rental agreement.
Re: (Score:2)
Re:Altitude Sickness... (Score:5, Interesting)
I think you misunderstand how it really works.
Modern cars have air mass sensors -- they sense exactly how much air is coming into the engine, no matter what the pressure of the air is. This control is instantaneous, there's no adjustment period. The amount of fuel injected into the air is based only one the air mass and some slowly adapted tuning constants. The "lesser volume of air, same amount of fuel" assertion is completely untrue!
So, you may ask, what if the relative partial pressure of oxygen in the air dropped with altitude -- that would be a problem, as the car only senses the air mass, not oxygen mass. It has to adapt the fuel amount relative to amount of air only based on the readings from the exaust oxygen sensor. This is not instantaneous -- the oxygen sensor readings are in effect low-pass filtered and affect the air-fuel mix very slowly, with time constants, I'd guess, on the order of an hour. Here's the good news: the relative partial pressure of oxygen stays pretty much constant at altitudes where there are roads. So it's not a problem.
Your car has a problem of some sort, what you describe is not normal behavior.
I was driving in a turbocharged car in the Alps and there were no performance problems related to altitude changes -- the absolute boost pressure was maintained by the ECU per throttle commands and load factor, as desired, delivering apparently same mass of air to the engine, at given load, as at sea level. This worked even on some of the highest paved roads out there. Even if there was no compressor in the intake, the engine would simply lose performance with altitude, but recover it without any undue effects when driving down.
Re: (Score:2)
Re: (Score:1)
Can you describe this with a car analogy?
Re: (Score:1)
Fifteen years and I finally have a use for remembering that term. =D
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Agreed. That's the key: you have to find a slow, steady pace. Going too fast makes for a very crappy day.
Re: (Score:2)
Re: (Score:2)
This was a about 10 years ago, I don't recall needing any special sticker/permit to gain access to the road, but that could have been handled by the driver while we were at a lower altitude (we stopped at a base station of some sort). This was probably before they started adding GPS units to most rental cars. I wouldn't be surprised if that practice is less common in Hawaii anyway as there are relatively few places to take a stolen car. While our rental was sputtering along, we did encounter other smaller c
Re: (Score:1)
I am going to assume that was Mauna Kea [hawaii.edu] and not Mauna Loa [wikipedia.org]. There is a road to the summit of the former, paved part of the way, for the people who work at the observatories up there. There is a 4WD unpaved "road" part way up the slope of Mauna Loa but no vehicular access to the summit.
Rental car companies don't like having to come collect their vehicles from Mauna Kea after people have destroyed the brakes in them riding them all the way down the hill (or having negative interactions with invisible cows [hawaii.edu] at
Re: (Score:2)
"Well, I've cut it three times and it's still too short"
Ouch!
Re: (Score:2)
Mauna Loa is about 13,000ft above sea level at the peak. I was the only one that didn't have a problem breathing, but our rental SUV barely made it up...
How many decades ago was that? Anything reasonably modern (and rentals are usually no more than a year or so old) would be fuel-injected and computer-controlled; altitude shouldn't affect it like that. I've driven up to Pikes Peak (about 14100', IIRC), and my car didn't have any issues. (It was a 2004 Oldsmobile Alero...nothing particularly exotic. Still drive it to work every day.)
Re: (Score:3, Informative)
...the only cure is to remove to a lower elevation.
Top Gear suggests viagra
Re: (Score:3)
This study [webmd.com] suggests there's a real effect going on there.
I didn't realize that Viagra is just a specific vasodilator. (It works on the smooth vessels found only in erectile tissue and in the lungs, apparently.) I wonder if there's an analogue that would work for vasoconstriction-induced migraines?
Re: (Score:2)
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
Taking off in an unpressurized small plane and flying up to that altitude (granted, starting at about 400ft) in a lot less time than 4 hours (we took about 15 minutes) didn't "get" me or the other two folks I was flying with. Then again, we didn't get out and go for a walk at that altitude either.
Re: (Score:1)
say what? (Score:2, Insightful)
134 million processors, 140 kilowatts?!?
1 miliwat per processor?
Re: (Score:3)
Re: (Score:3)
The processors are custom made processing boards.. they are not CPU's you would get off the shelf. They are made to do only one task.
Re: (Score:2)
"A three mile tall supercomputer? You're damned right that poses some unique challenges!"
Re: (Score:3)
More than one fiber would be needed. There are 50 antennas each with multiple fibers connected to the correlator. A lot of thought went into it and despite the complications it was simpler to put the correlator there than 'down the road'.
Particle problems, too? (Score:1)
The article mentions how hard-disks fail at this altitude because the heads can't glide over the platters on a layer of air because it's too thin. The thin air also is less effective for cooling. However, it didn't say if there's been any consideration of an increased incidence of high-energy particles from outer space causing random faults even in solid-state components.
Re: (Score:2)
The hard drives don't belong at that altitude, then. The correlators can well be completely diskless machines, even without a solid state drive. They can boot over the network from the lower-altitude server station.
Re: (Score:2)
This is ridiculous. Just pressurize the server room or whole building and be done with it. That layer of air would automagically reappear for the heads to glide over the platters.
As for particles, I know nothing about the subject but I guess that mountain isn't much closer than we are from the particle sources. I do not think the additional layer of atmosphere said particles have to go through to get to us makes a difference.
My understanding is that particles are deviated at higher altitude than mountains b
Re: (Score:2)
At that altitude, the pressure is about half that of sea level. So to pressurize the building, you would have to do so by several psi to bring it down to something a little lower.
well, 1/2 atmosphere = 7 psi.
5 psi should be enough,
8 psi causes stress on en airplane fuselage but planes have to be light. Submarines can stand 1,500 psi and above...
My point is that it just might end up being cheaper than fixing/adapting every piece of equipment individually, in a never ending quest. Who know?
Re: (Score:2)
Of course that means that the room pressurization would be a single point of failure for every hard drive you had. Lose pressure and every drive suffers a head crash simultaneously... Oops.
If I didn't need the storage volume I'd certainly prefer drives like SSDs that didn't require pressurization to work at that altitude. One le
Three-mile-tall (Score:1)
Re: (Score:2)
Would be much easier to just lay it on its side.
A/C Strikes Again (Score:1)
Any discussion in TFA about why they aren't just sending that data over optical cables to wherever would make sense to house a data-center? e.g. ground-level?
Why does it need to be so near the array?
Speed of signal over fibre can't be the difference (i.e. what difference does 1 km or 40km make? not much).
Re: (Score:2)
e.g. ground-level?
Whoa! A levitating three mile high supercomputer!
I Couldn't Help But Think of a Better Location (Score:2)
Put the computer at a lower altitude. (Score:2)
They are combining signals from dishes separated by up to 16km so it is not a necessity that the supercomputer be right next to one of the dishes. Why not build the supercomputer at the base of the mountain instead of the top. They are already beaming raw data around so it will not make a difference.
Virtualization! (Score:1)
Just virtualize the supercomputer in the clouds and put the virtual machine on the mountain!
See?