Elevation Plays a Role In Memory Error Rates 190
alphadogg writes "With memory, as with real estate, location matters. A group of researchers from AMD and the Department of Energy's Los Alamos National Laboratory have found that the altitude at which SRAM resides can influence how many random errors the memory produces. In a field study of two high-performance computers, the researchers found that L2 and L3 caches had more transient errors on the supercomputer located at a higher altitude, compared with the one closer to sea level. They attributed the disparity largely to lower air pressure and higher cosmic ray-induced neutron strikes. Strangely, higher elevation even led to more errors within a rack of servers, the researchers found. Their tests showed that memory modules on the top of a server rack had 20 percent more transient errors than those closer to the bottom of the rack. However, it's not clear what causes this smaller-scale effect."
Heat related? (Score:5, Insightful)
Top of the rack tends to get toasty, but is this too simple?
Re: (Score:3)
Top of the rack tends to get toasty, but is this too simple?
I logged in to say that.
It seems obvious -- heat rises, I would expect top of rack components to fail more often unless the cooling design is well done.
Completely fabricated statistic: Only 10% of datacenters have proper cooling design.
Re:Heat related? (Score:4, Interesting)
Vibration as well. The top of the stack moves quite a bit more than the bottom of the stack, even though the overall magnitude of the movement is small.
Re: (Score:3)
Although true, I don't imagine vibration has any effect on SRAM error rates. Hard drive failure rates, I could imagine (though that's a big stretch).
I wonder if it has to do with the upper servers shielding the lower ones on the rack from the cosmic rays. Time for a tinfoil hat for my servers!
Re:Heat related? (Score:4, Informative)
If it's cosmic rays causing a lot of the problem, the extra material of the racks above would make a difference.
Re: (Score:2)
That assumes that the rays tend to come down vertically. I don't know what the distribution would be, but I'd be very surprised if it was mostly vertical at any particular point on earth. So then it would depend on what the rays had to travel through to get to the memory chips. I'd further assume the computers were not exposed to the sky, so I remain skeptical of the cosmic ray explanation.
It would be easy to test though. Have a rack of servers with only the bottom one turned on. Then move that server
Re:Heat related? (Score:5, Interesting)
I was looking into RAM error rates a week or so ago. There's not a lot of research around, but I recall seeing suggestions that error rates were significantly smaller if the chips were mounted vertically rather than horizontally - because vertically mounted chips present a lower vertical cross-section and most error-inducing cosmic rays come at near-vertical inclination.
Re: (Score:2)
Right, cosmic rays have a hard time penetrating through too much matter, even air, so it makes sense. I've been reading articles about high energy neutrino detection and maybe confused the two just a little. I stand corrected.
Re:Heat related? (Score:4, Informative)
Cosmic rays (they are actually particles, not electromagnetic radiation) cover a whole range of stuff, with individual particles varying extremely widely in energy content. Primary cosmic rays originate outside Earth's atmosphere. When they collide with the atmosphere, secondary cosmic rays are generated. Primary cosmic rays are mostly (99%) nuclei of various atoms. The remaining 1% are mostly free electrons (beta particles). In turn, 90% of the nuclei are free protons (hydrogen nuclei), just because most of the matter in space is hydrogen. 9% are alpha particles (helium nuclei), and 1% are the nuclei of other (heavier) elements. There is also a very small fraction of more exotic stuff, like antimatter.
While the mean energy content of a cosmic ray particle is in the range of only about 10^-11 to 10^-10 J, extremely rare single particles with energy content up to 50 J exist. This energy is truly astounding, as it means a single submicroscopic particle has the same kinetic energy as a slowly pitched or fairly briskly thrown baseball!
Cosmic [wikipedia.org] rays [caltech.edu] are some of the most penetrating radiative phenoma known. Just compare their mean atmospheric penetrative power [bham.ac.uk] to that of other radiative phenomena. The following represent rough mean values of what are actually widely distributed ranges; in other words, some fraction of cosmic rays penetrate hugely in excess of the figure quoted below, just as some fraction falls far short.
cosmic "rays" - 10,000 m (about the same for both primary and secondary)
gamma rays - 1000 m
x-rays - 100 m
alpha particles - 0.1 m
It should also be noted that significant sources of radiative phenomena are generally point sources, or at least localized sources. They are attenuated in concentration, not total amount,by distance, even in a perfect vacuum. This arises due to spreading out according to the inverse square law. For example, if you want to escape the radiation from a nuclear explosion, even in outer space, you can just move away from it. Cosmic rays are completely different in that they are diffuse. They are not "radiating" from a single point at all. They are distributed in concentration and direction everywhere. There is no attenuation due purely to distance. The attenuation of cosmic rays by the atmosphere is a result of collisions of cosmic ray particles with the atoms in the atrmosphere.
Cosmic rays, or better stated, cosmic ray products (neutrinos) have been detected in deep mineshafts after penetrating kilometers of rock. Clearly the beta particles are not penetrating very much at all, and even the nuclei have limited penetration, but some of the subnucleic particles ain't stoppin' for nobody.
Re:Heat related? (Score:4, Funny)
BRB, going to convincine my boss to tip all our servers over.
Re: (Score:2)
It doesn't necessarily assume that at all, but what it can assume is that rays coming down vertically have less atmosphere to travel through than rays at any angle, and thus have more energy when they hit the server. Same reason the midday sun has more heat than the morning or setting sun.
Re:Heat related? (Score:4, Informative)
Back-of-the-envelope calculation using XCOM. [nist.gov]
Assume server rack and contents are made of aluminum (what is the predominant material in a server rack?). Let's say the server rack is 2m in height, but it's not fair to make the whole thing metal. Let's say 20% of it is metal (aluminum for this calculation), the rest is air (or, for the sake of calculation, vacuum). Alumnium has a density of 2g / cm^3 (so a 1m x 1m x 0.4 m slab of alumnium would weigh 800 kg, which appears to be in the middling range for what a server rack can accomodate--again, keep in mind, this is a really rough calculation).
Okay, plugging in Aluminum into XCOM gives a total attenuation in the 100-1k MeV range of ~0.03 cm^2/g.
e^[-(0.03 cm^2/g) * (2g / cm^3) * 40 cm] = 0.09
In other words, that's 90% attenuation. Keep in mind that this was a ridiculously sloppy calculation, with my material assumptions (and possibly energetic ranges) being way off (also, neutron cross-sections could easily be different than photon cross-sections). The point is, it's certainly possible (nay, likely) that the material of the servers themselves are providing shielding from the servers on the bottom of the rack.
Re: (Score:2)
Re: (Score:2)
just take them out of the rack to test..
Re:Heat related? (Score:5, Interesting)
Radiation blockage is mostly a function of mass the rays have to go through. The vast majority of cosmic rays are blocked by the 14 pounds per square inch/100 kilopascals of air above us. That means that a square inch of ground at sea level has 14 pounds of air above it. A square inch section of a rack above you would probably be in the pounds as well, and would block a good portion.
Re: (Score:2)
Radiation blockage is mostly a function of mass the rays have to go through. The vast majority of cosmic rays are blocked by the 14 pounds per square inch/100 kilopascals of air above us. That means that a square inch of ground at sea level has 14 pounds of air above it. A square inch section of a rack above you would probably be in the pounds as well, and would block a good portion.
So how many miles of air equivalent is the typical sheet steel of a cabinet?
Just asking, cuz I can't seem to find it on the net.
Re: (Score:2)
Muons (Score:5, Informative)
Then wouldn't you expect a cascading rate of failures from 20% down to the baseline bottom rack in a linear fashion?
The majority of cosmic rays that make it this far are muons. These are relatively penetrating and I highly doubt that a few centimetres of metal and plastic will have anything like a 20% effect. 60m underground with the ATLAS detector at the LHC we still get a reasonable rate of cosmic rays and we use them for calibration when there is no beam. While the rate is reduced 60m of rock is far, far more shielding than a few computers plus many cosmics passing through you come at an angle so the stack above will have no effect on shielding these.
I expect that heat and vibration will be the most likely causes.
Re: (Score:2)
Re: (Score:2)
The cascade particles each have lower energy than particle that started the cascade. Most of them don't have enough energy to penetrate the next box.
Re: (Score:2)
Re: (Score:3)
As single event upsets (SEU) are caused by cosmic particles which create alpha particles. It makes sense that equipment higher in the rack would absorb more of the alpha particles and block them from systems lower in the rack, but I am not a physicist. Alpha particles are relatively easy to block with shielding.
http://www.statemaster.com/encyclopedia/Single_event-upset [statemaster.com]
As the link said, this was first theorized in 1978 and supercomputer companies have been designing systems with this in mind for decades.
Re: (Score:2)
Re: (Score:2, Informative)
Top of the rack tends to get toasty, but is this too simple?
It is too simple.
In a data center with downflow CRACs that push air through perforated tiles, sufficient underfloor plenum pressure is supposed to be maintained so that the upward air velocity carries cold air all the way up the front of the cabinet, affording sufficient cooling to everything. Not that it always works that way.
But one thing to consider is dirt.
Even with MERV 8 or better filtration, dust will still circulate in a data center cooled this way. With the filtration on the CRAC return, the ligh
Re: (Score:2)
Stupid question: why do blowers used in a data center need belts? These days, they should all be direct-driven by brushless motors. At most you need a coupling, although blower-duty brushless motors should have bearings sufficient to support the blower, thus you need no couplings. That way only one bearing is anywhere near being exposed to air that is blown around. I've been to a clean room facility that had all ventilation systems completely direct-driven, and the facilities people loved it.
Re:Blower Belts (Score:2)
Re: (Score:2)
No. Or at least not with competently designed racks, as they are cooler on the top.
The reason is likely a lot more simple: Particles that cause this come from above. Traveling through a number of steel plates (2 per server) stops some of them and reduces energy for others. Hence less reach the bottom of the rack. In addition, those that do not come straight from above have to travel through more air, hence they are fewer or have less energy. See? Simple.
Re: (Score:2)
Top of the rack tends to get toasty, but is this too simple?
This is an explanation they suggest in TFA
Re: (Score:3)
Then they do not know much about rack construction. Standard racks suck in cold air from the front (cold isle) and blow it out the back (hot isle). There is no difference whether the computer sits on the bottom or the top of the rack as the hot air from any of them never gets to another computer directly.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Lead / concrete in the ceiling would seem to be an easier option.
That said, the error rates we're talking about are not large.
Re: (Score:2)
And, while we're at it, ECC RAM is able to correct any single bit error in an access (whatever width the bus is) and detect any double bit error. The likelihood of more than two bits in the same word flipping is so minuscule I think it's pretty clear it's not worth it.
...and metres thick (Score:2)
Re: (Score:2)
Re: (Score:3, Insightful)
Also stack turned-off servers above an active one on bottom to see if it's shielding.
Re: (Score:2)
Or grab the average temperature of the servers on the bottom of the rack and apply an equivalent heat source to the one server and see what the error rate is.
Re: (Score:2)
Or grab the average temperature of the servers on the bottom of the rack and apply an equivalent heat source to the one server and see what the error rate is.
If it were strictly temperature, it would indicate a less than effective rack cooling system.
Are the top machines really that much hotter than the lower ones?
(Biggest rack I've played with had thermometers at top and bottom, and there were at worst, only about 5 degrees difference.)
Fusion IO? (Score:4, Interesting)
basements (Score:5, Funny)
Another reason for nerds to stay in the basement
This isn't news (Score:5, Informative)
This isn't news. Companies that make supercomputers have known this for decades. The one I worked for 15 years ago used a high elevation test environment in Colorado to verify error correcting capabilities. Even the article says that the results were not a surprise.
Re:This isn't news (Score:5, Informative)
Re: (Score:2)
So we should build data centers in abandoned mines. Plenty of shielding from cosmic rays, a steady 55F ambient temperature, and all the heat exchange capacity you could want.
Re: (Score:2)
If the submitters actually took the time to read the articles, the quantity of stories posted here would drop substantially. NEXT PLEASE!
Re: (Score:2)
And yet the Slashdot summary makes it sound like something new.
Seeing how GP referred to "data centers at altitude", I would say this is indeed something new. You certainly don't see a floating data center in the sky every day!
Re: (Score:2)
A transient bit error on a bus that constantly refreshes the value is of no consequence. You'll get a temporary upset, but so what. Suppose the pedal is sampled at 100Hz (a reasonable value). A single upset will at most "floor" it for 10ms. You may hear it, but it's not unsafe.
Toyota's drive-by-wire system's transient error behavior is not to blame for any of the problems. A "stuck" throttle effect is either a permanent and undetected failure of the pedal sensor assembly, a latch-up of the throttle plate ac
Re: (Score:2)
Wait, so there was no state machine that actually drives task lifetimes, presumably by definition not ignoring any task death? Oh boy. I'd have thought that by now everyone has realized that if there's something that can have multiple states, especially something as important as a control task, there'd be a nice FSM or a HSM taking care of it. Sigh. I mean, come on, it's quite easy to do. It will work in spite of all those bugs that cause tasks to die.
Re: (Score:2)
It is interesting though, and not having found out about it when it was or would have been news makes it a good Slashdot topic (if for nothing else that making more people aware).
That's interesting! (Score:2)
A couple of years back at one of the Supercomputing conferences (I think in Phoenix), Fermilab had a cloud chamber in their booth, and you simply *would* *not* believe the amount of ambient radiation passing you at all times. I can easily believe that altitude would have an effect.
Another interesting idea would be to do the same experiment by latitude. Does the Arctic Region Supercomputing Center have a higher rate than the Maui Supercomputing Center? What happens during an aurora?
Re:That's interesting! (Score:5, Funny)
Another interesting idea would be to do the same experiment by latitude. Does the Arctic Region Supercomputing Center have a higher rate than the Maui Supercomputing Center?
They tried to do that test a few years back, but both research teams mysteriously disappeared. The leading hypothesis is that the Arctic team was eaten by polar bears, but nobody has any idea what happened to the Maui team. The only clue left at the scene was a nearly-empty glass of pina colada.
Re: (Score:2)
but nobody has any idea what happened to the Maui team. The only clue left at the scene was a nearly-empty glass of pina colada
Japanese tourists (have you seen how they get when a little alcohol's added to the mix??).
Re: (Score:2)
but nobody has any idea what happened to the Maui team.
Didn't you watch Lost? They were eaten by polar bears.
Re: (Score:2)
Re: (Score:2)
My wife grew up in Puno, Peru, at 3840 meters (12,600 feet) altitude. You will get sunburned so fast you won't believe it, even when you're dark complected like me. Black African tourists get sunburned. IIRC, most of the air molecules belonging to Earth are well below that altitude. If the effect on ultraviolet light shielding is that dramatic I can't help but think that other cosmic radiation is going to be stronger at that altitude as well.
This is news? (Score:5, Funny)
Re: (Score:2)
Explains why astronauts don't remember seening aliens.
They compared two completely different systems (Score:2)
"The group had found that, when all other possible confounding issues were factored out, Cielo's SRAM had a "significantly higher rate of SRAM faults,"
Re: (Score:2)
Water towers (Score:3)
Re: (Score:2)
Re: (Score:2)
Caches, eh? (Score:2)
As this is not (mainly) about the system RAM, it's about the CPU caches, I wonder if any attempt is being made to correct the errors, and if it's worthwhile. One would just need to reset the node on any sign of an error, so the capactiy penalty would be small compared to ECC. On the other hand, the errors could just as well happen in the actual logical units, and at some point it's impossible or very expensive to protect everything. Because the SRAM takes up a large fraction of the CPU area, it may be usefu
Re: (Score:3)
For IA CPUs the L1 cache has parity and the server grade chips have ECC on the L2 cache.
Hmmm .... (Score:4, Funny)
Is this why when I'm in an airplane I can never remember if I turned all the lights out? ;-)
Re: (Score:2)
Actually, my company makes flash memory and we ship most of it overseas for final packaging. We have to allow for a certain amount of die loss from cosmic rays striking the wafers while they are in flight.
On the bright side of all this. . . . (Score:3)
Product suggestion. (Score:2)
The 1U lead block. Place at top of rack to protect the servers below.
Does it work? Who cares. If people will pay £150 for a wooden volume knob on their audio system, someone is going to pay whatever you ask for a lump-o-lead that may or may not improve the reliability of equipment below.
Re: (Score:2)
Easier: Just put the rackmount UPS at the top of your rack. It's mostly lead.
Re: (Score:2)
Actually, lead alone is a very bad material to stop high-energy neutrons. What you really want is a 1U block of boron, with a 1U block of lead or UPS below it to catch the resulting gamma.
Re: (Score:2)
And Boron is cheap, ~$400/ton.
Now I want a ton of boron delivered to me for some reason.
Re:Product suggestion. (Score:4, Funny)
OK. Where am I going to find an RoHS lead block?
Waste of money (Score:2)
Why did anyone need to do this field survey? It simply confirms what we already know - cosmic rays create SRAM errors. Hot components fail more than cold components. Big whoop.
Re: (Score:2)
Well, there's two possible responses to this.
1) We're slashdot and we think we know everything, they should have just asked us, how dare they
2) Maybe we might trust that a "A group of researchers from AMD and the Department of Energy's Los Alamos National Laboratory" aren't idiots and wanted specific empirical evidence on the topic?
Re: (Score:2)
what we already know
Conditions change. Every 18-24 months a new node appears — 22nm is the scale of contemporary shipping devices. As features shrink their behavior changes and new data is needed. There are applications that need to know error rates to compute how much mitigation is required.
We're not all just making web pages out here.
cosmic ray flux (Score:3, Informative)
http://hyperphysics.phy-astr.gsu.edu/hbase/astro/cosmic.html [gsu.edu]
Proportional to the exposed surface. (Score:2)
Earth act as a shield that protect memory from radiation coming from the other side of the planet. In addition, the collision probability of a particle is proportional to the distance of his travel into the atmosphere, so there is more probability on the ground to be hit by particle coming from the vertical. On a desktop computer the RAM is usually oriented vertically and exposing his shorter side from the top: the exposed area is very small for radiation coming from the top. Not that because of the motherb
Oracle/Sun document that discusses this (Score:3)
There are statistics that cover the expected frequency of events caused by radiation in the first couple of pages.
http://docs.oracle.com/cd/E19095-01/sf3800.srvr/816-5053-10/816-5053-10.pdf [oracle.com]
I also (Score:2)
I'm also more prone to errors when I'm high
Air pressure (Score:2)
From deep within the PDF (second link):
(Living in Colorado, I thought perhaps chips suffered from the same spurting newly opened toothpaste tube problem when manufactured at low altitude and installed into operation at high altitude, but it turned out the hypothesis was different, and, of course, left out of the Slashdot summary.)
What materials block cosmic rays? (Score:2)
What kind of materials (if any) are effective in blocking cosmic rays? Would it be possible to integrate cosmic radiation shielding into an average-sized PC case? If that's impractical, are there building materials that can be used in roofs and/or walls to block this stuff without breaking the bank?
Re: (Score:2)
Depends on the type of cosmic rays you want to shield. For muons and the like, good luck. Huge amounts of mass are necessary, but then muons probably wouldn't interact much with your computer RAM anyway. The practical worry is alpha particles, which will flip a bit once in a while.
not that big of a mystery (Score:2)
Servers are made out of metal and have EM fields! This isn't hard.
buying land for server farms in death valley (Score:2)
who's with me?
Common wisdom (Score:2)
In aggregate entire atmosphere down to sea level works out to something like the equivalent of 30ft of water of shielding.. 20% reduction thru an entire rack of servers sounds to be in about the right ballpark.
People have been running the same experiments on international flights on laptops for years.
how high is your cloud? (Score:2)
Re: (Score:2)
"Pressure"? (Score:2)
Re: (Score:2)
It's those gamma ray bursts. (Score:2)
Ancient news (Score:3)
About five years ago, I was involved in the installation of a thousand-node cluster in Boulder. We knew *before we went in* that we needed to change our EDAC (memory error correction) code to account for the higher rate of bit-flips due to the altitude. Some of the people we were working with had been there when those same problems nearly caused a months-long delay in a larger installation at NCAR nearby. We ended up running into a more subtle problem involving lower air density, heat and voltage, but *this* problem was incredibly old news even then.
well... (Score:2)
known for decades (Score:3)
Perhaps the researchers are too young to have read this 1979 paper http://www.ncbi.nlm.nih.gov/pubmed/17820742 [nih.gov]
Re:This may be stupid... (Score:4, Insightful)
On Mount Everest, time slows by 0.00261261 seconds (2.6ms) compared to sea level.
Every foot higher you go is 90 billionths of a second difference, if you want to check the maths for me. The problem is, we're not talking about a sea-level / Mount Everest communication here. The RAM chips are about a foot long at absolute maximum.
And these sorts of effects then suddenly skitter into insignificance compared to solar radiation, different pressures, different air make-ups, heat, etc.
The fact is, we know that this effect exists. We know that time-slowing exists (GPS wouldn't work if we didn't compensate for such things). We know that solar radiation exists. But this single statistic barely bothers to eliminate memory manufacturer, operating voltage, or ambient temperature as a cause rather than these exotic causes.
Chances are, they might just have had a batch of dodgy RAM chips from a single manufacturer more than ANYTHING else combined.
And, even then, you'd need thousands of test sites / machines to even hint at the cause. But, why bother? We know there would be an effect, we also know it wouldn't be this large or obvious and that - chances are - there's a much simpler explanation. The whole "top of the rack fails more often" hints at what complete and utter bullshit this is. That would be an effect we'd notice at sea-level and most likely things like ventilation and heating have orders-of-magnitutide more to do with it.
Re: (Score:2)
That statement is void of any sense. The units of time dilation are not units of time. More like unitless (percent). Do you mean that a clock on the top of Everest is slower by 0.00261261 seconds per second, 0.00261261 seconds per year, 0.00261261 seconds since the dawn of time? What exactly do you really mean?
I doubt VERY much that the effect is 0.00261261 seconds per second (0.26%).
Re: (Score:3)
But could it be simply gravity?
You mean because the 1 bits are lighter than the 0 bits? But you've got to remember about packing density. You can fit a lot more 1s than 0s because they are thinner. Vibrations in the chips will help the 1s settle to the bottom, despite being lighter.
Re: (Score:2)
Re: (Score:3)
Enough with all the mixing of terminology.
You use 'altitude' when referring to how high something is above the ground. You use 'elevation' when referring to how high the ground is from sea level.
What you don't see are signs for city limits on the road with 'altitude' on them. They say 'elevation' for a reason. Just like you don't find an elevatometer inside an airplane. You find an altimeter.
Mixing these terms as you've done (and so has TFS, so I don't blame you as you were simply restating the flawed su
Re: (Score:2)
An altimeter in an airplane is normally adjusted to show altitude above sea level (although above a certain height, it's set to assume a standard sea level pressure rather than what the current weather is producing). The two terms have very similar meanings, although I think elevation is more often used to refer to a fixed location.
Re: (Score:2)
I have no idea if that's enough shielding to matter. However, if true, would we also see higher error rates in daytime when the body of the Earth isn't standing between the server and the Sun?
Re: (Score:2)
Almost no cosmic rays come from our local sun, mostly just slower solar wind particles.