Elevation Plays a Role In Memory Error Rates 190
alphadogg writes "With memory, as with real estate, location matters. A group of researchers from AMD and the Department of Energy's Los Alamos National Laboratory have found that the altitude at which SRAM resides can influence how many random errors the memory produces. In a field study of two high-performance computers, the researchers found that L2 and L3 caches had more transient errors on the supercomputer located at a higher altitude, compared with the one closer to sea level. They attributed the disparity largely to lower air pressure and higher cosmic ray-induced neutron strikes. Strangely, higher elevation even led to more errors within a rack of servers, the researchers found. Their tests showed that memory modules on the top of a server rack had 20 percent more transient errors than those closer to the bottom of the rack. However, it's not clear what causes this smaller-scale effect."
This isn't news (Score:5, Informative)
This isn't news. Companies that make supercomputers have known this for decades. The one I worked for 15 years ago used a high elevation test environment in Colorado to verify error correcting capabilities. Even the article says that the results were not a surprise.
Re:Heat related? (Score:4, Informative)
If it's cosmic rays causing a lot of the problem, the extra material of the racks above would make a difference.
Re:This isn't news (Score:5, Informative)
Re:Heat related? (Score:2, Informative)
Top of the rack tends to get toasty, but is this too simple?
It is too simple.
In a data center with downflow CRACs that push air through perforated tiles, sufficient underfloor plenum pressure is supposed to be maintained so that the upward air velocity carries cold air all the way up the front of the cabinet, affording sufficient cooling to everything. Not that it always works that way.
But one thing to consider is dirt.
Even with MERV 8 or better filtration, dust will still circulate in a data center cooled this way. With the filtration on the CRAC return, the lightest dust particles will float up to the return and get filtered, but the heavier particles will not make it that high. That is why a clean room has a downward airflow towards filters at the floor, unlike a data center.
What happens is that the lowest systems in a cabinet will get the heaviest coating of dust, made up of the largest particles, with the finer dust more frequently making it into the upper systems.
I have a good handle on dust introduced from outside air (filtration of makeup air, positve pressurization, policies against cardboard boxes, etc.), but one internal source of dust that is hard to eliminate is blower belts. Even when switching to cogged belts, black rubber dust particles will be created and get deposited on surfaces all over.
This is only speculation, but perhaps the finer particles are more damaging than the coarser ones.
cosmic ray flux (Score:3, Informative)
http://hyperphysics.phy-astr.gsu.edu/hbase/astro/cosmic.html [gsu.edu]
Re:Heat related? (Score:4, Informative)
Back-of-the-envelope calculation using XCOM. [nist.gov]
Assume server rack and contents are made of aluminum (what is the predominant material in a server rack?). Let's say the server rack is 2m in height, but it's not fair to make the whole thing metal. Let's say 20% of it is metal (aluminum for this calculation), the rest is air (or, for the sake of calculation, vacuum). Alumnium has a density of 2g / cm^3 (so a 1m x 1m x 0.4 m slab of alumnium would weigh 800 kg, which appears to be in the middling range for what a server rack can accomodate--again, keep in mind, this is a really rough calculation).
Okay, plugging in Aluminum into XCOM gives a total attenuation in the 100-1k MeV range of ~0.03 cm^2/g.
e^[-(0.03 cm^2/g) * (2g / cm^3) * 40 cm] = 0.09
In other words, that's 90% attenuation. Keep in mind that this was a ridiculously sloppy calculation, with my material assumptions (and possibly energetic ranges) being way off (also, neutron cross-sections could easily be different than photon cross-sections). The point is, it's certainly possible (nay, likely) that the material of the servers themselves are providing shielding from the servers on the bottom of the rack.
Muons (Score:5, Informative)
Then wouldn't you expect a cascading rate of failures from 20% down to the baseline bottom rack in a linear fashion?
The majority of cosmic rays that make it this far are muons. These are relatively penetrating and I highly doubt that a few centimetres of metal and plastic will have anything like a 20% effect. 60m underground with the ATLAS detector at the LHC we still get a reasonable rate of cosmic rays and we use them for calibration when there is no beam. While the rate is reduced 60m of rock is far, far more shielding than a few computers plus many cosmics passing through you come at an angle so the stack above will have no effect on shielding these.
I expect that heat and vibration will be the most likely causes.
Re:Heat related? (Score:4, Informative)
Cosmic rays (they are actually particles, not electromagnetic radiation) cover a whole range of stuff, with individual particles varying extremely widely in energy content. Primary cosmic rays originate outside Earth's atmosphere. When they collide with the atmosphere, secondary cosmic rays are generated. Primary cosmic rays are mostly (99%) nuclei of various atoms. The remaining 1% are mostly free electrons (beta particles). In turn, 90% of the nuclei are free protons (hydrogen nuclei), just because most of the matter in space is hydrogen. 9% are alpha particles (helium nuclei), and 1% are the nuclei of other (heavier) elements. There is also a very small fraction of more exotic stuff, like antimatter.
While the mean energy content of a cosmic ray particle is in the range of only about 10^-11 to 10^-10 J, extremely rare single particles with energy content up to 50 J exist. This energy is truly astounding, as it means a single submicroscopic particle has the same kinetic energy as a slowly pitched or fairly briskly thrown baseball!
Cosmic [wikipedia.org] rays [caltech.edu] are some of the most penetrating radiative phenoma known. Just compare their mean atmospheric penetrative power [bham.ac.uk] to that of other radiative phenomena. The following represent rough mean values of what are actually widely distributed ranges; in other words, some fraction of cosmic rays penetrate hugely in excess of the figure quoted below, just as some fraction falls far short.
cosmic "rays" - 10,000 m (about the same for both primary and secondary)
gamma rays - 1000 m
x-rays - 100 m
alpha particles - 0.1 m
It should also be noted that significant sources of radiative phenomena are generally point sources, or at least localized sources. They are attenuated in concentration, not total amount,by distance, even in a perfect vacuum. This arises due to spreading out according to the inverse square law. For example, if you want to escape the radiation from a nuclear explosion, even in outer space, you can just move away from it. Cosmic rays are completely different in that they are diffuse. They are not "radiating" from a single point at all. They are distributed in concentration and direction everywhere. There is no attenuation due purely to distance. The attenuation of cosmic rays by the atmosphere is a result of collisions of cosmic ray particles with the atoms in the atrmosphere.
Cosmic rays, or better stated, cosmic ray products (neutrinos) have been detected in deep mineshafts after penetrating kilometers of rock. Clearly the beta particles are not penetrating very much at all, and even the nuclei have limited penetration, but some of the subnucleic particles ain't stoppin' for nobody.