Forgot your password?
typodupeerror
Bug Supercomputing Hardware

Too Much Gold Delays World's Fastest Supercomputer 111

Posted by timothy
from the other-people's-problems dept.
Nerval's Lobster writes "The fastest supercomputer in the world, Oak Ridge National Laboratory's 'Titan,' has been delayed because an excess of gold on its motherboard connectors has prevented it from working properly. Titan was originally turned on last October and climbed to the top of the Top500 list of the fastest supercomputers shortly thereafter. Problems with Titan were first discovered in February, when the supercomputer just missed its stability requirement. At that time, the problems with the connectors were isolated as the culprit, and ORNL decided to take some of Titan's 200 cabinets offline and ship their motherboards back to the manufacturer, Cray, for repairs. The connectors affected the ability of the GPUs in the system to talk to the main processors. Oak Ridge Today's John Huotari noted the problem was due to too much gold mixed in with the solder."
This discussion has been archived. No new comments can be posted.

Too Much Gold Delays World's Fastest Supercomputer

Comments Filter:
  • I'll fix it (Score:4, Funny)

    by Anonymous Coward on Thursday March 14, 2013 @05:49PM (#43176599)

    Just give it to me and I'll get rid off the excess gold

  • by Anonymous Coward on Thursday March 14, 2013 @05:51PM (#43176623)

    ROHS strikes again

    • by bobbied (2522392) on Thursday March 14, 2013 @07:06PM (#43177237)

      Mod Parent up!

      If I ever get my hands on the guy who had this crazy idea of taking lead out of solder... Huge mistake, even with the environmental issues... /P?

      • by asserted (818761) on Thursday March 14, 2013 @08:09PM (#43177743)

        ...except that the report [semlab.com] linked from the article examines the problem of gold embrittlement of the tin-lead (63% Sn - 37% Pb) alloy. go figure.

        • by Shavano (2541114) on Thursday March 14, 2013 @10:34PM (#43179009)
          Yep. Gots to pay attention. Thick gold on the connector to connector contacts is best, but don't plate it onto the solderable end of the connector, or on the pads on in the through holes. Actually, a tiny amount is good because it prevents corrosion before you have the part soldered on, but it has to completely diffuse into the solder to avoid making a non-conductive boundary layer. If there's too much to diffuse, you're screwed. You'd think the engineers at Cray would know this.
          • Perhaps Cray has seen better years...

          • I'd guess that Cray is buying connectors from a well-established manufacturer, not fabricating their own.
            • by Shavano (2541114)

              I'd guess that Cray is buying connectors from a well-established manufacturer, not fabricating their own.

              The product designer decides what connectors to buy.

      • by tlhIngan (30335) <slashdot AT worf DOT net> on Friday March 15, 2013 @12:56AM (#43179809)

        If I ever get my hands on the guy who had this crazy idea of taking lead out of solder... Huge mistake, even with the environmental issues... /P?

        The problem in solder is not the lead. It's the tin.

        Tin by itself forms whiskers spontaneously. Some of the worst culprits in this isn't the solder, it's the hardware - the tin in hardware used to mount PCBs etc seem to whisker the most and cause problems. And plenty of research have shown what combination of tin ("bright" tin is the worst - and it was only until recently did manufacturers stop using it) led to the worst problems.

        Leaded solder suffers from whiskering as well. Anytime you use tin, you'll have whiskers. Its just a matter of time - use the wrong tin and it'll whisker quickly. Use the right tin and it'll whisker slowly. And it's not the result of electrochemistry, electromigration, or anything. It's just tin atoms wishing to migrate to relieve stress in the crystalline structure. They diffuse through the structure - the atoms aren't pulled locally, but from the entire bulk.

        We knew this when the first solders were created for electronics. At the time, they experimented and found lead worked "well enough".They never went to find out if there's any other substitute. Massive amounts of R&D is going on in materials science to find alternatives.

        • interesting, thanks!

        • by servognome (738846) on Friday March 15, 2013 @01:49PM (#43184381)
          Tin whiskers aren't the only problem. Tin-Lead solder bulk properties can allow it to relieve stress better than lead-free counterparts. This aids reliability when it comes to preventing fatigue related crack propagation.
          There are a number of process factors that also impact the reliability of a solder joint, including heating and cooling rates, flux chemistry, and the plating of the connected parts. These can effect microstructure, intermetallic formation, and void formation. Like you say, for Tin-Lead this has been studied in depth for decades, the focus on lead-free has only been going on for about 15.
      • Agreed. My previous camera (Canon SX20) had to be repaired twice, and then replaced (with an SX40), since it came from one of the first production lines which used leadless solder. The solder was bad, so the voltage converter would stop delivering the correct voltage after about 10 000 shots.
    • Re: (Score:3, Informative)

      by Sponge Bath (413667)
      You are blaming a manufacturing defect on environmental regulation? It is possible to make RoHS products that work correctly. Companies do it everyday.
      • Re: (Score:2, Insightful)

        by Anonymous Coward

        but almost everything, medical, industrial, military, aviation, aerospace etc. basically everything that _just have to work_ is exempt
        there's a reason for that

        • Quote from Wikipedia [wikipedia.org]:

          Legislation published in July, 2011 removes these exemptions.

          Apart from a few exemptions, RoHS2 covers all types of Electrical and Electronic Equipment (EEE) including some medical devices and monitoring and control equipment which have been exempt in the past. Previous exemptions to product from categories 8 and 9 will be gradually phased out,[16] with:[17]

          - Cat. 8: Medical Devices - 3 years after publication
          - Cat. 8: In-vitro-Diagnostics - 5 years after publication
          - Cat. 9: Control and monitoring instruments - 3 years after publication
          - Cat. 9: Industrial control and monitoring instruments - 6 years after publication

          The reason stated on Wikipedia for exempting these things in the first place was being cautious until enough experience had been collected, considering that they only constituted only a small part of the electronics garbage pile anyway.

      • by Anonymous Coward on Thursday March 14, 2013 @08:33PM (#43177995)

        The lead-free solder has cost billions in failures.

        http://en.wikipedia.org/wiki/Whisker_(metallurgy) [wikipedia.org]
        http://nepp.nasa.gov/WHISKER/ [nasa.gov]

        NASA lost satellites because of lead-free solder (despite them requesting leaded solder). The funny thing is, leaded solder completely prevents whisker formation.

        Now, you may not care about whiskers if you just throw away your electronics every year or two, but if you want longevity, these things will kill you. So for lead-free solder preventing pollution? We are producing much more garbage now thanks to whisker-caused short circuit failures.

        • by Shavano (2541114) on Thursday March 14, 2013 @10:35PM (#43179021)

          The lead-free solder has cost billions in failures.

          http://en.wikipedia.org/wiki/Whisker_(metallurgy) [wikipedia.org] http://nepp.nasa.gov/WHISKER/ [nasa.gov]

          NASA lost satellites because of lead-free solder (despite them requesting leaded solder). The funny thing is, leaded solder completely prevents whisker formation.

          Now, you may not care about whiskers if you just throw away your electronics every year or two, but if you want longevity, these things will kill you. So for lead-free solder preventing pollution? We are producing much more garbage now thanks to whisker-caused short circuit failures.

          I agree with everything except the part where that has something to do with gold contamination in solder joints.

        • Re: (Score:2, Flamebait)

          by drinkypoo (153816)

          The lead-free solder has cost billions in failures.

          No it hasn't.

          NASA lost satellites because of lead-free solder (despite them requesting leaded solder). The funny thing is, leaded solder completely prevents whisker formation.

          No, they didn't. They lost satellites because of misapplication of solder, not because it exists.

          Now, you may not care about whiskers if you just throw away your electronics every year or two, but if you want longevity, these things will kill you. So for lead-free solder preventing pollution? We are producing much more garbage now thanks to whisker-caused short circuit failures.

          I have had 0 items purchased since ROHS fail, and I've bought plenty. Maybe you should stop dropping your electronics in the toilet.

    • by Anonymous Coward

      sure there lots of challenges with ROHS solder, but gold embrittlement can happen with SN/PB solder too

      It's not even certain that they are ROHS, lots of stuff is exempt

  • pimp you super computer?

  • Wish (Score:4, Insightful)

    by Sparticus789 (2625955) on Thursday March 14, 2013 @05:55PM (#43176665) Journal

    I wish I had this problem in my life... too much gold!

    Did anyone else go right here? [youtube.com]

  • I bet... (Score:2, Funny)

    by multiben (1916126)
    ...that thing plays a sweet game of minesweeper.
  • by mydn (195771) on Thursday March 14, 2013 @06:02PM (#43176739)
    They realized that it had too much gold when they noticed its name was showing as "Titan, of the Shattered Sun"...
    • by Anonymous Coward

      They bought their cables from that company sells $6000 HMDI leads.

      And got too much gold in them. See... we told you those cables were just a load of fucking bullshit.

  • by Brentyl (685453) on Thursday March 14, 2013 @06:04PM (#43176751)

    Too much gold never slowed down Mr. T. I pity the fool.

    • by Tablizer (95088)

      That's because he outsourced the real work to modified monster metal SUV's.

    • It actually did! Sometimes, when he had to run fast, he had to press down his gold chains with one hand to keep them from dangling all around his neck!

  • Since a little bit is good, more must be better.
    • Yet 'Too much is always better then not enough' is true.

      The key is it only applies to things where 'not enough' doesn't trigger divide by zero.

  • why gold, an excellent conductor and with resistance to corrosion, is causing problems with electronics here?

    I'd have thought the more gold the better, to a point of being too soft anyway. (say 20 or 22k gold?)

    But then again I'm neither chemist nor metallurgist, so I'll invite one or both to explain to myself and everyone else here scratching their heads.

    • by mbkennel (97636) on Thursday March 14, 2013 @06:18PM (#43176883)

      I'm not a chemist either but fortunately somebody who is working on it is.

      Munger also reported the problems with the connector pins, which Oak Ridge Today‘s John Huotari noted was due to too much gold mixed in with the solder. Gold is used for connectors because it does not oxidize quickly, and because of its high electrical conductivity; however, when mixed with solder that contains tin, the gold and tin can combine, making the combination brittle (PDF) under certain conditions. Cray is reportedly replacing the connectors to alleviate the problem.

      • by ebno-10db (1459097) on Thursday March 14, 2013 @07:04PM (#43177225)
        What's strange is how the gold got mixed into the solder. Long gone are the days of cheap gold when they would plate every metallic surface on a connector. Now they selectively plate the mating surfaces. Certainly they don't plate the part you solder. Gold contamination of solder is a well known phenomenon, but I haven't heard of it in decades, literally. The only other thing I can figure is that sometimes they flash plate some gold on the PC board to reduce solder whiskers or something. But that's a well known process. What the hell happened here?
        • Did you miss the part where they said this was at Oak Ridge?

          I.E. bombs!

          I.E. unlimted budget for better nukes now!

          • Did you miss the part where they said this was at Oak Ridge?

            I.E. bombs!

            I.E. unlimted budget for better nukes now!

            zactly! They probably forgot the lessons of excessive gold plating and just relearned them.

        • For high end products like Titan, gold plating can still be as pervasive as in the good old days. The explanation in the article is vague. I wonder if this is really an issue exacerbated by the scourge of lead-free tin plating that is mandated nowadays rather than "too much" gold.

        • What the hell happened here?

          Maybe the Boomer who knew the process inside and out recently retired. I've seen this happen in a few tech/manufacturing industries lately. There's a chance he was hired back as a consultant to fix this mess.

        • Certainly they don't plate the part you solder

          I worked as a surface mount process engineer on microprocessors, every connection that was soldered has gold as it prevents oxidation of the underlying metal. Not enough gold and you get oxides that are too thick or too difficult for your flux to remove and you wind up with unsoldered connections.
          On the flip side too much gold and you start forming brittle intermetallics and changes in grain structure which will impact its reliability.

    • Apparently it's in the article - when mixed with solder containing tin, it becomes brittle. The article doesn't say if these connectors were broken during manufacture or installation, however.
    • Re: (Score:3, Informative)

      by Nuke Bloodaxe (582098)

      Quoting from the article "Gold is used for connectors because it does not oxidize quickly, and because of its high electrical conductivity; however, when mixed with solder that contains tin, the gold and tin can combine, making the combination brittle under certain conditions."

    • Well, why don't we go to our resident expert on the matter--Mr. Article.

      Munger also reported the problems with the connector pins, which Oak Ridge Today's John Huotari noted was due to too much gold mixed in with the solder. Gold is used for connectors because it does not oxidize quickly, and because of its high electrical conductivity; however, when mixed with solder that contains tin, the gold and tin can combine, making the combination brittle under certain conditions. Cray is reportedly replacing the co

      • by Anonymous Coward

        I think a "you must be new here" is in order.

    • by tibit (1762298) on Thursday March 14, 2013 @06:34PM (#43176991)

      I can only guess, but perhaps the coating on the terminals has to maintain certain mechanical properties over time. A wrongly formulated alloy, or a wrong thickness of plating will give you a connector that, perhaps, degrades in presence of heat and vibration. Or perhaps it plastically deforms on the contact area, thus lowering the contact pressure and eventually leading to loss of reliable connection. When you have small contact area, the contact pressure is sufficient to provide essentially a gas-tight connection. As the contact area grows, the pressure drops and eventually you expose your contact area to the atmosphere. At that point things usually go wrong.

      Pure gold is soft and by itself it has about the worst properties imaginable for any sort of a connector surface. It literally rubs off, it's so soft. Its low resistance is irrelevant, since the gold layer is very thin. Gold's bulk conductance plays little role in overall resistance of a mated contact pair. You could replace gold with a metal that has 10x lower conductance, usually with little or no measurable change in contact resistance -- that is, if you can find something that can match gold in other properties (wetting of underlying surfaces, resistance to oxygen, etc.).

      Gold is also useless as plating for high current terminals. I have designed plenty of connectors where some pins were for small signals and were gold plated, and others were for power and were silver plated. Gold plated power contacts simply lose the gold and then you have all the problems of an unplated contact pair that's exposed to the atmosphere since the gold erodes away leaving craters. It's no fun.

      When you get relays with gold-plated contacts, there are often two sets of ratings. One is for low-current use, where the gold is guaranteed to stay on the contacts. Another rating is for sufficiently high current use where the gold is vaporized away and you're left with some other coating material that works well in this application. You can't swap such relays around without realizing what's going on, since contact pairs that were exposed to high currents will perform horribly in small signal, small current applications.

      I also can't quite understand why people still buy gold jewelry -- all it took for me was a gold wedding band. I switched to tungsten carbide after a decade and I'm not looking back. The standard 18K alloy is a joke.

      • But that still doesn't explain how the gold got into the solder. Connectors are selectively plated with gold only on the mating surfaces. It doesn't go on the part that gets soldered.

        You're doing something wrong if your documentation sounds better when translated into Latin.

        The Catholic clergy disagrees.

        • Connectors are selectively plated with gold only on the mating surfaces. It doesn't go on the part that gets soldered.

          My guess would be this was a custom connector and either

          1: some subcontractor decided that given low volumes it was cheaper to just plate the whole thing at the highest specified thickness than to work out how to selectively plate it (possibly in violation of a spec, possiblly following a spec that only specified minimum thickness of gold not maximum).
          2: whoever was designing the connnector specified uniform gold plating

          • 1: some subcontractor decided that given low volumes it was cheaper to just plate the whole thing at the highest specified thickness than to work out how to selectively plate it (possibly in violation of a spec, possiblly following a spec that only specified minimum thickness of gold not maximum).

            I'm guessing this is the case. Even in high volume it's cheaper to plate everything at once so you don't have additional plating baths and processes to manage.
            I actually ran into this issue with a board that had b

    • by Immerman (2627577)

      Alloys work in mysterious ways. Some alloys are "simple" having properties somewhere between the pure metals. Others have properties significantly different than their components. It can be a very non-linear process, in some cases even a fraction of a percent of "contamination" can drastically alter the properties of a metal, or you may have a "sweet spot" where the properties get better and better until you add just a little to much and change things completely. In this case someone said it was a matte

    • As you add more gold you are changing the properties of the alloy you are forming. A small layer of gold to prevent oxidation doesn't really cause much impact, your solder (in this case eutectic Sn-Pb) still melts and solidifies basically the same way. The more gold you add the more complex the system becomes. Rather than a nice eutectic material that goes from liquid to solid directly, you get different phases that solidify at different temperatures. This results in the formation of brittle intermetall
    • If you're interested in another problem with gold intermetallic compounds, look up "purple plague".
  • Let this serve as a reminder that overgolding can happen in any community. If you overgold, or someone you love overgolds, get help before it's too late.
  • by deroby (568773) <deroby@yucom.be> on Thursday March 14, 2013 @06:17PM (#43176877)

    ... I assumed everybody knew by now you should always go with Monster Cables ! ...or maybe they didn't run them in properly ?

    • Well... the thing's name is Titan; I'd say it probably out-did Monster in the gold contacts department. That said, maybe it couldn't handle the high bit-quality and kept getting distracted by being able to hear the imperfections from the original recording equipment....

  • See, everything works out fine.
  • First world problems...
  • Just a question for anyone in the know, and I admit it is heavily loaded:

    If lead was present in this solder, would the outcome have been any different?

  • by bmo (77928) on Thursday March 14, 2013 @07:58PM (#43177603)

    CONQUISTADOR: Welcome to New Spain! This is your new Father - Father Corona.
    FATHER CORONA: Pax vneuti nicutm! down on your knees, now! D'ye recognize what I'm holidn' over your head, lads?
    INDIAN: It's a Cross. The Symbol of the Quartering of the Universe into Active and Passive Principles.
    FATHER CORONA: God have mercy on their heathen souls!
    CONQUISTADOR: What the Father means is - what is the Cross made of? Gold! Have you got any?

  • I will happily scrape the excess gold off of each board for them...

  • I'll bet all the gold makes the processor run terribly :(

  • Me, I think gold is over-priced. Give me platinum any day of the week.
  • """
    It is rumored that the building at Fort Meade will contain the world&#226;&#8364;(TM)s fastest computer, the speed of which will be measured in Exa-FLOPs. A FLOP means FLoating point OPerations per Second and exa means a 1 followed by 19 zeros.
    """

    no it doesn't.
  • Did you know that people in North America pronounce "solder" as "sodder"? I had no idea until I moved to the US and I still find it hilarious!
    • Did you know that people in North America pronounce "solder" as "sodder"? I had no idea until I moved to the US and I still find it hilarious!

      I did indeed know that, being from North America. However, I just learned that people elsewhere pronounce it differently. That's interesting. What, do you pronounce the L or something? Do you pronounce the L in "walk" too? Here "walk" and "wok" are basically homophones.

      Uhhhh, you may now resume your regularly scheduled scheme to become jillionaires by mining computers for a few milligrams of gold.

  • Ah, first world problems, too much bling in your computer.

"There is nothing new under the sun, but there are lots of old things we don't know yet." -Ambrose Bierce

Working...