The Truth About Last Year's Xbox 360 Recall 255
chrplace forwards an article in which Gartner's Brian Lewis offers his perspective on what led to last year's Xbox 360 recall. Lewis says it happened because Microsoft wanted to avoid an ASIC vendor. "Microsoft designed the graphic chip on its own, cut a traditional ASIC vendor out of the process, and went straight to Taiwan Semiconductor Manufacturing Co. Ltd., he explained. But in the end, by going cheap — hoping to save tens of millions of dollars in ASIC design costs, Microsoft ended up paying more than $1 billion for its Xbox 360 recall. To fix the problem, Microsoft went back to an unnamed ASIC vendor based in the United States and redesigned the chip, Lewis added. (Based on a previous report, the ASIC vendor is most likely the former ATI Technologies, now part of AMD.)"
I'm Shocked.... (Score:5, Funny)
Re:I'm Shocked.... (Score:5, Funny)
The Red Rings were a FEATURE! I tell ya!
Re:I'm Shocked.... (Score:5, Funny)
Re:I'm Shocked.... (Score:4, Funny)
Re:I'm Shocked.... (Score:5, Funny)
Re:I'm Shocked.... (Score:5, Funny)
Re:I'm Shocked.... (Score:5, Funny)
Re: (Score:2, Interesting)
Re: (Score:3, Informative)
What the article insinuates is that the fabricated part (the MASK) was designed by Microsoft.
Microsoft bought the IP for Xenos from ATI. They did this because of the poor relationship they had buying GPUs directly from Nvidia with the Xbox. Microsoft saw how Sony bought the IP for graphics parts for the PS2 (and now PS3), and created their own ASIC layouts. Microsoft figured they could do the same.
The only problem with that logic: Microsoft has never d
yes, go cheap, that's the way (Score:4, Interesting)
Re:yes, go cheap, that's the way (Score:5, Insightful)
MBAs are good in cutting corners in traditional businesses, but generally have no understanding of technology risks....
Re:yes, go cheap, that's the way (Score:5, Funny)
Re:yes, go cheap, that's the way (Score:4, Funny)
Re: (Score:3, Insightful)
Re:yes, go cheap, that's the way (Score:5, Insightful)
You can have business savvy and technological expertise, but it's a roundabout path through today's educational system if you're not teaching yourself at least one. And I think we all know the proportion of people who are capable of serious self-education.
Re:yes, go cheap, that's the way (Score:4, Insightful)
And for balance the problem I have with engineers as managers is that it's possible to learn the people skills stuff but you have to understand why it's important and want to do learn it. It's all too easy to stay in the comfort zone where you basically sit in a dark corner somewhere and write code if that's what you enjoy rather than forcing yourself to talk to people.
Re:yes, go cheap, that's the way (Score:5, Insightful)
I don't think you're getting it. Cutting costs is one thing. Cutting corners is another. Cutting costs is fine, but cutting corners implies the product is worse off because of it. Few engineers would say "It'd be cheaper to roll our own graphics chip," because they realize the immense technical challenges involved. Few MBAs are likely to understand that, however.
There's a big difference between what you just said and what the OP said. Nobody said MBAs can't be tech savvy. However, the fact of the matter is, most of them aren't.
Also, just to be pedantic, having an MBA has little to do with having business savvy.
Re: (Score:2, Insightful)
Are you telling me that Intel, AMD, ATI, NV, etc. have never released a flawed chip?
Were the people at MS who made the chip really incompetent - or did MS just hire them from another ASIC company? There is no guarantee this wouldn't have happened if they did go to a ASIC.
Re:yes, go cheap, that's the way (Score:5, Insightful)
That's true, but, if the did go to an ASIC vendor they could have got a contract indemnifying them from taking losses when the chip turned out to be flawed. By doing the chip design themselves, they saved a little bit of costs, but also took on all the risks of having a bad design.
That's what the parent poster is alluding to. A manager with experience in technology would have understood that, while designing your own chip might have been cheaper, it would have also introduced significant downside risk, which ought to have been factored into the equation. Farming the chip design out to a third party, while more expensive in the short term, would have entailed less long-term risk.
Re:yes, go cheap, that's the way (Score:5, Informative)
The mistake seemed to be to let Microsoft's in house group do this rather than outsourcing.
But you've got to remember this is an article in EEtimes from an analyst with an agenda
http://www.eetimes.com/news/latest/showArticle.jhtml;jsessionid=51TYZYXYRWUZUQSNDLSCKHA?articleID=208403010 [eetimes.com]
"System OEMs have no business designing ASICs any longer," said Lewis. The reality is that system companies are finding it hard to do enough ASIC designs to keep in-house design teams employed.
Basically he's trying to create business for ASIC design houses by telling people that putting a bunch of licensed IP onto a chip is rocket science and they shouldn't try to do it in house.
Is it really? I honestly don't know. I suspect it depends a lot on the quality of the in house people and the quality of the ASIC design house.
And it depends on what you're trying to do. In the embedded area lots of companies much smaller than Microsoft put an processor and a bunch of their own peripherals onto a chip and it works. I guess that console or PC graphics cores use a lot more power than that. But I don't know if "an ASIC design house" would have done a better job than Microsoft's ASIC group.
Or more to the point, maybe a $1B recall is the price you pay for learning about this stuff. Microsoft can afford it obviously and it will influence how the successor to the XBox360 is done. Whether they hire more engineers and do it in house or outsource it is a business decision it seems. I guess the in house people and the design house will both try to argue for the best option from their point of view and some manager will decide.
But if you're a cash rich company then the bias will be to try to do as much as possible in house, because that gives you more freedom to value engineer later.
Re:yes, go cheap, that's the way (Score:5, Insightful)
It is true. You should not unnecessarily muck with VHDL/Verilog and 3rd party cores even if you work with an FPGA. This will not kill you, but it will make you poorer. HDLs are notoriously kludgy, and it takes a lot of effort to do it right. Proprietary cores rarely work as documented, and you have no visibility into them. When multiple cores are used, it's one large fingerpointing game between vendors. And you need to have good, experienced HDL coders. And you need to have all the tools, they cost big bucks.
But that's with mere FPGAs, where you can update your design whenever you wish. However here they are talking about ASICs - where all the wiring is done with masks when the IC is made. You'd have to be certifiably mad to even think about a casual design like this. ASIC designs are done by very competent teams, using "10% coding / 90% verification" time allocation, because you can't afford /any/ mistakes. And even then you make mistakes; but experienced teams with good tools make those mistakes smaller, and they call them "errata" - something that is not right but can be worked around. When you make the F0 0F bug, though, you trash the whole run.
So Microsoft risked a lot when it went for an in-house design. I am not surprised that they failed. They should have counted all the successful 3D video companies on the market and asked themselves why there are so few, and why top gaming cards cost so much.
But if you're a cash rich company then the bias will be to try to do as much as possible in house, because that gives you more freedom to value engineer later.
I am not MS, but I don't really see much business value in rolling your own video controller. More likely the NIH syndrome kicked in, or some people were overly concerned about their job security.
Re: (Score:3, Insightful)
The funny thing about this is: ATI is not an ASIC vendor! ATI does chip design and, since they're fabless (or were until AMD bought them,) they get them made at TSMC or sometimes Chartered, or, most often, use NEC Electronics Amertica as their ASIC vendor. ATI partnered
Re: (Score:3, Funny)
You're a perl programmer then?
Re:yes, go cheap, that's the way (Score:5, Funny)
Going cheap may well be the sensible way... (Score:3, Insightful)
Consider: would you rather spend $10M on a platform that may flop and not make a dime
OR
Spend $1B on a platform that has made multi-billions.
Re: (Score:3, Insightful)
still got a while to pay back for the original xbox sink hole
Re:Going cheap may well be the sensible way... (Score:4, Insightful)
Re: (Score:3, Informative)
Loss of $423 million
Equals a spread of $947 million, almost 1 billion.
Add the $1 billion recall, still looks like Vista and Office are paying for the XBOX 360.
Re:Going cheap may well be the sensible way... (Score:4, Insightful)
Re: (Score:3, Insightful)
Re:yes, go cheap, that's the way (Score:5, Insightful)
Bleh... (Score:4, Insightful)
When I read that I pictured Ballmer: (Score:5, Funny)
Re:When I read that I pictured Ballmer: (Score:4, Funny)
You picture Ballmer?
Your therapy bill must be astronomical
Probably not too far off the truth, but still...you think they'd learn.
Re:When I read that I pictured Ballmer: (Score:5, Interesting)
Re: (Score:2, Insightful)
Re:When I read that I pictured Ballmer: (Score:5, Funny)
Re: (Score:2, Informative)
http://www.imdb.com/name/nm0224007/
Re: (Score:3, Informative)
Re: (Score:2)
Chickens are coming home to roost... (Score:5, Interesting)
Or is it all just a hoax? [fugue.com]
Hope not.
Re:Chickens are coming home to roost... (Score:5, Funny)
Re: (Score:2)
But this story is about a billion-dollar smack on the wrist. The previous ones concerned a delay, at least, to their hard-won (okay, paid for...) OOXML ISO certification, and the EU's competition commissioner putting a thinly veiled smackdown on them.
I realise that mob-style business practice has built MSFT into a giant, but as the public and their representatives catch up with the new paradi
Re: (Score:2)
Quite. I was reminded of this [thedailywtf.com] story, more specifically this Douglas Adams-esque line:
Re: (Score:2)
Someone mod parent + funny!
Another Talisman CF (Score:5, Interesting)
Never, and I say NEVER let a bunch of software engineers try to design a hardware chip. This was the biggest CF I'd seen in all my years (30+) as a chip designer. That they did it again, and with such stupidity again is no friggin surprise.
It is not that software engineers should not be involved, of course they should but when they drive the architecture in complete void of any practical chip design constraints..... and continually refuse to listen to any reason from the hardware designers..... well as they say, garbage in, garbage out.
Re: (Score:2)
Re: (Score:2)
its just that brute force is _sooo_ much easier to implement....
Re:Another Talisman CF (Score:5, Insightful)
I suppose if we can all agree to stay out of the other guy's yard, we can get along. You do hardware, I'll do software.
-g.
Re:Another Talisman CF (Score:5, Interesting)
I testify, Brother, I TESTIFY!
30 Years ago, I ended up in therapy (literally) after dealing with an assembly program written by a hardware guy. The program emulated a CDC communications protocol that was originally done in hardware. This was on a Cincinnati Milacron 2200B, a machine that had both variable instruction length and variable data length. The hardware guy had implemented the protocol state changes by putting a label *on the address portion* of jump statements (he did this in 50 different places in the program) and then in some other area of the code he would change where the jump branched to next time through. It bordered on an implementation of the mythical COME FROM instruction. Of course, there was zero documentation and almost zero comments.
After one marathon debugging session I was so frustrated I was in tears. My manager came in and wanted to know what the problem was. I gave him the listing and left to walk around the building a few times. When I came back, he told me that it was, hands down, the worst piece of crap he had seen in 20 years. He had me rewrite it from scratch, which I did over a long weekend.
The program's name was RIP/TIP (Receive Interrupt Processor/Transmit Interrupt Processor) and I was in therapy for most of a year. (There were a few other issues, but this was the bale of hay that made me snap.)
Re: (Score:2)
> guy's yard, we can get along. You do hardware, I'll do
> software.
Wrong. I'll agree with "You do the hardware, I'll do the software," but it's important that both of you visit the other's yard frequently. From experience, there are worse things than the hardware guy throwing hardware and documentation over the wall, and going away to let the software guy do his thing. (Such as throwing the hardware over the wall with no documentation, a
Re:Another Talisman CF (Score:4, Informative)
I can tell you first hand that a lot of the people on the Xbox hardware team a extremely talented HARDWARE specialists. The way you talk you would think MS locked a bunch of IE developers in a room and didnt let them out until they had designed the chip.
And as for the argument of 'well if they are so talented, why is the chip such a POS?', it is not only software engineers that design shitty hardware. Look at AMD, with the TLB defect in the Phenom chips, is that the fault of the software engineers?
This response may be overkill, but somehow you were modded +5 interesting, but you completely miss the point.
Re: (Score:2)
Re: (Score:2)
Wow! You're an orphan of the Talisman project? I remember seeing the hype from that on the eve of my Uni graduation. I then went on to work at Intel, working on chip design tools.
No surprise, the hardware guys looked down on the software guys from a QA perspective. Probably because it's a lot harder to patch hardware once it's in the field, and software guys have a hard time learning that lesson.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re:Another Talisman CF (Score:5, Interesting)
I am a person that designs both hardware, and software, but not chips, At the risk of talking outside of my expertise, I will have a go at answering your question.
Firstly, there are things that software people really like, but it is often better to not do them in hardware. This category contains things like Read/Write I/O registers. From a software point of view, they are nice, but they can double your gate count. They can also increase your capacitive bus loading. DAC and ADC designs can also be affected this way. A software person might use a proper ADC and expect proper ADC registered results. A hardware person might select a resistor, capacitor, a voltage comparitor, and a couple of spare I/O pins. The cheesy R/C approach may save the hardware design from a whole slew of problems including cost. A software person may opt for a synchronous logic approach with all registers clocked every clock cycle. The hardware designer may opt for a much more asynchronous approach, that minimizes the number of clocked registers. This reduces power consumption, and potentially the number of registers too. Often the hardware designer will consider thermal, cost, electrical layout issues as part of his design process. The software person will not be as familiar with how to design a good circuit board and chip design in a cost-effective manner. A good software engineer can learn all of this material with time, but the hardware engineers will do them naturally.
The second category of problems is tools. The modern chip designer is working with a fairly advanced set of tools that the software person is likely to be quite unfamiliar with. This starts with the IC design tools, which are quite specialized. It ends with the hardware engineering tools. Have you ever X-Rayed a circuit board to analyze the cracks in the Ball Grid Array where it bonds to the circuit board? Are you familiar with thermal issues, and thermal images? How about EMI test results? Modern IC package design limitations? A good team of engineers will be familiar with these tools, and know how to use them to get good results.
The third category of problems is mistakes from inexperience, or lack of experience in the correct field. I work with industrial electronics. I think from an industrial point of view. What happens when someone attaches 600 (VAC) to the ground wire of the computer? What happens to the remote sensors when the plant gets hit by lightening? In IC design, there are some known gray areas too. Does the chip reset properly on power up? Do metastable, astable, or self-oscillating states exist in the IC design? Can the chip survive with no cooling? Does the chip have an overtemp shutdown function? What happens if someone starts the chip up in sub-zero weather? Do the analog electronics have sufficient electrical separation from the digital electronics, while avoiding nasty things like ESD latchup conditions?
I've completed chip design courses before, but have never had to design a modern production gate array design. As a person that has done both software and hardware, I know that my skills are not good enough for the most modern IC design processes. My limit is FPGA work, and my preference is clever opto-isolation, power semiconductor, TTL and micro-proccessor based circuits. In analog, my expertise in analog is industrial sensing and survivability. You have to know where your field of expertise is, and what your limits are.
Re:Another Talisman CF (Score:5, Insightful)
Re: (Score:2)
Re:Another Talisman CF (Score:4, Insightful)
Well... there's "real" software engineering too...stuff involving resource deadlock, race conditions, critical section synchronization, in applications like virtual memory management, network protocols, time sync, file systems, security, fault tolerance, etc that are subject to all sorts of 'physical reality nastiness'.
Its not all wizards and automatic code completion you know.
Re: (Score:2, Insightful)
Engineers aren't stupid, and you can certainly cross-train one to do another's job. But you aren't going to do that overnight. If your product is overbudget/behind schedule/etc, you don't want to make it into a very educational failure by regularly having people
Re:Another Talisman CF (Score:5, Funny)
US-based vendor... (Score:2)
Re: (Score:2)
keywords "his perspective" (Score:2, Insightful)
that is all
Ridiculous (Score:5, Informative)
Furthermore, the recall was for overheating in general which -- though unquestionably affected by the GPU -- is a more comprehensive system design failure, not just a single component. (Look at the stability success they have had simply by reducing the size of the CPU.)
I'm looking forward to "Jasper", the code name for the next XBOX 360 mother board that will include a 65 nanometer graphics chip, smaller memory chips and HOPEFULLY a price reduction.
Vote parent up (Score:5, Insightful)
Years before the xbox360 has been released ATI was already announced as the system parter for the GPU. No "secret unnamed ASIC vendor" anywhere.
The recall, again, was thermal problems.
Do you really think a completely different GPU by a completely different company could have been designed in a year _and_ totally compatible with the original one?
Re:Vote parent up (Score:5, Informative)
Re:Vote parent up (Score:4, Informative)
Indeed. (Score:3, Insightful)
Speculation that is well known to be false and could've been showed up as such with a quick look at the XBox 360 specs which are available in many places that I'm sure Google would oblige to discover.
The issue has already been outed as being to do with cheap solder iirc that simply couldn't stay put under the heat of the system over extended periods of time.
Re: (Score:2)
What's going on..... (Score:5, Informative)
Lewis seems to be just plain wrong, which is kind of upsetting for "chief researcher" at a firm like Gartner, especially when the correct information is freely available.
While the cooling solution for the GPU is the likely cause of most of the failures, that's not necessarily the GPU's fault, or ATI's, especially for a fault so widespread.
Re: (Score:2)
Microsoft left a trail of bad mojo with Nvidia over pricing of chips when Microsoft intended to lose money and kept beating them up... then they didn't cut Nvidia in on the new (profitable) one. I'm sure AT
Re: (Score:2)
Re:What's going on..... (Score:5, Interesting)
Everything I've ever heard as a "Gartner opinion" got one of two reactions from me:
1. Well duh.
2. No, that's obviously wrong.
Looks like this is #2.
Re:What's going on..... (Score:5, Informative)
Obligatory Simpson's Quote (Score:2, Funny)
Penny wise - Pound foolish (Score:2, Funny)
But it's the effort that counts, isn't it?
Comment removed (Score:4, Insightful)
this doesn't seem accurate, it was solderability (Score:5, Interesting)
The problem wasn't any chip at all. It wasn't even heat. The problem was the chips were not soldered to the board.
http://www.bunniestudios.com/blog/?p=223 [bunniestudios.com]
Doesn't matter who designed or made the chips. If they aren't soldered down, they won't work. And that's what the problem was. That's why X-clamps (mostly) work.
Heat is semi-tangential. If the chip is soldered down, heat won't pop it off and if it isn't soldered, any kind of movement will break it loose, even when cold. This is how MS could ship you replacement units that were RRoD out of the box. They were fine before they were shipped and were broken loose during shipping.
Most of the problem appears to be solderability problems, not a problem with chip design or manufacturing.
Re:this doesn't seem accurate, it was solderabilit (Score:2)
Just take the cover off and press down firmly with a little wiggle....
Re: (Score:3, Informative)
Re: (Score:3, Insightful)
Bunni
Microsoft: Jack of all trades... (Score:2)
From what I understand (Score:2)
Re: (Score:2)
The small case and the lack of cooling is part of (Score:2)
Recall? (Score:2)
Some Facts... (Score:3, Informative)
2) Microsoft did design the GPU in concept, but worked with some bright people from ATI and other GPU gurus for the specifics. People can make fun of MS design a GPU, but this isn't their first time around the block, and also gave them the intimate change of pairing GPU hardware and OS technologies.
Look at the PS3, in addition the 'cell' processor that 'didn't need' a GPU to the shipping PS3 with the 'cell' and full Geforce 7800 in it, and yet between the two technologies it still can't hold framerates or do anti-aliasing like the Microsoft designed XBox 360. (See recent games like GTAIV where it runs at lower resolutions on the PS3.) (And I won't even go into how slow Blu-Ray makes the device for a game player being significantly slower than DVD and why MS refused to put HD-DVD or Blu-ray in the console as the primary drive. Gamers hate load times and crap framerates.)
3) The 3 Rings of Death is about the Thermal sensor plate and flexing due to high heat. 99.9% of the time. (Also the 3 Rings does not always mean death, most units continue to work once they cool down, etc.) (Google It)
4) As for MS Saving Money for using a non US fab plant and then having to move back to one, sure this is possible, but technically there would be little to no difference UNLESS Microsoft also changed the specification of the chip between the move process. I don't care if the fab plat has Donkeys and a Mule pulling carts out front, the silicon is created according to specification, and you don't get much more exact than this level of specificatinos.
The real story here would more likely be the plastic/plate fab company that was creating the inner X plate/case holder that was warping and causing the 3 Ring problem, a) it was the real problem not the chip and b) would more likely fail specs easier than silicon.
Re:Some Facts... (Score:4, Interesting)
The PS3 COULD run it in 360-resolution, but it might have to sacrifice some of those filters and special effects. I'd rather have a special effect laden game run at slightly lower resolution myself, as long as its hard to notice.
Re: (Score:3, Interesting)
Um, this is what PS3 owners like to tell themselves before they start crying at bed time maybe...
However, the PS3 is using a virtually off the shelf core Geforce 7800 GPU. The XBox 360 i
Re: (Score:2)
Re: (Score:2)
apology (Score:2, Funny)
I apologize for you getting rtbl'd.
Re: (Score:2)
Re:Feb. 2008 SquareTrade found a 16.4% failure rat (Score:2)
This is true of all products. Nothing lasts forever. In this case, not forever is a few months
Re:SLASHDOT SUX0RZ (Score:5, Funny)
Re: (Score:3)
Re:More info please (Score:4, Informative)
The General Hardware Failure error could be caused by cold soldering. The added mass of the CSP chips (including the GPU and CPU) resists heat flow that allows proper soldering of the lead-free solders underneath the motherboard.
Another General Hardware Failure is shown by the ring of light flashing one red light, and an error code E 74. This too renders the Xbox unusable.
The Nyko Intercooler has also been reported to have caused a general hardware failure in a number of consoles, as well as scorching of the power AC input.
An update patch released on November 1, 2006 was reported to "brick" consoles, rendering them useless.
In June 2008, the EE Times reports the problems may have started in a graphic chip.