Facebook VP Slams Intel's, AMD's Chip Performance Claims 370
narramissic writes "In an interview on stage at GigaOm's Structure conference in San Francisco on Thursday, Jonathan Heiliger, Facebook's VP of technical operations, told Om Malik that the latest generations of server processors from Intel and AMD don't deliver the performance gains that 'they're touting in the press.' 'And we're, literally in real time right now, trying to figure out why that is,' Heiliger said. He also had some harsh words for server makers: 'You guys don't get it,' Heiliger said. 'To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap, and they have to be super power-efficient.' Heiliger added that Google has done a great job designing and building its own servers for this kind of use."
You're Computin' for a Shootin' Mister (Score:5, Insightful)
You guys don't get it
Is it possible to take out a massive life insurance policy on Jonathan Heiliger?
To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap, and they have to be super power-efficient.
I assure you, despite your misconception that the world revolves around you everyone has those requirements. From the people who build supercomputers right down to the netbook I am typing on while watching Gurren Lagann.
Can we get like a panel of hardware engineers to have a discussion with this guy and can I get some popcorn?
Re:You're Computin' for a Shootin' Mister (Score:4, Informative)
Can we get like a panel of hardware engineers to have a discussion with this guy and can I get some popcorn?
Slashdotters might want to take a look at the details of the Google servers [cnet.com] to see what Heiliger is looking for. There's also a video tour. [youtube.com]
Re: (Score:3, Insightful)
and why have a PSU for each unit, why not just run 12v power rails to each server and do the ac/dc conversion on a larger transformer further down the line with larger batteries providng the back up for clusters of servers? after all no psu is cheaper than a psu with just the 5v taken out.
Re:You're Computin' for a Shootin' Mister (Score:5, Informative)
Re:You're Computin' for a Shootin' Mister (Score:5, Funny)
I vote for the seal clubbing thing.
Re:You're Computin' for a Shootin' Mister (Score:5, Insightful)
When you need the cheapest, most power-efficient servers you can find, to the point where you criticize your suppliers publicly, you're not willing to pay for the most expensive cables out there.
Besides, all the seal clubbers are buying those up.
Re: (Score:3, Funny)
But that will void your warranty!
Re:You're Computin' for a Shootin' Mister (Score:5, Funny)
Re:You're Computin' for a Shootin' Mister (Score:4, Informative)
I think they run AC to the row or rack of servers, then they have just one super efficient PSU powering all the servers in a rack rather than 42 separate power supplies (plus UL enclosures, connectors, extension cords, etc, etc)
Essentially Google builds "rack-sized" blade centers... or at least catching up to what IBM and HP are doing but on a bigger scale, like full racks or multiple racks managed at once rather than just one chassis.
I do agree that chip makers aren't thinking "big enough" with things like their Blade lines.. Google wants to see reference specs that include options for bare motherboards to slide right into your basic 42 unit rack with IO, disk and power all pulled out to the raw basics so Google can decide how to manage the bits rather than having stock OEM boards with such limited options. Google wants to manage a "rack" as a single machine and optimize power and parts across 40 servers as one group, not 40 separate little systems.
Re:You're Computin' for a Shootin' Mister (Score:5, Informative)
No, they don't [cnet.com]. They use motherboards built to their own specification that require only 12V power. This power is supplied by the server's own PSU, which takes 240V input. The PSU hooks into a 12V sealed lead acid battery, implementing UPS functionality (there is no centralized UPS).
I think it's a very elegant design.
Re: (Score:3, Insightful)
"Google wants to see reference specs that include options for bare motherboards to slide right into your basic 42 unit rack with IO, disk and power all pulled out to the raw basics so Google can decide how to manage the bits rather than having stock OEM boards with such limited options."
Sounds a lot like a VME backplane...
Re:You're Computin' for a Shootin' Mister (Score:5, Funny)
What's with everyone creating new units of measure? "We're going to need some 3 Seal Cable for this job!"
Re:You're Computin' for a Shootin' Mister (Score:5, Funny)
Re:You're Computin' for a Shootin' Mister (Score:5, Funny)
But ... but ... you'll need Cat 9 to get rid of the cat completely?
Re:You're Computin' for a Shootin' Mister (Score:4, Informative)
Re:You're Computin' for a Shootin' Mister (Score:5, Funny)
If we're gonna start the AC vs. DC war again, I call dibs on tasering the elephant this time.
Re: (Score:3, Insightful)
Most battery UPS's upconvert the 12VDC to 120VAC to provide a standard power supply that you can plug anything into. That's because most of them run off standard boat or motorcycle 12V batteries which you can get at your local car parts store. Diesel or Gasoline UPS's are electric generators and usually cost a *lot* more. They make sense for keeping an office building powered, but not for keeping just a computer or thirty up. And that's above and beyond the power losses from transmitting 12V over a distance
Re:You're Computin' for a Shootin' Mister (Score:5, Funny)
i don't know that i agree with some of googles design choices
I'm sure they'll get right on that, random slashdot guy...
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
that guy is an ass.
the latest generations of server processors from Intel and AMD don't deliver the performance gains that 'they're touting in the press
then
Google has done a great job designing and building its own servers for this kind of use
I wonder who makes the server processors for Google's servers. Hmmm.....
Re: (Score:3, Insightful)
Wrong.
He is criticizing, in the bits in TFS, two groups:
1) The marketing guys in two of the world's biggest chip makers (he's not complaining that the chips are flawed from an engineering perspective, he is complaining about the claims, which apparently conflict with Facebooks experience in testing them chips, about the performance of the chips), and
2) The people settin
Hm... (Score:4, Insightful)
To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap, and they have to be super power-efficient.
Hm, lets see... perhaps because Facebook and Amazon are niche markets? The average server isn't going to even need all the computing horsepower and the power efficiency is simply a drop in the bucket for most companies electrical bills. The average server is going to be much more I/O intensive than CPU intensive unless you do cluster computing or render a lot of stuff. The average server such as a web server or a file server doesn't use that much CPU and usually you are running 1-3 servers, not the hundreds that Facebook or Amazon would run.
And really, why is a VP complaining about this stuff? That he can't either afford custom solutions or spend the money buying more servers?
Re: (Score:3, Informative)
Hm, lets see... perhaps because Facebook and Amazon are niche markets?
-Maybe-. Even if they are a niche market, they're a big enough one to hold the attention of the big chipmakers.
A traditional business model might use large orders, especially advance orders, to offset or defray the cost of setting up a production line or facility, and get most of the profit from smaller sales. Or they may choose only to do production runs for large, inherently profitable orders. Even in a firing-from-the-hip model, large customers cost less per unit in marketing and sales than do smaller
Re:Hm... (Score:5, Funny)
Google says you lie.
http://xkcd.com/357/ [xkcd.com]
http://community.discovery.com/eve/forums/a/tpc/f/9701967776/m/4071911989 [discovery.com]
Re: (Score:2)
That he can't either afford custom solutions or spend the money buying more servers?
Tell me again what Facebook's revenue model is...??
Surely that's obvious (Score:4, Informative)
They collect a large amount of data on people and mine that for marketing information to turn around and target those same users.
It's the same model as google.
Re:Hm... (Score:5, Informative)
As someone who designs and deploys large storage environments for a living, I call BS. While the current generation of HBAs are 8Gb FibreChannel, I would say that the "average server" (as you put it) could happily live on a 1Gb HBA. Recall that almost all servers, or atleast those you care about, have DUAL HBA connections to their respective storage. So that's actually 2Gb of storage connectivity. Sure there are servers which have multiple HBAs, or use a higher utilization of the HBAs, such as database servers or backup/media servers. Most servers today are deployed with dual 4Gb HBAs as the 8Gb SFPs/optics are still quite pricey, and you cannot, in all seriousness, purchase 1 or 2Gb FC HBAs.
Even as we deploy VMware based servers, the VMware servers themselves tend to be more memory/cpu strapped than IO.
It would be very rare, or almost impossible for a server to be driving linerate HBAs, with still plenty of headroom left in the CPU. Even basic test tools like IOmeter require significant CPU usage to drive an HBA to capacity. And that is when it's writing/reading all zeros. It's doesn't actually need to do anything with the data. As would be the case if a database server was requesting 2Gb/s from a disk array, and then had to join/sort/add/whatever the tables retrieved.
Re: (Score:3, Informative)
As someone who designs and deploys large storage environments for a living,
Then you should know that throughput is not the only (or - typically - the most important) measure of IO performance.
Typical computing tasks tend to be I/O bound - specifically by random I/O performance. To a large degree, this is due to the massive disparity in performance improvements between CPUa and storage.
Re: (Score:2)
they may be "niche" but data centers are the top multiple unit customers, buying thousands of the exact same configurations from OEMS .... they've surpassed the "enterprise cube farm" desktops buy a good margin in that data centers buy the MOST expensive server CPU they can where Enterprises are buying the cheapest desktops they can... Data center customers and Gamers are the only ones that NEED new CPUS every six months... nobody else really cares as they're out buying no-profit Atom based netbooks now.
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
Kindof depends on how you read 'niche.' yes, there is a relatively small number of companies (customers) that have such requirements, but if each of them have a massive, massive number of servers, then i wouldn't call that niche any more, because it still represents a large turnover.
Facebook's application is poorly coded (Score:4, Insightful)
I have heard from some reliable sources that Facebook and Twitter's backend applications are poorly written.
Are Intel and AMD's claims overblown, sure what hardware manufacter doesn't cherry pick performance claims.
But I don't care what sort of hardware you through at crap code you are always going to get crap performance.
Re: (Score:3, Insightful)
Re: (Score:3, Insightful)
Not if your code is not tuned for this new "advanced hardware". Surely there are new compile flags to consider, and if you are not tuning your code for the new processor features it could very well be slower than before.
Re:Facebook's application is poorly coded (Score:5, Informative)
Facebook is written in PHP; there are no compile flags.
apache and the php engine have plenty of compile flags. not to mention whatever the database is.
Re:Facebook's application is poorly coded (Score:5, Informative)
# hdparm -Tt /dev/sdc
/dev/sdc: /dev/sdc | grep Model /dev/sda
/dev/sda: /dev/sda | grep Model
Timing cached reads: 5120 MB in 2.00 seconds = 2562.04 MB/sec
Timing buffered disk reads: 84 MB in 3.02 seconds = 27.77 MB/sec # hdparm -i
Model=ST3200822A, FwRev=3.01, SerialNo=xxxxxx
# hdparm -Tt
Timing cached reads: 6078 MB in 1.99 seconds = 3052.95 MB/sec
Timing buffered disk reads: 338 MB in 3.01 seconds = 112.22 MB/sec
# hdparm -i
Model=ST31000333AS, FwRev=SD1B, SerialNo=xxxxxx
It's not even a full order of magnitude faster, but 112MB/s is still nearly four times faster. And these are both magnetic discs, rather than SSDs.
Re: (Score:3, Informative)
That may be so. The new drive may indeed have four times the raw read throughput. But how much larger are they? Five times.
And even more tellingly, look at the seek performance. I looked up those two drives you mentioned. You'll find it's unchanged at 8.5ms. So we're seeking at the same speed, for more data.
In practice, then, in terms of throughput per provisioned GB, we are 24% worse off, and in terms of seek time per megabyte we are TEN times worse off today!
To illustrate what I mean, based on those
Re: (Score:3, Insightful)
Facebook is written in PHP
There's your problem right there... ;)
Re:Facebook's application is poorly coded (Score:4, Insightful)
Developers have been known to trade off performance for development ease. Frequently the result is crap code. Yes, it performs like crap on both sets of processors. But if the application is CPU-limited (rather than IO or memory or...), then throwing faster CPUs at it ought to make it proportionally faster, no? Obviously they thought the previous performance was acceptable, is it unreasonable to think that buying CPUs marketed as 50% faster should give a 50% performance increase? Clearly crap code will still run like crap, but you ought to be able to throw more CPU power at it and get 150% of crap performance.
Depends on 'headroom' of other subsystems. (Score:2, Informative)
Not necessarily, no.
It's all about how CPU limited the workload is.
You might be running a program that's CPU limited on one processor, then upgrade the processor and discover that it's suddenly discover that instead of being CPU-bound, now you're memory-bound. Or I/O bound. Or whatever.
Point is, just because you've hit the wall in terms of CPU doesn't mean you'll get a 50% improvement with a 50% increase in CPU ... you'll only get that if all the rest of the server's systems have 50% overhead to spare. An
Re: (Score:2)
Well, more generally, you have to apply Amdahl's law. Most complicated programs operate in a realm where there aren't well defined bottlenecks, and you can get more performance through more computer power, more memory, faster memory, faster disks, faster networking...
The point I was trying to make is that if you have crap code, and throw more compute power at it, it's unreasonable to expect it to perform like it isn't crap code -- but probably entirely reasonable to expect faster crap code.
Re: (Score:3, Insightful)
throwing proportionally faster CPUs at *good* code should make it proportionally faster.
Crap code.... probably not. For example, I once had to improve the performance of an app. The app spent most of its time context switching from one thread to another, more time was taken up stopping a thread, switching to another, refilling the cache lines, and so on that was spent processing the data! Think what a faster processor would do here - the CPU would process the little bit of data it was given faster thus prov
Re:Facebook's application is poorly coded (Score:5, Interesting)
One of the server techs from Twitter was at SXSW 2 years and gave some details about how their backend servers worked. If I remember correctly (there were 4 sites on the panel, so I may be confusing them with someone else), the original code was written in Ruby on Rails which did not scale well to the multi-server systems that they had setup. They have spent a lot of time improving their code over the years, but from what I could tell, their initial implementation wasn't the most thought out thing in the world.
Re:Facebook's application is poorly coded (Score:5, Interesting)
Given the quality of Facebook's developer API (it's horrible), I'd be amazed if the back-end of the actual site wasn't poorly written.
Well I suppose... (Score:4, Funny)
Well, I suppose that if he does not like the offerings from Intel and AMD, they could always go with...
Uh..
Oh.
Re:Well I suppose... (Score:5, Informative)
Re:Well I suppose... (Score:4, Interesting)
I was just going to say that. If Facebook et al are not looking at the Sun coolthreads servers, they're idiots. A T5240 would deliver a whopping 128 hardware threads per 1u of rackspace.
Re: (Score:3, Insightful)
Really? What sort of test was it?
We took a Java application off a E6900 using 35% of 48 1.35Ghz US-IV cores. We put it on a T5240 with 16 1.4Ghz cores we saw it only use 14% of the machine with improved user response time.
We also ran a database benchmark for some tests we were running between some AIX and Linux boxes and threw it against a T5240 running Oracle 11g for comparison. Because it was predominately a single threaded operation it ran slower than the 2.2Ghz Power5 LPAR, but the overall difference
Re: (Score:2)
Re: (Score:2)
Re: (Score:2, Informative)
>None of these offer much better performance. None.
There are IBM and Sun systems that are in an entirely different league, in terms of IO and memory bandwidth, than any Intel- or AMD-flavored system.
Re: (Score:3, Informative)
POWER6 absolutely ass-rapes Nehalem. Period. 4.7GHz (clocked up to 6GHz internally), faster per-cycle than any x86 processor currently on the market.
According to the SPECCPU2006 benchmarks, a 3.33Ghz Nehalem provides nearly identical performance to a 5Ghz POWER6 (@ 8 cores each).
Re: (Score:2)
Sun.... (Score:3, Insightful)
Re: (Score:3, Funny)
Sun?.... you are crazy to go with sun and there platform now.. its all dead now..
Larry? Is that you?
Re: (Score:3, Interesting)
Re: (Score:3, Informative)
the two points are somewhat independent of each other. The second I suspect is due to their being a lack of any standard for power efficent servers. Google did it by running single voltage power supplies. A standard around something like this would be useful, and not just for servers I suspect.
ARM? (Score:2)
I wonder when we'll see servers with CPUs based on (many...) ARM cores.
Yes, they are an order of magnitude slower, but three orders of magnitude more power efficient. For the same CPU performance you'd probably be around two orders of magnitude more power efficient (for CPUs at least). If your app runs on a large farm already...
ARM to the rescue? (Score:2)
"...To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap, and they have to be super power-efficient..."
Sounds like ARM processors are being described here. Whether they can deliver is another subject in itself. On this front, I have my doubts on ARM's ability to deliver. That's my bias.
Re: (Score:2)
As the AC noted, ARM's are available, cheap and power efficient. What they are not, however, is very powerful.
They have their uses, but interpreting large amounts of crappy scripts is not one of ARM's strong points.
Re: (Score:2)
However, aren't ARMs so cheap (also in production, transistor-wise) and power efficient that you can throw 10 times as many at the problem (hey, seems you're operating a farm already) and still decisively win on price and power?
Something about his arguement doesn't work (Score:5, Insightful)
1) Facebook & Amazon need cheap, power efficient systems
2) Intel and AMD aren't measuring up with processors to power these systems
3) However, Google has systems appropriate for this use (presumably using Intel or AMD processors)
If that's his argument, then it would seem that the real conclusion is that Facebook can't build systems as good as Google's, even though they are using the same processor technology.
Re: (Score:2, Insightful)
In addition, there seems to be something else wrong with his arguement
"To build servers for companies like Facebook, and Amazon, and other people who are operating fairly homogeneous applications, the servers have to be cheap"
Which he later follows up with the following insight
"There's a pretty simple answer for scaling infrastructure. It's, 'Don't be cheap,'"
so which one is it?
Re: (Score:2, Informative)
Re: (Score:2)
Of course, that's just my interpretation...
Re: (Score:2)
Both. Stupid people oversimplify. In this case, he did not mean cheap. He meant... damn, there is no English word for "preiswert". Essentially it is, when you get a good value for your money.
He meant that you should buy much (don't be cheap) with little money (the servers have to be cheap).
Of course, with that level of differentiation of thoughts, it's no wonder that he got problems. ^^
Would you expect otherwise (Score:5, Funny)
Google's core business is intelligence.
Facebooks core business is stupidity.
Re: (Score:3, Insightful)
If that's his argument, then it would seem that the real conclusion is that Facebook can't build systems as good as Google's, even though they are using the same processor technology.
Google does have approximately 30x as many employees as Facebook, so it's not implausible that they've got a much greater ability to build in-house custom tech.
PHP (Score:3, Interesting)
It's because your shitty website doesn't have a single line of compiled code. PHP only goes so far.
Re:PHP (Score:5, Interesting)
Re: (Score:3, Insightful)
Exactly. All these interpreted languages, even with some special tricks, will have serious scalability issues. At some point you have to look at the application and ask some serious questions.
Re: (Score:2)
Exactly. They should start small. Focus on performance profiling, find the most expensive parts of their code and reimplement those in a compiled language.
If they're using Zend and still can't get the performance they need, they just need to accept that PHP's flexibility and rapid development friendliness come at a price-runtime speed.
PHP "extension" (Score:5, Insightful)
I once did a large project in which I took a large, slow site in PHP (it was pretty complecated, it was a CRM with a lot of custom business logic) and rewrote all the core functionality from PHP to C / C++, and made it a "module" of PHP. The rewriting was mostly simple translation -- litterally removing all dollar signs, adding some types, and attempting to compile, and just fixing the compile errors until it would build. Then going back through it with a fine-tooth comb to track down all the memory leaks.
The speed increase from doing that is pretty surprising. Simple loops that do a bit of math or something speed up by 100 times, and a loop that creates and destroys an object within the loop will be 100,000 times faster. This is without actually trying to write fast C/C++ code, and not create and delete the same thing over and over in a loop -- just pure dumb translation of the code.
At that point, the web site guys can keep tweaking and changing the web page in PHP just like before; but they load that module in the php.ini and then they have a basic library of stuff, like login_user() or get_user_balance() and etc, that are really fast and do all the heavy lifting.
I would be surprised if Facebook has not already done this. How to do it is well documented in several books, and there are lots of PHP modules written in C/C++ to look at for examples.
I suspect that Facebook's VP is right that AMD and Intel exaggerate their claims, but is also generally true that most computer programs are more IO bound that you expect. This is not a reason to avoid something like I describe above; once you have the more complete control of programming in C, IO issues may be easier to find and address.
He also mentions that the servers offered by Dell and others aren't very power efficient or practicle for him, and he mentions Google designing their own servers. Nothing google did was really rocket science, from what we know, and Facebook probably doesn't have to go as far as they did to get a reasonable benefit. It's not that hard to set up motherboards to run without a case, booting off the network with no harddrive attached.
I wonder if he's not thinking about scaling enough (Score:2)
There's a pretty simple answer for scaling infrastructure. It's, 'Don't be cheap,'" Heiliger said. He added that Facebook does drive hard bargains with its hardware and software infrastructure suppliers, and is careful not to overbuy.
I remember reading about how
Re:I wonder if he's not thinking about scaling eno (Score:2)
Re: (Score:2)
Java? Yes, absolutely. Not sure about .net tho.
Sounds like a bunch of excuses to me (Score:5, Insightful)
Assuming that a solution was properly engineered, this should not have been a surprise.
Cheap. power efficient, performance. Pick two.
Re: (Score:3, Interesting)
Actually, Google got all three of those in their system-level design (when cheap is measured per CPU). What they didn't get was per CPU reliability. That's pretty miserable by the standards of commercial servers. Luckily, all Google software is architected, designed, and written to work around frequent hardware failures, so that's ultimately covered.
A Familiar Tune from Facebook (Score:4, Interesting)
Having a Bad Day? (Score:2)
He woke up on the wrong side of the bed, and then he had to sign the check for the electric bill.
He's just grumpy.
Rub a lamp, Heiliger (Score:5, Insightful)
NEWSFLASH! Customer are tightwads.
Performance/Reliability/Price.
Pick any two, Heiliger.
so what about google then? (Score:3, Insightful)
I'm bemused that he implies the problems with his servers are due to Intel and AMD no delivering with their chips, yet at the same time he admires google for how good a job they do in building out their machines.
he must be aware that google uses Intel and AMD chips.
his reasoning just doesn't square.
And yet... (Score:5, Interesting)
Every major server vendor has jumped on the bandwagon of 'look how efficient we are, and 'cheap'. Three years ago, by and large the tier ones wouldn't bother designing systems without forcing even the cheap design to have parts included to facilitate purchase of redundant add-ons (i.e. power distribution cards designed for dual power supplies regardless of one being bought or not). They would always put a high end storage controller on the planar. They would always make their 'entry' platform be burdened with expensive components to make it easier to option it up.
Now, we have tons of 'internet scale', or 'cloud', or whatever buzzword you feel like. They tend to stress energy efficiency, low cost components, with sales and management strategies targeted at thousands of servers (i.e. IBM iDataplex, HP SL6000). Basically, precisely what he prescribes, though probably not as 'cheap' as he wants. The incentive he gives is that the vendors should have zero margin, which is not particularly compelling for companies to work toward. Google's situation works because they brought it in-house and thus have fewer middle-men. Honestly, from all the rumours I hear, it's the logical thing to do when your server consumption is larger than some respectable computer companies' entire production. If he thinks the volume of servers is high enough to pull a google, by all means do it. Otherwise, be prepared for people not jump at the chance to give their designs to him at zero margin.
Of course, if he is calling them out on performance per-watt by avoiding non-x86 solutions, including ARM, that might be a fair criticism. However, I think company forays into 'exotic' architectures have not panned out in the market recently. Sun's niagra, despite all the worthy praise, couldn't attract a mass-market required to subsidize it for those who benefited most from it. Last year, IBM seemed to be saying Cell architecture would light the world on fire, but have been a lot quieter about it now. The message their buisness leaders have probably taken in is that while these things have their target market, that market isn't worth the expense of developing products that are refused by the larger market and focus instead on leveraging commonly accepted building blocks to do as best they can for that niche, even if it means skipping the 'perfect' solution. Sure, IBM still sells plenty of POWER, but I haven't heard that be *particularly* praised on the performance/watt category like I hear a lot for Niagra, Cell, and ARM. And if not for POWER's legacy, it probably would be still born in the market today. The PA-RISC->Itanium decision for HP probably sank their HP-UX product line faster than banking on legacy of PA-RISC installs, and it seems IBM won't make that mistake, but at the same time I don't hear much about *new* POWER customers.
Strange... (Score:5, Informative)
Since when do we listen to manufacturer's claims? You take the new hardware, stress test it with your custom software, record results, plan servers accordingly. How hard is it really to commission a server design that meets your needs and then QA some prototypes?
Re: (Score:2)
Mod the above up.
Intel, AMD, Sun, HP, Dell, IBM, whoever. They all provide a wide range of configurable, scalable, inexpensive (to expensive) building blocks to get your business need accomplished. With all the amazing things that have been created using computers in the last thirty or forty years, I can't blame the providers of those building blocks for something not coming together. The latest chips are absolutely faster than previous ones, but maybe you need to rethink your network fabric? Maybe your
Hang on a minute... (Score:4, Funny)
Hah, hah, hah, hah, hah !At least google needed to actually engineer their solution, but Facebook, come on ! The next time I need to write a PHP script for displaying photos and text, I'll hire my 13 year old daughter.
not just the CPU it's overall system performance (Score:3, Insightful)
This isn't just about the CPU, it's about overall system performance.
Despite improvements in CPU performance, memory and IO performance is lagging behind.
A modern SATA drive delivers about 90MB/sec ( peak sequential read ).
Some RAID controllers can do about 600-800MB/sec ( peak sequential read ).
An average AM2 ( K10 core 65nm ) gets about 34,849MB/sec L1, 12,169MB/sec L2, 6371MB/sec L3, 2,741MB/sec DDR2-800 5-5-5-12.
Obviously Opterons scale a lot better since they each have an onboard memory controller and additional HT links which greatly increases bandwidth as you add more CPUs. However adding more cores on the same die which have to share a single memory controller can cause starvation.
Another major issue is software parallelization, writing parallel code is still a difficult problem. If your software doesn't parallelize well it doesn't matter if you have 8, 16 or even 32cores on a single die.
If you had an equal number of CPU cores and memory controllers you could achieve much better performance, however your relatively very slow storage subsystems would still be a major bottleneck.
Re:WTF? (Score:5, Insightful)
Re:WTF? (Score:5, Interesting)
Looks like that to me; he scoped for cheap and cheerful and was bit on the ass when he realised that sometimes you get what you pay for. Like what's the point in having quad-core server CPU without the high-bandwidth buses of server-grade hardware.
In the concurrent DNS/Kaminsky thread, I saw a reference that facebook's DNS TTL is low. A quick investigation reveals that they have a 30 second TTL and are using DNS round-robin for their load balancing.
He's nothing but a blame-shifting cretin.
Re:WTF? (Score:5, Insightful)
I think we read different articles. He's not saying he didn't plan well enough, he's saying that Intel and AMD promise that Gen Y processor is 35% faster than Gen X processor, and he's not seeing anywhere near 35% in real world performance. The 35% is a made up number but it doesn't matter what the number is that they claim. He's probably correct. Manufacturers pull this stuff all the time. Look at the recent articles on battery life claims on notebook's. AMD came out and called BS on the whole thing and basically said if you guys don't stop lying through your teeth, the FTC is going to regulate us. From the perspective you are taking, that would mean we have to call AMD incompetent for not understanding how batteries work and not properly selecting them.
Manufacturers ALWAYS overstate claims in computer related products. CRT actual inches vs viewable inches (thank you lcd's for finally being honest... about inches anyway.. brightness and contrast however....) Computer speaker wattage being 1/2 or 1/4 of what's claimed. Power supply efficiency or wattage not measuring up to claims... you name it. He's calling out what he see's to be bogus claims based on a real world experiences in a demanding environment, the type of environment where one is always looking for better performance. I think we should get some more information before declaring him to be the problem as I'm sure he has a whole team of people that are working on this issue.
What I'd like to see from him is some numbers. On this Intel (or AMD) rig, we get so many operations per hour/minute/whatever. On this new Intel (or AMD) rig which they claim is 20% faster than the previous rig, we're only seeing this number of operations per hour which amounts to only a 7% gain, thus we're seeing 13% less than they are claiming. Again, numbers made up for examples sake. I'd also be very interested in what a typical rig of theirs looks like... X Processor, Y Ram, what type of storage system is it connected to, etc. I think such numbers are vital to understanding the issues at hand. We all know that vendors will run the benchmarks that makes their stuff look the best, and that is often not reflective of real world performance. If I was Intel/AMD I'd be chiming in right about now and opening a dialog with Facebook and looking to see what the issues are. Facebook is a big customer with huge name recognition and you want to be able to use them as an example of your solution delivering the promised performance for your marketing. I'm going to assume (I know I know) that they are already working with the server vendor to try and see what's going on here.
Re:WTF? (Score:5, Insightful)
I think we read different articles. He's not saying he didn't plan well enough, he's saying that Intel and AMD promise that Gen Y processor is 35% faster than Gen X processor, and he's not seeing anywhere near 35% in real world performance.
If the application was purely CPU bound, and Y wasn't giving me 35% more than X, I'd complain.
However, if it's a complex system like almost everything else, why would they expect their application to get 35% faster when there's probably 6 or 8 critical subsystems that could all be bottlenecks as well?
Re:WTF? (Score:5, Interesting)
One of the fun toys Intel has to play with is a complete system simulator, which simulates every single component in a computer for early testing. This lets them vary parameters that aren't feasible yet while they're working on their design goals. A few years ago they did a test; what happens to the system performance if you make the CPU infinitely fast? They adjusted the simulator so that every CPU operation took zero simulated time and ran their benchmark suite. It ran twice as fast (in simulated time) as it was before.
A CPU-bound workload can quickly become a RAM-speed bound or a disk-speed bound workload if you make the CPU faster but don't upgrade anything else.
Re:WTF? (Score:4, Interesting)
Uhhh, correct me if I'm wrong. I've been looking at after market bolt on parts for my car. The headers claim increase fuel mileage, the spark plugs, the air filter, the tires, as does a turbocharger. The glass pack mufflers, and the resonator. Oh yeah, the aerodynamic rims, the hood, and spoiler. Don't forget the carbon fiber body panels. Taken all together, those increased MPG's add up to about 150 MPG. You're saying I may not see that much improvement on my 1968 Chevy Malibu? It's just hype? Man - you just saved me about $5,000!!!
Re: (Score:3, Interesting)
How can you be blamed for finding an acceptable solution when there simply isn't one available? He is a software developer, not a hardware one. Not everybody can just go out and design their own servers like Google does. He's saying he's been tripped up by the fact that the server manufacturers aren't delivering on their promises; hardly something he should be blamed for. Your attempts to read more into his comment about "not being cheap" and compare it to the false words of a politician seems like a prett
Re: (Score:2, Informative)
I have some sympathy for this guy. Some years ago, I built a fileserver using the best SATA RAID (hardware RAID) cards I could find (~$300) from major manufacturers and enterprise disks (specified for use in RAID systems)
Performance absolutely sucked. The cards were fast enough it I tried to read/write single large files, but when reading/writing large numbers of small files, they were very slow. The first manufacturer's card was appallingly slow. I replaced it with another manufacturer's card and performan
Re: (Score:2, Insightful)
Re: (Score:3, Interesting)
No, he just found that RAID controllers suck. Which they do, universally, all the time. The only ones that actually perform decently are the ones in external SAN boxes, and inside they are typically servers with software RAID...
Re: (Score:3, Interesting)
You can better identify your bottlenecks by benchmarking. Facebook's scalability is likely not as cpu-bound as predicted, thus the dude's angst on discovering that CPU upgrades weren't a silver bullet.
In your case, you haven't looked past the RAID configuration for the root-cause of your performance issues. Without benchmarking you don't really know if it was an issue with: the filesystem, the block size, stripe size, or a caching tunable.
Systems architecture isn't as easy as PC builders would have you beli