Earth Simulator, G5 Cluster Drop In 'Top 500' List

daveschroeder writes "The November Top 500 supercomputer list has been published at SC2004. Topping the charts is IBM and the US Department of Energy's 'BlueGene/L DD2' beta system, at 70.72 TFlops, followed by NASA's 'Columbia' at 51.87.TFlops. For the first time in several publications of this list, Japan's Earth Simulator is no longer in the number one slot, falling to third. Virginia Tech's 'System X' Xserve G5 cluster, while 20% faster than the original cluster that debuted at number 3 last November, has fallen to number 7 due to the new entries, but remains the fastest supercomputer at an academic institution. Here's an excellent cost comparison (Google cache) of the top machines ('System X' is significantly cheaper than anything else in the top 20, not to mention cheaper than many things far below it in performance)."

VA Tech Supercomputer

[dretay]

I was down at Virginia Tech last year when I was looking at colleges. They would not let anyone near that computer. Even the guy who was giving the tour was complaining about the limited amount access Tech students were given. The main reason he cited was that the companies developing the supercomputer had technology that they didn't want people who had not signed NDA's to see. Anyway, the point was that while the computer may be owned by the university, students aren't even allowed to see what $5 million of their tuition bought.

Re:Power architecture does well

[Brett Johnson]

It is actually 6 of the top 10, and 13 of the top 25.

CPU benchmarks

[3770]

You can compare CPU benchmarks here [spec.org].

AMD is beating the crap out of Intel.

Don't get carried away with LINPACK flops/cost!

[Richard Mills]

You raise good points, and the team at Virginia Tech did do something remarkable. That said, cost per flop of the LINPACK benchmark is interesting but not particularly meaningful. For instance:

"Another example is PNNL's 1936 processor Itanium2 cluster: 3.5Tflops less performance than System X, for $25 million"

What is not captured by the LINPACK scores is that PNNL's machine will absolutely spank the BigMac cluster at what the PNNL machine is intended for: running computational chemistry codes such as NWChem. A lot of the cash for the PNNL machine went into large memories and fast I/O that simply does not show up in the LINPACK benchmark. Furthermore, there are a lot of very high-profile scientific publications that have come out of the computational chemistry abilities of the PNNL machine. That's something else extremely important that doesn't show up in the rankings.

There are a lot of similar examples, but the PNNL one is one that I know something about, so I chose it. Basically, I'm saying to not read too much into those cost comparisons. It really is comparing Apples to oranges... er, HPs in this case. =)

Re:Power architecture does well

[Brett Johnson]

I think it is interesting that 11% of the top 500 are Power architecture, and 64% of the top 500 are intel based systems. Yet 50% of the top 10 are Power architecture and only 20% of the top 10 are intel architecture. Also interesting is that the Power based systems seem to have twice the Mflop/dollar ratio over the intel systems.

Re:power costs?

[daveschroeder]

That's easy...the Xserve G5s consume a LOT less power (and therefore generate less heat, resulting in lower cooling costs) than any competitive products (Xeon, Itanium2, Opteron)...and this was true even when they were using the 970 (as opposed to the 970fx they are using now).

Several of the researchers at Virginia Tech have referred to this in various news stories numerous times - one estimate was over two times less power than comparable systems.

Re:VA Tech Supercomputer

[jdog1016]

It's not even technically on campus. Its at the nearby Virginia Tech Corporate Research Facility. And in any case, you can arrange a tour [vt.edu] if you want.

Re:CPU benchmarks

[Anonymous Coward]

This is the best specint AMD has to offer (yes, the fastest operon is slower):
AMD Athlon (TM) 64 FX-55 1 core, 1 chip, 1 core/chip 1750 1854

Here is the best intel:
3.4 GHz, Pentium 4 Proce 1 core, 1 chip, 1 core/chip with HT Technology enabled 1667 1705

Here are the best specfp numbers for AMD:
AMD Athlon (TM) 64 FX-55 1 core, 1 chip, 1 core/chip 1741 1782

Which just edges out the best Pentium
3.6GHz Xeon) 1 core, 1 chip, 1 core/chip 1700 1721

But gets beaten by a low end I2
1300 MHz, Itanium 2 1 1808 1808

Also by a Fujitsu SPARC
PRIMEPOWER900 (1890MHz) 1 core, 1 chip, 1 core/chip 1510 1803

Gets trounced by the second-best I2 available
1500 MHz, Itanium 2 1 2161 2161

And gets demolished by the top end POWER5
1900 MHz, 1 CPU 1 core, 1 chip, 2 cores/chip (SMT off) 2576 2702

Re:EARTH TO MAC ZEALOTS:

[Anonymous Coward]

I don't know how you can say that They aren't easy to do upgrades. The 128 processor single image system I maintain is as easy as upgrading a workstation. And the applications generally don't require any optimization just to get running. Most cases it is as easy as running on a workstation, just alot bigger.

Re:Pizza arguments

[CODiNE]

I remember the power usage and heat generation was lower than the competition... they were able to save a lot of money on cooling systems by going with the G5's. That has to be saving them a lot of money day to day. I don't have the wattage numbers on hand right now, but I do remember G5's beating P4's, Opterons and especially Xeons, so there's no way Virgina Tech is paying what those x86 top ten people are in energy bills. The XServe's keep track of their fans and internal temperature, automatically letting the admin know when a machine is GOING TO FAIL and preemptively swap out faulty components before they cause serious damage to the systems. I don't know what sort of Apple Care that kind of system comes with, but I'm sure it's competitive with their consumer systems, I'd imagine any hardware failures are completely covered absolutely free for at least the first year, probably three or more if they pay Apple for any support.

-Don.

Re:Pizza arguments

[hernick]

A major cost is power and cooling requirements. According to Apple, a single drive, dual processor 2.0GHz XServe will use about 250W peak. Virginia Tech has 2.3GHz machines and Infiniband PCI-X cards. There's also networking gear and other support equipment to consider. So, we'll use a high figure of 350W per node, giving us a 40% overhead. As for heat production, same overhead, 1200BTU/h per node.

We're going to consider the worst-case scenario, under which we have a 100% load, year round, on all 1100 nodes. That gives us a power consumption of 385kW and 1320kBTU/h of heat generation.

Now, we need to get rid of that heat, and that's going to require a lot of power. My research indicates up to 300kW may be required, but that's a high number and actual requirements may be lower.

So, here we are, with 685kW required for power and cooling. That means a 6000MW/h a year.

Now, the cost of power is high, since you need to amortize and maintain the UPS equipment and the generators. We'll use a figure of 0.15$/kW/h, or 150$/MW/h. Very generous.

So here we are. The absolute worst case for power and cooling. Full load, year round, expensive cooling, overpriced power and amortized UPS and generators.

900 000$/yr. Below a million. It's not that bad, is it ? The real cost is likely below a half-million.

As for the rest, well, how much pizza is really required to entice graduate students and professors to work on that machine ?

Re:Pizza arguments

[Anonymous Coward]

"And on top of it all, last November, they hit #3 in the world, #2 in the US, and #1 academic, as well as the first academic site to ever exceed 10Tflops, all for less than $7 million in total - including all improvements to buildings, physical plant, and other infrastructure."

I'm not trying to take any thing away from the VT cluster, but the prices quoted on the earth sim always include the total cost of the specifically designed, earthquake proof building and everything associated with the project, whereas prices for the VT cluster negate this and only count the hardware VT had to pay for - none of the freebies.

How is it one can get a dell for less than an apple but mysteriously as soon as it becomes a cluster (using the same interconnect) the G5 is cheaper? apples to apples....

#4 is also academic

[az4+h0th]

Just a minor comment...

I work at UPC and there has been a lot of hype here for machine #4, which is (or is going to be) a >4500 PPC970s machine running linux (nice work, ibm). I disagree with the claim that the Virgina Tech cluster is the first academic supercomputer. As far as I'm concerned the Technical University of Catalonia (UPC) is also an academic institution.

Anyway. we now got europe's fastest supercomputer. That's what matters. ha! ;-)

Re:Hmm

[dustpuppy_de]

Deep Thought, actually. This supercomputer calculated that very number, as the answor to the Ultimate Question of Life, the Universe and Everything. Earth, Deep Thought's successor, was pretty close to calculating the question, without which the answer obviously doesn't make very much sense - but as we all know, it was destroyed by the Vogons to make way for a new hyperspace bypass.

More Information here. [wikipedia.org]

Re:CPU benchmarks

[Rufus211]

Usefull site for getting usable spec numbers is ace's SPECmine [aceshardware.com]. Try SPECint2000 or SPECfp2000 and check "CPU MHz". The opteron's fare quite nicely.

Virginia fastest academic?

[shiruf]

Virginia Tech's 'System X' Xserve G5 cluster [...] remains the fastest supercomputer at an academic institution.
Beg to differ: #4 is about 5 mins from home (by bus), in the northern campus of universitat politècnica de catalunya. And, yes, part of the institution, not some loaned space or something. Mind you, one wishes Spanish Universities involved their students a tenth as much. S-2.

Coup? Cuckoo!

[PingPongBoy]

5.7 TFlops in less than a square metre [nwfusion.com] coming to a store near you, an IBM Blue Gene. If these start appearing at Wal Mart, it's time to redefine supercomputer.

Re:EARTH TO MAC ZEALOTS:

[UnknowingFool]

System X is not a single system image machine. It will not work well for problems requiring low internode latency and high internode bandwidth in comparison to a real supercomputer.

Christ, clusters are not the end all and be all of high performance computing systems.

I guess I don't get the arugment because many of the other entries are clusters and not not single image either. Of course, given OS and architecture differences, all the supercomputers may perform differently in real world applications than the benchmark tests. The point is that System X was built using off-the-shelf components at a fraction of the cost of comparable systems. The entry right above System X is the US Army cluster of Apples (Mach 5). Obviously Apple is starting to be considered for high end computing.

In your face, Amiga! In your face! :)

Re:Pizza arguments

[frankie]

Virginia Tech paid Apple's standard educational price [google.com], which is about a 10% discount on retail (and tax exempt). The only special break from Apple was that VTech got the first 1000 G5 towers off the assembly line, before other customers and stores.

Re:need?

[Kyusaku Natsume]

It's because the die size is smaller, the G5 has less surface to dissipate heat, that's the reason for the high eficiency liquid cooling.