Building The MareNostrum COTS Supercomputer

karvind writes "IBM Power Architecture Community Newsletter has a story about making a supercomputer (Number 4 on top 500 list) from easily available components (like BladeCenter and TotalStorage servers, 970FX PowerPC processors, and Linux 2.6). A joint venture between IBM and the Spanish government, it is named MareNostrum: the Latin term meaning 'our sea.' Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers."

specifically

[bersl2]

Mare Nostrum refers to the Mediterranean Sea.

Mare Nostrum

[nebaz]

Mare Nostrum literally means "our sea". It is what the Romans called the Mediterranean Sea during the Empire. As you can see [dalton.org], it was an apt name.

Re:Beowulf cluster?

[travisco_nabisco]

It probably has to do more with the fact that as you increase the number of nodes, your increase in performance decreases on a per node basis. To get that many nodes working together takes an incredible amount of resource management. It makes you wonder where the limit currently is for if it is worth adding an extra node, or if the resource management requirements negate the extra nodes computational power.

timely and focused PR

[bandix]

This is all about timely and focused execution. The speed at which this project was realized is important. Consider: from the initial concept in late December of 2003 to assembling the computer in Madrid took less than a year. Normally, this kind of supercomputer projects take years.

Lame!

SGI had NASA AMES' Columbia online in 120 days, and landed #2 on the Top500.

Re:War in the age of information warfare

[Anonymous Coward]

and crapflooding message boards

Yep, that's gonna cause mass mayhem and planet-wide panic. Them terrorists, messing up with our patriotic message boards! ... or something

Kid, when you grow up you'll relize there are some differences between a Beowulf cluster and a botnet of crappy PCs.

Re:IBM eServers?

[xose]

This was a political decision, HP already complained because there was not a sellers competition.

Last government of Jose Maria Ansar aka 'Estamos trabajado en elloooooo' brought this computer to try get the ITER.

PD: Africa comienza en los Pirineos.

Re:War in the age of information warfare

[grozzie2]

These things are already being used by rogue states. The us military has a bunch of them dedicated to modelling nuclear events.

With regard to denial of service attacks, there's a cluebox over in the corner, you need to go grab a couple out of the box. DOS attacks dont require a big computer, they require massive bandwidth with massive routing diversity available. The actual computer power required borders on insignificant. A supercomputer like this is useless for that kind of thing, by necessity, it will have an internal networking and communications environment, and likely only a relatively low speed interconnect to external networks.

But look on the bright side, the knee jerk 'terrorist behind every lamp post' reaction is just what the american government has been trying to instill in the population for the last few years. Your post here shows, it's been an effective campaign, money successfully spent, and the objective achieved. It's become the 'trendy' response to just about everything these days.

Re:Off who's shelf?

[Anonymous Coward]

Can you even buy bare PPC CPUs and mobos?

Not 970-based yet, but anyway:
Pegasos [pegasosppc.com].
Terons [mai.com] (which are also marketed, by raping the corpse of the Amiga, to a bunch of clueless zealots as "AmigaOnes". The CX and PX models are discontinued due to hardware flaws, the jury is still out on the newer Mini model).

Re:Who cares about easily available?

[jokumuu]

Well, if you really need a supercomputer, the first step is not to get the computer, but instead the funding. As super computers go, if your needs are served by a distributed computing environmet, an "entry level" supercomputer does not really cost all that much compared to "traditional" supercomputers. Yes we are still talking about hundreds of thousands of dollars. (there seems to still be 256 processor 2.8ghz xeon based computers with gig ethernet connectivity on the latest top 500 list) If you need to solve problems that do not work in paralell, you are definitely out of luck.

Re:Beowulf cluster?

[photon317]

It's really hard to scale these kinds of clusters beyond certain limits, once you start getting into all the practical details.

There's the basic facilities stuff: fitting enough power and cooling density into a single datacenter to cram that many nodes within a reasonable distance of each other for cabling purposes.

There's the network architecture. A single switched network between your nodes doesn't get you all that far. Depending on the characteristics of the expected workload and all that jazz, there are many different technologies and topologies to choose from.

Don't forget storage and data moving in general. The data has to reach the processors somehow, and 2000 nodes mounting an nfs share from some central box just isn't going to work at all...

Then there's node management: installing/imaging, booting, detecting failures, recovering with minimal human intervention (automatic re-imaging), monitoring it all, etc. You could skip this step and hire a truckload of junior sysadmins and have them running all over the place with CDs and keyboards and monitors, but that doesn't really scale to thousands of densely packed nodes does it?

As another reply states - if you don't find the right solution to all of these problems, you face scalability limits. With a given overall design, there's going to be a maximal node count, beyond which scaling is infeasible or futile. It really is hard stuff.

Luckily the opensource world is making headway on some of the software-side manageability issues. For an example check out rocks.npaci.edu.

Re:Beowulf cluster?

[antifoidulus]

Eh, not really. In a lot of the algorithms, there comes a point when adding nodes will make it SLOWER because the increases in communication time are greater than the decrease in computation time. Now granted this does depend a lot on a) what you are doing with the machine and b) the machines themselves, but just thinking that people who make these things love to just pile on hardware is a bit naive....

Re:Beowulf cluster?

[Anonymous Coward]

VT got Dual 2.3Ghz XServes a couple of months before they were publically released, but they are publically available now.
You can go to the online Apple store and order some if you like, it says they'll ship it on the same business day.

Re:Off who's shelf?

[blamanj]

Well, the info is right here [ibm.com] for quantity 1, and there a button that says "Configure and order a JS20," so if you're willing to order a few thousand of them (they're about $4K/ea with 2.5G RAM), you can build your own.

Actually, that's a reasonable price, considering it's IBM, who aren't usually considered a bargain brand.

Re:Off who's shelf?

[clem.dickey]

You'll need a chassis for that blade, of course. Luckily, they're half-off through the end of March. Buy the entry model [ibm.com] and get change back from your $1000 bill. Oh, and you'll want a rack [ibm.com] to put the chassis in. But at $1489 for the rack, check the local surplus store first. And while you there pick up a display, mouse and keyboard.