Building The MareNostrum COTS Supercomputer 187
karvind writes "IBM Power Architecture Community Newsletter has a story about making a supercomputer (Number 4 on top 500 list) from easily available components (like BladeCenter and TotalStorage servers, 970FX PowerPC processors, and Linux 2.6). A joint venture between IBM and the Spanish government, it is named MareNostrum: the Latin term meaning 'our sea.' Peaking at 40 TFlops, the beast consists of 2,282 IBM eServer BladeCenter JS20 blade servers housed in 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers."
specifically (Score:5, Informative)
Re:specifically (Score:2)
Re:specifically (Score:1)
Mare Nostrum (Score:5, Informative)
Re:Mare Nostrum (Score:1)
Re:Mare Nostrum (Score:1)
Humans are so behind the curve (Score:5, Funny)
(I tried to register an account but
Re:Humans are so behind the curve (Score:2)
So celebrate all ya want, bipod just don't run out of my crackers!
I have an account now! *squawk* (Score:1)
Re:Humans are so behind the curve (Score:3, Funny)
What kind of troll would parrot back such a hackneyed old line.
Re:Humans are so behind the curve (Score:3, Funny)
It's all fine (Score:5, Funny)
Why ohh why (Score:2)
Now im going to cry!
Re:Why ohh why (Score:2)
Besides, last I checked (back when I bought mine) there is a 6-8 week waiting list for Xserves. (Confessions of an Apple Fan Boy.)
Top 500? (Score:4, Funny)
This is like those CDs that have 'best of the Top40' and not contain the top10 list of that
and in 32 years (Score:3, Insightful)
Re:and in 32 years (Score:2)
Beowulf cluster? (Score:2)
Re:Beowulf cluster? (Score:2, Informative)
Re:Beowulf cluster? (Score:4, Interesting)
If you're the buyer, there is no limit. Every extra node == _MORE_STATUS_
If you're the guys writing code for it, there is no limit. Every extra node == job security++
If you're the people administering this, there is no limit. Every extra node == bigger budget next year
See, citizen? Size does count.
Re:Beowulf cluster? (Score:4, Informative)
Re:Beowulf cluster? (Score:2)
Or a bit brazenly honest. There's no limit to the benefit from scaling your cluster up, since
(1) there are always applications that run better on more cpus, even with a crappy network design.
(2) there are always better network designs that improve the remaining applications.
(3) there is always the option of partitioning it into mutiple sub-clusters if you're too dumb for (1) and too cheap for (2).
Re:Beowulf cluster? (Score:1)
Re:Beowulf cluster? (Score:4, Insightful)
Re:Beowulf cluster? (Score:2)
Not at the time.
The cluster was announced as G5 but they used G4 for the first incarnation and later performed a system-wide upgrade (or two, who knows
http://www.thinksecret.com/news/virginiatech3.h tml
Note how Apple's site (http://www.apple.com/education/science/profiles/v atech/) talks about 2.0GHz, and both 2.3GHz and later 2.5GHz processors were used...
And this* "These systems were custom built by Apple for Virginia Tech utilizing dual 2.3GHz G5 proces
Re:Beowulf cluster? (Score:1, Informative)
You can go to the online Apple store and order some if you like, it says they'll ship it on the same business day.
Re:Beowulf cluster? (Score:2)
Re:Beowulf cluster? (Score:2)
You should be lucky I chose to berate you for mistaking Thomas Jefferson Community College for Virginia Tech instead of moding you down.
Aside from that gaff, however, you are essentially correct. The VT cluster may as well have been OTS, even if the units were built especially for VT. I think most of the "custom" part of the custom G5s was simple packaging and the fact that VT got the first run of chips. (don't quote me on that one...I just know what I've read in the newsletters).
Re:Beowulf cluster? (Score:2, Informative)
There's the basic facilities stuff: fitting enough power and cooling density into a single datacenter to cram that many nodes within a reasonable distance of each other for cabling purposes.
There's the network architecture. A single switched network between your nodes doesn't get you all that far. Depending on the characteristics of the expected workload and all that jazz, there
Re:Beowulf cluster? (Score:2)
for which logarithmic cross-section bandwidth growth is sufficient, there are well-understood architectures that scale without bound.
This stuff was cute and clever in 1992, but these days it's a pretty well-trodden path.
Re:Beowulf cluster? (Score:2)
At least in the world I play in, logarithmic cross-section bandwidth is nowhere near sufficient. The interconnect design and technology is still very critical for us, as is storage.
Off who's shelf? (Score:4, Insightful)
Right, like I regularly go to Fry's to stock up on some DS4100s and Bladecenters. I'd love to be the geek for whom that stuff is "off-the-shelf". Can you even buy bare PPC CPUs and mobos?
Re:Off who's shelf? (Score:3, Insightful)
Just because you "can't" buy the stuff in pieces (IBM will probably gladly sell you the stuff as spare parts, if you prefer to assemble it yourself) doesn't mean it's not off the shelf.
Re:Off who's shelf? (Score:1, Informative)
Not 970-based yet, but anyway:
Pegasos [pegasosppc.com].
Terons [mai.com] (which are also marketed, by raping the corpse of the Amiga, to a bunch of clueless zealots as "AmigaOnes". The CX and PX models are discontinued due to hardware flaws, the jury is still out on the newer Mini model).
Re:Off who's shelf? (Score:2, Insightful)
$775 [ultraspec.us] for a G4 (1GHz?) on an ATX board with specs comparable or less than a $112 [newegg.com] Athlon64 motherboard.
$3,900 [mai.com](!) for a board with a 750FX processor (unknown speed) and technology comparable to what was going out of style 4 years ago. (USB1.1? 10/100 Ethernet? PC133 Memory???)
I'm not holding my breath for these. Call me when I can get a decent motherboard with a 1GHz processor for less than $200.
Re:Off who's shelf? (Score:3, Insightful)
Re:Off who's shelf? (Score:2)
Re:Off who's shelf? (Score:2)
Another example would be ordering a Dell. While there are Dells that are available retail, most of them can only be gotten directly from the manufacturer. The same is true for High-end Macintoshes, most mainframes, and a good deal of tape library market.
Re:Off who's shelf? (Score:2)
Right, like I regularly go to Fry's to stock up on some DS4100s and Bladecenters. I'd love to be the geek for whom that stuff is "off-the-shelf". Can you even buy bare PPC CPUs and mobos?
Let alone Linux 2.6! Where the hell does one find that?!?
Re:Off who's shelf? (Score:2)
Re:Off who's shelf? (Score:3, Informative)
Actually, that's a reasonable price, considering it's IBM, who aren't usually considered a bargain brand.
Re:Off who's shelf? (Score:3, Informative)
Re:Off who's shelf? (Score:2)
Actually, I don't. I just clicked a link in one of the IBM articles about the cluster and got to the JS20 page.
I do know that blades are designed to be small. The enclosure is 7U and you can put 10 blades in it, because they're mounted vertically. In the same space you can only fit 7 XServes, which are mounted horizontally.
I don't know details about heat, etc. I think the JS20 defaults to no HD, as well, whereas the Xserve comes with 80G minimum.
Cue the lame gags... (Score:1, Funny)
Re:Cue the lame gags... (Score:1)
War in the age of information warfare (Score:5, Funny)
With supercomputing powers now avaible to any country or group with a few readily available components, it is only a matter of time before these supercomputing powers may be used by a rogue state [korea-dpr.com] or radical group to cause havoc among electronic communications using methods like denial of service attacks, spyware, and crapflooding message boards.
I think it is high time the nations of the world put their heads together and addressed this issue. For example, I don't think the US Federal Government even has any cabinet-level position like Secretary of Information Technology or something like that. When are they going to get with the times? It will probably take another terrorist attack or something.
Re:War in the age of information warfare (Score:2, Insightful)
Someone's been watching too much 24.
Re:War in the age of information warfare (Score:2)
Re:War in the age of information warfare (Score:1, Informative)
Yep, that's gonna cause mass mayhem and planet-wide panic. Them terrorists, messing up with our patriotic message boards!
Kid, when you grow up you'll relize there are some differences between a Beowulf cluster and a botnet of crappy PCs.
Re:War in the age of information warfare (Score:5, Informative)
With regard to denial of service attacks, there's a cluebox over in the corner, you need to go grab a couple out of the box. DOS attacks dont require a big computer, they require massive bandwidth with massive routing diversity available. The actual computer power required borders on insignificant. A supercomputer like this is useless for that kind of thing, by necessity, it will have an internal networking and communications environment, and likely only a relatively low speed interconnect to external networks.
But look on the bright side, the knee jerk 'terrorist behind every lamp post' reaction is just what the american government has been trying to instill in the population for the last few years. Your post here shows, it's been an effective campaign, money successfully spent, and the objective achieved. It's become the 'trendy' response to just about everything these days.
Didja miss the 'funny' tag there mate? (Score:2)
Bit of hint there that the grandpappy was sending up the 'terrorists are everywhere' idiots ;-)
J.
Re:War in the age of information warfare (Score:2)
While the parent poster may have gotten the details wrong (you are correct in that massive DOS requires distribution and bandwidth, not computing power), the manner in which you dismissively respond was unwarrented.
Yes, the US Government has certainly been doing a lot to enforce the idea that Terrorists are Out Ther
Re:War in the age of information warfare (Score:2)
Not if the algorithm is good. Time and energy for a brute-force crack increases exponentially with key length; the energy output of the Sun over its entire lifetime isn't enough to even count to 2^256, let alone crack a 256-bit k
Re:War in the age of information warfare (Score:2)
Re:War in the age of information warfare (Score:2)
You are SO informed! That is exactly what you can NOT use a supercomputer for! Supercomputers are used for scientific calculations. Sure you can use them to calcul
Re:War in the age of information warfare (Score:1)
Re:War in the age of information warfare (Score:1)
Re:War in the age of information warfare (Score:2)
This got me into thinking, too - with posts like this, rogue states will quickly realise they can save some of that computing power just by leaving slashdot alone.
Re:War in the age of information warfare (Score:3, Interesting)
Next there have been several CS security czars, but all have quit because they have not had the response from the admin that they thought was needed.
Will it take a terrorist hit to get us really thinking about all this? Nope. In spite of 9-11, we are still not really any more "secure" than we were on 9-10.
What it will take is lawsuits against companies. A good one that is going through now, is the one where a guy got r
Re:War in the age of information warfare (Score:2)
Look, North Korean secret agent guy, just because you have a supercomputer doesn't mean you need to go posting this ton of crap to Slashdot thanks very much.
That's almost as ludicrous as Intel's faster CPU to surf the web faster theory. The trolls on slashdot are ample proof that you need nothing more than a low end pentium and lots of time to keep the drivel flowing all day.
Re:War in the age of information warfare (Score:4, Insightful)
Keep in mind that these are Top500 KNOWN supercomputers.
It's quite possible that many more computers that exist but are operating under classified conditions. For instance I would be suprised if the NSA had something for breaking crypto that rivaled some of the machines in the top 50 or so super computers, but it's not something that would appear on this list. This is for boasting rights only, if a place chooses not to publicize their computer, it won't end up on the list.
American computers pretty much dominate. Some like the Spainish one are built by American companies (like IBM), but just happen to be installed in foreign places.
Now the NEC Earth Simulator is a BIG exception. It dominated the top 500 for a considurable period of time and is completely japanese built, designed and owned. It is a huge technological acheivement and the pinnacle of technology for it's day. It's hard to relate to how significant it was. It is the top of the line when it comes to old-school massively parrallel supercomputers, blew American supercomputers out of the water.
But along came Linux clusters (which in many ways is competely unsuitable for some of the things that Earth Simulator can do) and now Blue Gene. Which can possibly do twice the work as Earth Simulator, but consumes a fraction of the power and space needed to house these types of computers.
Re:War in the age of information warfare (Score:2)
Re:War in the age of information warfare (Score:3, Insightful)
I shudder to think of what a "Secretary of Information Technology" would be. In my tinfoil sheilded skull I imagine a jack boot on
Cold War Paranoia about Foreigners' Computers (Score:2)
Anyway, the Feds had a bunch of export control laws to prevent Commies from getting Big Computers that they could use to design better Nukes, as well as laws to prevent Commmies (and Americans) from getting crypto.
timely and focused PR (Score:5, Informative)
Lame!
SGI had NASA AMES' Columbia online in 120 days, and landed #2 on the Top500.
Re:timely and focused PR (Score:2)
SGI had NASA AMES' Columbia online in 120 days, and landed #2 on the Top500.
I am sure that what they mean is that every phase of the project was according to schedule, specifically including the PR efforts.
Sounds like (Score:5, Funny)
Sounds like the specs of Microsoft's Xbox 3...
Re:Sounds like (Score:2)
WHAT'S WITH THE FREE ADVERTISING? (Score:2, Insightful)
The MareNostrum supercomputer at the Barcelona Supercomputing Center, ranked number four in the world in speed in November 2004, is constructed of such totally off-the-shelf parts as IBM BladeCenter JS20 servers, 64-bit 970FX PowerPC processors, TotalStorage DS4100 storage servers, and Linux 2.6
That's it - the first sentence of the article, if you exclude the title, the
Who cares about easily available? (Score:1, Insightful)
Screw easily available. Anyone with a budget to buy 2,282 servers and 4,564 processors can afford custom parts.
Call me when it's also easily affordable and I can pick up the parts at my local Fry's, or better yet, my local supermarket.
The only real item of interest was that it was made with all IBM parts and Linux.
...
Come to think of it, if Fry's also sold groceries I won't have to shop anywhere else...
Re:Who cares about easily available? (Score:3, Informative)
Re:Who cares about easily available? (Score:2)
Re:Who cares about easily available? (Score:1)
Re:Who cares about easily available? (Score:2)
Ray Kurzweil told me so.
In other news, SHA1 was broken... (Score:2, Funny)
It's just the thing to find SHA1 collisions of ISO images in 2^56 operations...
Well yeah,... (Score:2, Funny)
Re:Well yeah,... (Score:2)
That's a beefy computer (Score:2, Insightful)
but I bet Windows still runs slow on it.
Re:That's a beefy computer (Score:1)
DOES THAT COME IN A DESKTOP? (Score:1, Funny)
Supercomputer = Pile of computers? (Score:2, Insightful)
Re:Supercomputer = Pile of computers? (Score:2, Funny)
Haha, but the funny thing is that they'll end doing high performance computing based on Web Services!!
Best regards,
diego.
riiight . . . easily available components (Score:2, Insightful)
In what world is 163 BladeCenter chassis, 4,564 64-bit IBM PowerPC 970FX processors, and 140 TB of IBM TotalStorage DS4100 storage servers easily available??? Maybe if you are Big Blue, but then, why would it be more difficult for
Re:riiight . . . easily available components (Score:3, Insightful)
But really... (Score:2, Funny)
Even easier then that... (Score:1)
Add more Xserves to get more power...
Mare Nostrum (Score:3, Funny)
But I thought it was Ouray Easay!
What's going on?
This is all so confusing! I need to take a nap.
Re:Mare Nostrum (Score:2)
No memory specs? (Score:3, Interesting)
It mentions how many nodes, how many CPUs, how many racks, how much storage, but not how much RAM.
OK then (Score:1)
CELL Supercomputer (Score:2, Interesting)
Granted, the CELL isn't exactly off the shelf, and I'm willing to bet 4,564 970FXs will be cheaper than 1600 CELLs for quite some time, so the project still has merit.
Re:CELL Supercomputer (Score:2)
All it takes anymore... (Score:2)
Sounds to me like building the biggest supercomputer today is nothing more than throwing more money at it than your compeition. No real technical skill needed beyond beyond that of wiring up your new home entertainment center a few hundred times over.
I will help out (Score:2)
Re:Off the shelf (Score:2)
Re:IBM eServers? (Score:1, Informative)
Last government of Jose Maria Ansar aka 'Estamos trabajado en elloooooo' brought this computer to try get the ITER.
PD: Africa comienza en los Pirineos.
Re:IBM eServers? (Score:1, Funny)
umm
Really? ALL of them, no kidding
Re:Imagine... (Score:2)
--Ender
Re:Imagine... (Score:2, Funny)
Re:Imagine... (Score:2)
Mycroft