Wanted: Turn-Key 10-Node Beowulf Cluster 239
forgotten password writes: "I'd just started working on my morning M&Ms, when I
was asked where my group can buy a good turn-key ~2CPUx10-node
Beowulf cluster in two hours. I suspect the time frame is
longer than that, although the window-of-opportunity for
the money is apparently on the order of days, and a quote
before the procurement meeting would help. Any ideas?
Who's good? What it should cost? Thanks!" If you're quick, maybe you can become the world's newest manufacturer of custom beowulf clusters.
Atipa (Score:4, Informative)
Re:Atipa (Score:2)
Re:Atipa (Score:3, Informative)
If you are interested, SGI can sell you turn key beowulf solutions also. You can also go to http://www.beowulf.org and they have a list of commercial companies that provide beowulf clustering solutions.
Re:Atipa (Score:2)
Re:Atipa (Score:3)
Been There Done That (Score:2, Insightful)
Here are some benchmarks:
Octavian Benchmark runs
as of 8/17/01
Problem description:
Acetabular cup with a spherical metal ball compressing the liner into the
shell.
The effect of holes used for screw fixation to the bone is included
Total Mesh Statistics
45814 Nodes
37696 8-noded solid elements
2 contact pairs
Dynamic Relaxation Solution:
Execution Statistics
2 Processors : 62 Minutes
6 Processors: 34 minutes
10 Processors: 24 minutes
Analysis of the execution throughput indicicates a linearly increasing speed
(1/wall clock time)
And here is a review of the results:
the cluster is performing BEAUTIFULLY, and we have been crunching problems on it pretty much nonstop since it was brought online. It has really saved our butts, as we could never have met some key project deadlines without the speed. I've included some bechmarking stats below FYI for a contact problem that took only 24 minutes to run on octavian using 10 cpus. The comparable time on our $50K dual CPU octane workstation is 3 hours and 26 minutes, which translates into a speed up of 8.6 times for about 1/5 the cost.
So, What I am trying to say is build the thing yourself. You will know much more about the system, and you will be able to install any software you want to without having to deal with "customer service". Also, it will save you a bundle as a turnkey solution is nearly 3 times the price. (Even if this cluster was built with Myrinet it would still be far less then any of the pre-built solutions) Lastly, design the cluster for what you need. If your problems involves lots of RAM, then spen money there. If CPU is the bottleneck spent money there. If communication is the bottleneck....
Best of Luck,
Eric
planning (Score:1)
Re:planning (Score:1)
Been there, done that. Planning has not much to do with it, but getting a busy and distracted person's ear does.
Remember: Computing per se is not the primary focus of most compute power consumers.
Re:planning (Score:1)
the fiscal year ends in 10 days and any money not spent goes back to the treasury and scrutinized...
one of the contracts that i'm on right now is freezing as of 09/30 unless our client gets at least some of their funding soon...
the House seems to be busy doing other things right now instead of approving the budget... understandable though...
Western Scientific ... (Score:3, Informative)
Price depends on bells and whistles, but the 8 node, dual processor P-III system we got with SCI cards ran around $35K.
http://www.wsm.com
x4? xMore? (Score:1)
With RAM so cheap nowadays, one could have at least a capable cluster without monster chips. Coupled with Xeons, I fail to imagine how great the price:performance ratio is compared to Big Iron like IBM's Power3 and Power4 systems...
*drool*
Re:x4? xMore? (Score:1)
on a miserable 800 MB/s (450 sustained) bus,
so unless your code is totally cache-friendly,
the CPUs will starve. and if your code is so
cache-friendly, you should probably be looking
at different hardware in the first place.
Re:x4? xMore? (Score:1)
Re:x4? xMore? (Score:2, Informative)
On the other hand, if you have a true multi-threaded, highly integrated task that requires high inter-process communication, separate boxes are a poor choice. Something like a large relational database or multi-dimensional vibration calculation wherein each calculation requires knowledge of it's neighbors motions, is far superior on a multi-CPU box. Unless you implement an expensive Dolphin/Myrinet network, the process communication alone, be it over ether, SCSI, or FC, kills a multi-box solution. Not to mention the fact quad Xeon boxes typically take 4-8GB of RAM so everything is always local.
Re:x4? xMore? (Score:1)
I still think that the $:performance ratio is very exciting on the PC side.
Don't forget Linux, which is free/
So they want a cluster in 2 hours to do .... ? (Score:1, Redundant)
I have the feeling that friend forgotten_password's group have no clue about Beowulf clusters.
Re:So they want a cluster in 2 hours to do .... ? (Score:1)
Re:So they want a cluster in 2 hours to do .... ? (Score:1)
Re:So they want a cluster in 2 hours to do .... ? (Score:2, Funny)
Gee, I know he was asking for a quote, but if the price is that steep just for the RFP I'm afraid we'll have to pass on the consulting fees....
(I normally don't respond to
Re:So they want a cluster in 2 hours to do .... ? (Score:1)
for shame!
Re:So they want a cluster in 2 hours to do .... ? (Score:1)
I dont think you can ... (Score:1)
Scratch That (Score:1)
http://www.par-tec.com/
http://www.aspsys.com/
I have no knowledge about how good these are.
BTW, you DO know that you need a special type of paralell programming to really make use of all the added processors rite ? its not like your regular apps will run faster.
Good luck
T
OUCH (Score:2, Informative)
Package Pricing (Including Server):
8 Processors: $ 8,625
16 Processors: $16,325
32 Processors: $31,725
64 Processors: $62,525
dammm this is like a 1k per proc ! i am sure you can build it cheaper
If I remember my Beowulf correctly... (Score:5, Funny)
+2 Funny on the MQR standard (Score:2)
It's a joke for gosh sake; he was playing off the first person's Grendel reference by alluding to The Three Billy Goats Gruff [coreknowledge.org]!
-- MarkusQ
Beowulf Site (www.beowulf-underground.org) (Score:3, Informative)
this month's Linux Journal... (Score:1)
www.aspsys.com
From Linux Journal (Score:2, Informative)
I hope this helps!
Kent
Vendors (Score:3, Informative)
IBM [ibm.com]does a lot of linux stuff, they even have beowulf traning classes - I imagine that they have some turnkey solution.
Compaq [compaq.com] sells 'em. too.
In other words, almost any company that sells Linux servers sells beowulf clusters o' servers as well. And if you want training, quite a few of them out there have classes for it too
Maybe something here ... (Score:2, Redundant)
If you don't find any answers to your quest then you could always buy 10 dual-processor machines, configure one and then copy its HD image to the other 9 ( I have never tried this ).
Linux Networx (Score:1)
We build beowulf clusters and have (if I do say so myself) the most advanced cluster management utilities in the industry.
Check us out. [linuxnetworx.com]
I'm not a salesman, but we've got some people that would be more than happy to talk to you. :)
Re:Linux Networx (Score:2)
I am a man and will to refer to myself as such.
However, if you inquire at the web site a salesperson will contact you. :)
By the tone of your question (Score:1)
Re:By the tone of your question (Score:1)
RTS+FPS Sim (Score:2, Interesting)
Hmm...
My $0.02 worth of a guess.
Hmmmph.. (Score:2)
Think about it though, how likely is the military to be in possession of simulation software like I mentioned with full-out graphics capabilities that actually runs on those spiffy supercomputers they own? Maybe very likely. If not, then a Beowulf cluster is the perfect starting point along with existing code perhaps from the likes of Loki.
Hmm....
;+)
Don't forget Scyld. (Score:3, Informative)
From the Scyld website (Score:2)
Re:Don't forget Scyld. (Score:1)
Many of the changes do come from my driver releases, but are incompletely or incorrectly copied over the driver in the kernel.
The was all part of the kernel development fall-out about two years ago. I only wanted to release working drivers into the kernel, while Linus wanted all development moved into the regular kernel releases. Linus just started accepting patches from anyone, which resulted in much lower quality drivers.
Re:Don't forget Scyld. (Score:2)
I also upgrade my kernel regularly.
Are you sure that your cards or MB chipset aren't bad or anything?
Also remember that DB wrote many rock-solid drivers for high-end cards like 3Com, etc. I have some of those, too.
Besides, when you buy cheap-ass $5 NICs and they don't work properly, you should blame the cards first.
Applications? (Score:1)
Of course, if the issue is just spending the funding fast, go right ahead. Don't expect results for a long time, though.
Just wondering... (Score:2, Interesting)
Re:Just wondering... (Score:2)
You're wrong about the price though. Two quad boxes would be _much_ more expensive than four duals, because you can no longer use your traditional PIII/Athlon processors. You have to up it to an Alpha or Xeon. Both those processors and their respective motherboards cost 1.5-3x the price of PIII's, for the equivalent number of processors.
The point of Beowulf clusters is to be able to put bunches of computers that are not very powerful in and of themselves together to make something really powerful, so using quad processor boxes kinda defeats the purpose. The ideal price/performance balance is two processors per box, and this is easy to see by looking at the commercially available solutions.
Re:Just wondering... (Score:2)
We bought one of the APPRO 1U dual TBirds, and this thing -screams-. It also howls, but that's the four big blower fans.
APPRO is at http://www.appro.com, and Anandtech had a writeup on the server [anandtech.com].
Linux Labs (Score:2)
My rough estimate from reading their website is that a 10-node 1.33-GHz Athlon cluster from them would price out at something like $16-17k.
Missing Critical Linux (Score:2)
See Commercial sites at bottom of www.beowulf.org (Score:1)
What do you run on the darn thing ? (Score:2)
I've got a bunch of old Pentium I's and II's in a basement of a charity I do some volunteer work for. Can get a hold of a router. I've got all the nerd-boys a bit pumped to beowulf a cluster this winter when it's too cold to paintball.
So my question is, what to run on the blasted thing once you get it up ? Is there anything open source out there worth looking into ? Or am I just going to have to buy an application. If so, which one, how much ?
Re:What do you run on the darn thing ? (Score:1)
Re:What do you run on the darn thing ? (Score:3, Insightful)
IIRC, Mosix allows for using the machine as one large machine, essentially allowing each process (or groups of processes) a single CPU, but funneling the result back to the main controller, so to the user, it looks like one large machine. This is different from a standard parallel processing Beowulf cluster, which behave like classical parallel processing machines.
For that domain (parallel processing), aside from coding your own stuff (hard to due, even if you are a master coder, from what I understand - due to having to understand what classes of problems can be broken down into parallel tasks, and then actually applying that knowledge to real tasks), there is one application that would prove to be "fun" - Ray Tracing.
Fortunately, POVRay has a paralleled version available, for Beowulf clusters. I don't have a link, but I know it exists (heck, you can probably get it at the POVRay site).
One other disclaimer: Everything I have said should be interpreted as "coming out my ass", simply because I have no experience at building or using Beowulf clusters or any other parallel processing architectures. Most of what I know about them have come from
Re:What do you run on the darn thing ? (Score:2)
The POVRay rendering dropped off at 8 procs or so, if i can remember correctly.
We render a hand full of frames per box and do this on several boxes. This keeps the network overhead between machines low.
Sometimes when people talk about beowulf, what they really mean or really want is just a bunch of managed computers running tasks. If the task can be broken down at submission time, like most renders, then take advantage of a NOW.
Re:What do you run on the darn thing ? (Score:2)
In a way, from what I understand, this is kinda what Mosix does for you, automatically, and and thus different from a true Beowulfed parallel processing app.
I guess it also depends on if you want to learn parallel processing techniques, or if you are trying to get a job done.
The drop off I would imagine is due to slow interconnects among the nodes - I am sure there would be better gains the higher speed your interconnects were.
Anybody know how the POVRay for Beowulf machines really works? Does it break the viewport into n sections, handing each section to a different node, or does it process multiple rays (ie, a node per ray), or what? Anybody got a link on this?
Re:What do you run on the darn thing ? (Score:2)
Heh - I think with Slashdot, this is usually an assumed competence. But we appreciate you candidness anyway.
Re:What do you run on the darn thing ? (Score:3, Flamebait)
The guy knows what a Beowulf does, he needs somebody to BUILD one for him.
Goddamn karma whore.
Hello? (Totally OT - Mod me down, I need the fun!) (Score:2)
I wasn't replying to the article, but to an individual who was asking what he can use a Beowulf cluster for, as he obviously didn't know.
I am the last person needing to whore, since I am capped, and been that way for months...
Re:What do you run on the darn thing ? (Score:2)
Have a look at the info contained at MOSIX.org [mosix.org] they provide kernel patches && apps to build clusters also, but these clusters allow -depending on your configuration (how/who/what of each computer in the cluster) - processes to migrate from 'more idle nodes'.
From what i understand, you could setup your 'head or workstation' machine to let procs dribble off onto other nodes.
have a look - much more usefull to geek exploration than beowulf... even if beowulf gets 'all the
Re:What do you run on the darn thing ? (Score:1)
Hell, I don't need beer, but what's life without some fun?
Re:What do you run on the darn thing ? (Score:2)
Because we can !
Aspen Systems (Score:2)
- j
Cheaper solution (Score:2)
Better and cheaper idea, do it yourself. Instead of buying 10 dual proccessor systems at $2000+ each, goto Fryes and get 30 Emachines or whatever they are selling for $299, add a couple hundred dollars for 30 decent network cards, one monitor for the control node (barrow a couple more to do the installs) a few hours to install and configure RedHat 7.1, which comes with the clustering software and you'd be done by morning. You will probably get better performance at half to two thirds the cost. This is what clustering is all about, turning cheap off the shelf systems into a super computer.
Re:Cheaper solution (Score:1)
No, this is not what clustering is about. Beowulf, specifically, is about gaining raw performance without concern for stability or fault tolerance/redundancy.
There are many other uses for and of clustering; not all of them are concerned with speed (your super computer quotient, if you will), less of them are concerned with saving money.
Granted, the original post specifically mentioned Beowulf, but from that one should not assume that clustering == Beowulf.
Oh, and why on earth do you need a monitor to do the install??
Re:Cheaper solution (Score:1)
I'll tell ya why you need a monitor to do the install...
so that when the first machine is up and running you can download ROBOCODE [ibm.com] and do some javabot simulations on your new cluster and see how many battles you can simulate in 1 hr. =)
JOY JOY HAPPY HAPPY JOY JOY!!!
Re:Cheaper solution (Score:1)
whitebox machines also tend to be truely horribly configured - you probably don't want UMA video stealing dram bandwidth, for instance. and even for a smallish cluster, it's smart to get better power supplies and fans, simply because the noname stuff has short expected lifespans.
Re:Cheaper solution (Score:1)
Re:Cheaper solution (Score:1)
Re:Cheaper solution (Score:2, Interesting)
I've just finished building a (really small) 4-node cluster for some research I'm doing.
Each node is a Duron 750 w/256MB RAM and a 100Base net card, power supply and floppy drive. They're vertically stacked on some 2' threaded rod from McMaster Carr. [mcmaster.com]
But get this: I built the whole thing for $170 per node . That's everything, shipped to my door.
They load the kernel off the floppy and grab the NFS root from a fileserver I've got set up, so no need for hard drives. Granted, there's a bit more network and memory overhead (both
Re:Cheaper solution - add shared storage (Score:2)
I know that SCSI can operate on a shared bus for 2 nodes, but I'd really like to test out Oracle 9i Real Application Cluster (RAC) (formally Parallel Server) on a 4 node setup on Linux - but would need a shared storage device for log files, data files, control files.
FC would be a little pricey.
NetApp filers are outta my league.
thanks.
fast and cheap (Score:1)
1) Find a single machine to use as cluster server.
2) Install some linux (I prefers debian.)
3) Set it up as a diskless client server.
Debian have some diskless packages, but I prefers
to use cluster-nfs form clusternfs.sourgeforge.net
it makes it easier to maintain the system.
you can find a howto for setting up a mosix cluster
at the cluster-nfs site dropping the mosix stuff and installing
mpi and pvm should do it.
4) To use your groups computers as diskless nodes
go to http://www.rom-o-magic.net make some boot floppies
and use them to boot against your server.
5) Have fun with your new Beowulf cluster.
Whenever you need a Beowulf cluster just boot up
via the floppys.
In this setup it is only one the server you need to maintain.
Knud
Re:fast and cheap (Score:1)
The links is Cluster-nfs [sourceforge.net] and rom-o-matic [rom-o-matic.net]
Knud
Re:fast and cheap (Score:1)
chriss
We've bought from two places (Score:2, Informative)
Linux Journal Ads for Clusters (Score:1)
From the Oct. 2001 Linux Journal
http://www.microway.com - Alpha, Athlon & Pentium
http://www.aspsys.com - Athlon and Pentium
These ads mention clusters specifically.
Some vendors.. (Score:1)
[planetcluster.org]
http://planetcluster.org/links.php?op=viewslink
Hope that helps
APPRO AMD Servers (Score:1)
http://www.appro.com/ [appro.com]
or for the AMD systems specifically http://www.appro.com/1124.html [appro.com]
Should give you enough information to put together a 10U Beowulf cluster...
Re:APPRO AMD Servers (Score:1)
Who is this for? (Score:1)
Your tax dollars at work.
Re:Who is this for? (Score:1)
I remember some cool shopping trips with my US Government VISA card in September. Basically we throw stuff in the cart just to spend the money. Whether you need it or not important.
Re:Who is this for? (Score:1)
http://www.terrasoftsolutions.com/products/briQ/po wer.shtml
[terrasoftsolutions.com]
Don't know that it's a beowulf cluster though.
Scyld (Score:2)
Haven't dealt with them directly, but I believe they have Don Becker, one of the Beowulf pioneers.
I think they have developed a system to help provide a single system image, along the lines of MOSIX, but not MOSIX, IIRC. This can help managing such a cluster which could otherwise be like managing 10 separate machines - a hassle.
RLX Technologies (Score:1)
Just to say it out loud...I work there, so it's a blatant self-promotion, I guess. But it is a bad ass little product that packs a punch.
Wow! (Score:2)
a Beow...
oh wait. Never mind.
LOL [NT] (Score:1)
What is turn-key? (Score:2)
Re:What is turn-key? (Score:1)
In other words, no installation, everything is configured to work out of the box...
Must be a very big box though...
Re:What is turn-key? (Score:1)
Buy it in two days (Score:1)
Yes with the fiscal year ending we all have lots
of money to spend this year or we'll loose it
next year. Don't you love working for the government?
Sometimes I think the only job in America that
is better than working for the government is being a domestic cat.
Western Scientific (Score:1)
Other vendors (Score:1)
Not sure how "out of the box" the Einux ones are, but you can get a 10 node dual-Athlon setup for around $25k.
Smaller companies (Score:1)
YASOB: Yet Another Supplier Of Beowolves (Score:2)
You should think a bit about whether the extra abilities of the Alpha boxes are worth the extra bucks for your application. One thing which I think that I remember about the Alphas is that they use a crossbar switch to link the several processors on a motherboard to memory, et cetera. This should give better throughput. They also have huge caches which should help with big matrices. I think that if you have lots of little problems which should be run in parallel, more nodes with lower price and capability per node might be the way to go.
I remember back in the days of the XT, Microway used to sell math coprocessor and video boards for PCs which cost more than the box you hooked them to, along with high-grade compilers which would put that hardware to work. They were once the place to get hardware and software for doing seroius number-crunching on a PC.
off topic: instability, windows and science (Score:3)
One portion which shocked me was:
They crashed a unix os? Wow! That doesn't match up with my limited experience. The only way I've ever done that was by trying to do stupid things as root, like running mindi with a buggy kernel. I wouldn't have thought that this would be a problem for a normal user.
Here is something which didn't surprise me at all:
Mine, too.
Sure, I can sell you one in less than an hour, but (Score:2)
rm -rf
Well.. (Score:2)
DO you have some app that needs it? I mean, you can't just run anything....
Did some department just come up with some app already designed with the PVM libraries or something?
/. helps war effort! (Score:2)
I'm not complaming - i'm just finally glad to see
fight on!
This is technically off-topic (Score:2, Funny)
why not a Mac? (Score:2)
A quick search of Apple's site actually mentions clusters [apple.com] [apple.com]. Perhaps 10 dual G4's would suit your needs?
Re:10,000 G (Score:1)
Errr.
No, I don't think so. The article you are refering to was actually about building a server with 1 TB of storage for under $10,000. I don't think it had anything to do with clustering. See it here:
http://slashdot.org/article.pl?sid=01/07/19/155421 6&mode=thread [slashdot.org]
That was over 2 months ago, so the price is probably even less than $10K now...
Take care,
Brian
--
AssortedInteret.com
Featuring 100% Linux Based Web Hosting
Try us Risk Free with our 30-Day Money Back Guarantee
http://www.assortedinternet.com/hosting/ [assortedinternet.com]
More like 3-4 grand for 1TB fileserver (Score:1)
Re:Could you imagine... (Score:2)