Ask Slashdot: Clusters On the Cheap?

First time accepted submitter serviscope_minor writes "A friend of mine has recently started a research group. As usual with these things, she is on a shoestring budget and has computational demands. The computational task is very parallel (but implementing it on GPUs is an open research problem and not the topic of research), and very CPU bound. Can slashdotters advise on a practical way of getting really high bang for buck? The budget is about £4000 (excluding VAT/sales tax), though it is likely that the system will be expanded later. The computers will probably end up running a boring Linux distro and Sun GridEngine to manage batch processing (with home directories shared over NFS)."

Uhm AWS EC2 Cluster Compute

[Anonymous Coward]

Why waste money on building a cluster when you can rent the best in the world * by the hour * ?

Re:Uhm AWS EC2 Cluster Compute

[jpedlow]

AWS EC2 was my response aswell. :)

for raw horsepower on the short - medium term, use AWS http://aws.amazon.com/ec2/ [amazon.com]

ec2 should do well for this, imho :)

Re:

[KiloByte]

From a back-of-the-envelope estimate, I see that AWS gets even with buying your own hardware in three months. Except, you still get to own the gear.

Thus, if you need a week or maybe a month of computation, AWS might be a better option, but for anything above that, forget it. If your needs are more bursty, that shifts the balance towards AWS, but again, you need to estimate what you need.

Re:

[Spazmania]

Does your estimate include power consumption for the computers and the requisite cooling for that many computers?

Re:

[Joce640k]

How about setup/installation time. Installing and configuring a whole bunch of machines takes a while.

Re:

[KiloByte]

Electricity but not labour -- but for cash-strapped kinds of research, there's plenty of free or near-free student help.

A cluster you can get for L4000 is not going to need dedicated air conditioning.

Re:

[petermgreen]

Quite likely neither is relavent. At least here individual research groups aren't charged for electricity and labour is pretty much a fixed cost. As for aircon that isn't really an issue at this scale.

Re:

[hardtofindanick]

There must be a better way than ec2

Cluster GPU Quadruple Extra Large Instance
22 GB of memory
33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core “Nehalem” architecture)
2 x NVIDIA Tesla “Fermi” M2050 GPUs
1690 GB of instance storage

Personal experience: Disk IO: not dependable, network IO excellent.

4000 British pounds sterling = 6302.8000 US dollars
Assuming for each "experiment" you run 20 instances in parallel for 15 minutes (partial hours count as full hours, so remember to round up even if you use for a minute), you spend 40$ per session.

You get to make 160 experiments, and you are over budget.

Or if you adjust everything optimally, e.g., end the experiment at 59 minute mark, and assuming your "pa

Re:

[Nikker]

Well 640 experiments is all you will ever need...

Re:

[lee1]

The summary: "implementing it on GPUs is an open research problem and not the topic of research." Not every problem is well suited to GPUs, which is what FASTRA uses.

Re:

[ebonum]

Would there be a problem if all the code is in C++ and uses a propriety database?

Or will you have to re-write everything from scratch to match Amazon's API?

trade-off

[TheSHAD0W]

Actually, that's a good question... Assuming no time constraints, at what point does it make sense to buy hardware rather than use the cloud? Take that budget above (roughly US$6K) and the best hardware you can get for that price: How many months would you need to run it, flat out, to equal the number of floating-point ops EC2 would give you for that cost?

Re:trade-off

[subreality]

Sometimes, never. Don't forget to add up power, cooling, sysadmin time... And that's before getting to intangibles like being able to spin up 400 cores for an hour and getting your result same-day instead of only owning 40 cores and having to wait until tomorrow.

Cloud computing really cleans up for batch computing jobs like this.

Re:

[account_deleted]

Comment removed based on user account deletion

Re:

[mrt_2394871]

Question: How much bandwidth would that run? [...] If he is like many of us they probably have bandwidth limits and/or have to share that bandwidth with other users so if it takes a big ass pipe I could see that possibly being a problem.

If the group is set up at a University, odds are it's on JANET [wikipedia.org]. Those are big pipes (our connection was 155 Mb/s several years ago).

Re:trade-off

[cas2000]

or use a 16GB or 32GB USB flash (or better yet, a small SSD - swapping to USB flash would suck) as the boot drive on most machines and have one machine (the head node) with hard disks as a file server - NFS will do for small to medium size clusters (anywhere from a handful of nodes to a few hundred nodes). The OP is going to need a head node anyway to run Slurm or Torque as the scheduler/resource-manager (yes, i have built clusters before).

put a 2nd NIC in the head node, so the compute nodes can run on a private 192.168 network (you'll need a 24 or 48 port switch as well), and also install DHCP, tftp, and apache. Set up the last three to allow the compute nodes to netboot clonezillla....install everything you'll need on one compute node (openmpi, libatlas, octave, R, open source and proprietary scientific software as needed, etc) and use clonezilla to mass produce the rest (also allows you to quickly and easily add new nodes or replace failed nodes). LDAP or NIS will be needed for sharing account/auth details between machines.

i built something quite similar to this last year (but using some sunfire 1RU opteron rackmount servers as the compute nodes)

I'd go for an x4 CPU, they're not that much more than an x3 and the extra core is useful. 8GB RAM too, 2x4GB only costs about $40). given the budget, it's probably not worth getting a custom power supply for the tray-mounted motherboards, so each will need its own dedicated PSU

each node is going to cost somewhere around $250 (very rough estimates: $50 for the m/b, $40 for 8GB RAM, $50 CPU, $50 PSU, $60 for 32GB SSD - but possibly a fair bit cheaper as a bulk purchase), and the head node will cost roughly triple that (you'll need a case w/ hot-swap bays for the drives - a Norco 4224 is probably overkill but at well under $400 for 4RU with 24 SAS/SATA hot-swap bays, it would be hard to find a significantly cheaper case even with less drive bays) so for $6K you can build a cluster with 20 x 4 core compute nodes plus a good head node for the scheduler & file server). 80 compute cores for $6K. that's good, even considering that with cheap crap motherboards you'll have a noticable failure rate. the cluster i built last year with name brand hardware cost closer to $50K. I could build a better system today (far less nodes with a lot more cores and RAM each), also with name brand hardware, for about $20K - $30K

trays for the motherboards, the rack(s), and cooling will cost extra. as will licenses for any proprietary software they might need to run (could easily cost as much - or more! - as the hardware). if the OP's friend is at a university, she can probably scavenge an old rack or two from another dept, but even if she has to buy one new she could easily build 15+ compute nodes entirely within the $6K budget

Re:trade-off

[pz]

Don't forget to add up power, cooling, sysadmin time...

If the friend's research group is in an academic institution, power and cooling are outside of the acquisition budget, along with space, network, etc., as those are typically part of overhead. Depending on the institution, sysadmin services are too. Often the institution will even have embarrassingly large discounts with hardware and software vendors (at my institution, a licensed copy of Matlab, for example, is about $100 per seat per year).

GBP 4000 buys a rackfull of modern computers that can be run as long as you want. It can be used to explore ideas without concern for cost. In contrast, once the GBP 4000 has been paid to a cloud service, the money is gone. Given that the pressures for a new researcher are already immense (and I speak from recent first-hand experience) not worrying about running out of compute resources, even if it means the instantaneously available compute power is somewhat lower than what you could get from a cloud service.

If this new research group is going to be competing for research funds, for example, then the compute resource is going to be highly utilized for the first 12-18 months to get preliminary results in order to write grants. I can't imagine that GBP 4000 is going to last long enough. Looking at Rackspace, as another poster suggested, they charge about USD 350 per decent configuration (8GB RAM / 320 GB disk) per month. That single server is going to last 18 months before the money is gone. If the memory demands of the computation aren't so large, then the charges are lower, say USD 45 per month (1GB RAM / 40 GB disk), then you get to use 7 virtual machines for the same 18 months.

Given that a highly capable system can be purchased new for USD 500, the same money gives the researcher about a dozen real machines for 18 months, and beyond (buying off-lease machines can easily double the amount of hardware). From my perspective as a researcher, there's no comparison: when money is tight, buy your own hardware and take advantage of the services provided by your institution.

Re:

[Jawnn]

GBP 4000 buys a rackfull of modern computers that can be run as long as you want.

On what planet? Even with the most "embarassing discounts" that number is not going to by anywhere near that many "modern computers", unless you're trying to spend as little as possible per rack unit, which, of course makes no sense. Sure, I could fill 1U with a 1U chassis, filled with cheap consumer grade components for a couple hundred bucks, maybe less if I was extremely well connected, but why? For a few hundred more I could plug that hole with many times that computational power but again, why, when I

Re:

[DuckDodgers]

I think your logic is correct. If our hypothetical researcher had to provide his own facility, do his own system administration or hire someone else to do it, pay his own electrical bills including the cost of cooling, manage his own network, etc.... then cloud computing is economically far better. But since this person will be working in a context where he or she is only responsible for the hardware and software costs of the physical machines, buying machines is much cheaper.

Re:Uhm AWS EC2 Cluster Compute

[subreality]

+1. It is very nice to be able to spin up 50 instances, run the hell out of your job, then delete them. It gets done faster, and you don't have to deal with maintenance, upgrades, and obsolescence. Realized you need more RAM? Just adjust it! And so on. It'll likely come out cheaper than owning your own after you add up all the hidden costs (power, cooling, space, time, etc).

The only downside is there are no GPUs. But that's not really a downside: if you do end up developing a GPU version, your cluster configuration would completely change (1x2 cores per box, 3-4U boxes with many PCI-E slots, instead of 2x8 cores or however many you can economically cram into a 1-2U pizza box), so the investment you'd make now would be completely wrong for that future development. With cloud servers you minimize sunk costs.

I use Rackspace Cloud [rackspace.com] and it performs as promised. It's definitely worth a look.

Re:

[jpedlow]

http://aws.amazon.com/ec2/#instance [amazon.com]

amazon has gpu instances :)

Re:

[subreality]

Sweet!

Re:

[GeorgeK]

Actually, Amazon now offers instances with GPUs. See their page on High Performance Computing [amazon.com] for more details.

Re:

[ron_ivi]

EC2's not the cheapest place to rent servers.

If you're going to rent for a whole month or more, it seems you get a lot more bang for the buck by renting dedicated servers like these:

http://www.server4you.com/root-server/ecoserver.php [server4you.com]

Re:

[account_deleted]

Comment removed based on user account deletion

Re:

[Cwix]

Hmm I wonder if the College has a datacenter already. You know, so ya dont have to worry about the overhead.
Ohh and man hours to configure the thing? You know I'm sure there are a load of students there working for free.

This is NOT some company trying to spin up a new service. This is a school doing a project. That means shoe string budget, and the people get paid with good grades in their class.

4000 pounds wont get you very far at all on Amazon.

I love it when people take the question given and then ins

Re:EC2 is expensive

[TheRaven64]

The problem is the constraints. The cheap cluster in my old department cost £100k. £4k does not buy you a lot of hardware. You will probably find a lot more lying around in the undergrad labs. For some of my work as a PhD student, that's exactly what I used - each lab had 40 machines on a GigE network and closed overnight, and for work that wasn't that latency sensitive, I could distribute it across the machines there and run it at night without anyone minding.

If you're serious about needing a cluster, then you need to spend a lot more than £4K. If you only need a cluster for a short time, then £4K can buy you a chunk of time on someone else's hardware. Since this is the UK, they should contact the Manchester Supercomputing Centre, which provides this kind of service to UK universities at quite a reasonable price (and will also lend you people who are good at optimising code for their systems). If the university doesn't already have some clusters lying around, then you should get in contact with a few other research groups. £4K won't go very far, but if half a dozen research groups each put in £4K then that gives you enough for a reasonable cluster to share between the various users.

Re:

[lee1]

That's helluva lot of computing power right there, at $0.76 an hour.

I've never used the service, but if what another poster says is true, that you pay for a full hour when you use any time at all, the cost per hour can be huge (theoretically infinite). The first debugging session will use up all your money. (There is no avoiding doing some debugging and testing on the finial compute configuration, which will involve a bunch of short runs of your code - each run of a few seconds will count as an hour.)

Re:

[account_deleted]

Comment removed based on user account deletion

Re:

[networkBoy]

And to this end, the compute task is CPU bound. Is it actually memory and thread bound or is it core CPU tick bound?
If it is memory bound, not thread heavy, and not a lot of floating point math I would suggest a cluster of Atom dual core ITX boards. 2 gig of ram per node, boot off a USB key and share all the data over GbE. That is likely to be the most active cores for the buck, but if there is a lot of floatin point math or thread/context switches... ouch.

Another option is to go with a lot of slightly o

MB stacks

[Max Romantschuk]

I've seen quite a few projects where people have stacked motherboards with spacers, using booting over Ethernet and a single power supply for multiple MBs. Google should be of use here, I'm trying to get my offspring to school so I'm cheating and not providing any links...

But the idea is that skipping the case and other components makes things cheaper. Leaving the rig exposed without a case also eliminates the need for most cooling.

Re:

[Required Snark]

In what plane are the motherboards stacked? Are they in the horizontal or vertical plane? Vertical stacking allows the hot air to exit the top, while horizontal implies that external airflow must be provided to get the hot air out. Also, if you have multiple layers of vertical stacks then the top boards are getting the hot air from the lower boards.

A random suggestion: Have the motherboards all parallel at a 45 degree angle. This could provide passive heat driven air flow. The cool air enters at the lower

Re:

[JorDan Clock]

Does it matter which way they're stacked? If it's horizontal, you could just flip it on its side if heat becomes an issue...

Re:

[hedwards]

You're better off doing it from the start rather than waiting for it to be a problem. One of the things I remember from college was that if you had a DEC Multia, you had best be standing it up on its side, as they would have some serious problems very quickly if you lay them on their side.

Best thing is to avoid the possibility and the head aches of reorienting however many motherboards after it becomes a problem. Chances are you'll know it's a problem because they're unstable and possibly damaged.

Re:

[Bill_the_Engineer]

Nothing that a fan blowing across the motherboards can't fix. When you have multiple motherboards in a single enclosure you will need active cooling anyway.

I wouldn't recommend the stack motherboard method anyway. You can make beige box 2U rack mount PCs with two 6 core Xenons and 24GB of memory for around $3000 US and this includes 4TB of SATA drive storage in a hot swappable chassis. (Check out new egg). If you need more power then just build another chassis and at it to the rack. The trick is to start m

Caseless Example: Ikea Cabinet Cluster

[cmholm]

An near-example of what Max is talking about can be found at the Home Linux Render Cluster [helmer.sfe.se]. The builder threw six dual cpu motherboards into a small, gutted filing cabinet and Gig-e. Cheap, expandable.

However, if your friend hasn't got a very good idea how much mmmph she needs, the AWS EC2 rental idea has merit.

Re:

[tempest69]

I did this back in '03 (ok, I had discrete power, but a diskless boot) there is a project called warewulf that was pretty decent. The pxe boot was a little odd with the hardware at hand, so make sure the MB supports that sort of thing should you go this route. If you have a small enough data requirement (or fast enough broadband), a web service might be the way to go. Uploading/Downloading terabytes of data is a horrible thing over a low grade connection, and certainly isnt pretty over 100mbps lines. go

Re:

[MimeticLie]

My college has a program [littlefe.com] doing just that. They've been working with educational institutions, but the directions and parts list are available for anyone who wants to create their own. I'm not sure if it would be the best performance you could get for the price, but it's pretty easy to set up and low-maintenance as far as clusters go.

Re:

[Just Brew It!]

If you don't mind going really ghetto, you can also mount motherboards in plastic storage crates using zip ties: http://techreport.com/forums/viewtopic.php?p=445461#p445461 [techreport.com]

The pictured systems were diskless; everything was network based -- PXE boot, with swap partition and home directory mounted via NFS. With more modern hardware (the linked forum thread is from 2005) you might have a harder time getting 2 to a crate since the CPU heatsinks may be too tall; but using heatsinks designed for 1U/2U rackmount

Re:

[deadline]

You may be thinking of this [basement-s...puting.com] (Limulus Project)

Has she investigated existing clusters?

[Goonie]

Many universities/consortia have supercomputers available on which researchers can apply for (or buy) time. For example, my university is a member of VPAC [vpac.org], which has a big-arse cluster shared between a number of institutions. She might get much better bang for buck if she uses the money for that, rather than splashing out for dedicated hardware.

Re:

[RuBLed]

Also this guy have some and he's not sure what to do with it. They could also have brunch or something.

http://linux.slashdot.org/story/11/09/13/2111210/Ask-Slashdot-Best-Use-For-a-New-Supercomputing-Cluster [slashdot.org]

Re:

[Tim C]

That guy is in the US, while quoting a budget in GBP clearly indicates that the submitter's friend is in the UK; brunch seems unlikely.

Re:

[serviscope_minor]

Thanks for the suggestion about university facilities. This university is not great in this regard. Also, the cluster is used to process experimantal data, and can do either one large dataset or many tiny ones within 6 to 12 hours. The advantage of having a personal cluster is that the latency is low: you can tell by tomorrow if some of today's experiment worked, which is necessary to have a decent turn around on the experiments.

Re:

[bakuun]

A followup on this: since you gave the budget in £, I assume that you're working in the UK. The national UK computational grid service is called... "national grid service". You can find it here: http://ngs.ac.uk/ [ngs.ac.uk] . It is a little complicated to get access (you need to visit somebody in order to prove that you are you, so you can get a login certificate), but once you have an account it's easy enough to use. You need to submit requests for CPU time though, and while I never had any problem getting mine

Use the Existing Grid

[Roger W Moore]

Why buy your own when you can use existing GRID infrastructure? For 4k you can't do much more that get a few decent desktops for yourself and a few grad students and/or postdocs. Rather than blow it on a massively underpowered cluster use the grid. I know the UK has massive clusters available to researchers so find out how to get an account and resources on them and use those. For test jobs, interactive analysis and other low latency tasks use your desktop.

buy one Opteron 6100-based box

[Chalex]

You can get a SuperMicro reseller to sell you one workstation with 4 sockets of CPUs and a bunch of RAM. UK£ 4000 = 6 299.2 U.S. dollars

That buys you a box with 4 x Opteron 6134 (32 cores) and 128GB RAM (32 x 4GB sticks). And some hard disks.

Re:

[Anonymous Coward]

parent is correct.

or for some more get the the 4x6168 (48 cores for about 770$ each ~ 3000, MB is around 800$, supermicro cabinet - 800$ OR use a chenbro fileserver for 300$) and 8GB sticks are pretty cheap these days and almost proportional in price compared to the 4GB - get the KVR1333D3Q8R9S/8G for about 90$ or about 90*8 = 720$ for 64GB. Around 5000 in total, though there are some additional costs like coolers etc but still wont break budget.

i got a similar config for doing something that is cpu intensi

Re:buy one Opteron 6100-based box

[toruonu]

Yes, my recommendation would be also, we do loads of LHC data analysis and simulations and have found that for real science real cores outweigh hyberthreaded ones so we run Opteron 6172 x2 in supermicro chassis that fits 4 servers into 2RU. The cost of such a box of course is ca 11keur, but it gives 96 cores 192GB ram. Now she can get for half the money that she has about half of that so 48 cores 96GB ram should be doable using SM boxes and you can scale up/down with CPU frequency to adjust the cost and maybe adjust total RAM alongside to fit in the budget. If she plans to later expand she may actually want to spend the money to get the 2U chassis with only 2 of the 4 machines present and later add one/two more by just buying the board with cpu/ram.

Re:

[serviscope_minor]

You can get a SuperMicro reseller to sell you one workstation with 4 sockets of CPUs and a bunch of RAM. UK£ 4000

Can you provide a link? Most places will sell that in 1U form. A workstation formfactor could be very useful, depending on where it ends up.

Consider other options

[msobkow]

You can't really get a cluster for that kind of money. You can barely get one decent box.

But you shoud be able to rent a lot of computer time in the cloud for that kind of money, or use it to buy time on someone else's cluster.

Re:

[allenw]

This is absolutely correct and if I had mod points, I'd spend them here.

If your budget is only £4000, you don't have the funding to build a real, actual grid for something that is CPU bound. If you are lucky, you have enough to get one or two boxes and some network gear to put on the top of someone's desk.... at least if you are doing AMD or Intel higher end procs.

Here are two ideas worth exploring...

1) Look at boxes like SeaMicro and other Atom-based mini-grids-in-a-box.
2) Look at building your own

Theoretical analysis

[Warlord88]

OP hasn't mentioned a lot except budget. Since you are on such a tight budget, I would highly recommend doing some theoretical analysis first. Do you have a serial code? How much parallelism exists in the code? You say the task is 'very parallel', but Amdahl's law [wikipedia.org] (which is really common sense) will tell you that even for small amounts of serial sections of code, your speedup will be limited. You should also consider the amount of time the code actually runs. Achieving a speedup of 2 for a serial code that runs for one minute is near worthless.

After you estimate speedup, do some rough calculations on the basis of average cost of a processor and the the number of processors required. This should give you an estimate of the hardware cost required. Compare that with the cost of CPU cycles per dollar you get using a cloud service such as Amazon.

Re:

[serviscope_minor]

I've already done the analysis (in an ad-hoc manner). The problem is data-parallel, where each process runs on a separate chunk of data and never interacts with the others. I've been running it on a spiffier cluster which I will soon loose access to.

Amazon AWS.

[Haven]

$1.60 / hour for the largest non-GPU cluster instance. This also provides you with rather fast interconnects and scalability with multiple instances.

Only £4,000 in hardware would be a waste of money. You wouldn't have all that much computing power, and it would be obsolete immediately.

Re:

[serviscope_minor]

$1.60 / hour for the largest non-GPU cluster instance. This also provides you with rather fast interconnects and scalability with multiple instances. Only £4,000 in hardware would be a waste of money. You wouldn't have all that much computing power, and it would be obsolete immediately.

Interesting option. It looks like it depends on the usage pattern. The large amazon ones are 8CPUs (by the looks of it). For 4000, one can get about 80 high clocking x6 1100 cores. At 1.60 per hour, 4000 would buy

Re:

[Ironhandx]

No, you've done them correctly.

Anticipate your usage patterns. If you think you're going to need the power in short bursts of "I need it NOW" rather than "I can let this run while I'm working on something else" then Amazon AWS is your best bet.

Otherwise a Supermicro Opteron based system will win, hands down.

As an aside I have an old 2nd gen opteron system with two dual core opterons at 2.2 ghz in it that still outpeforms my friends brand new i5 that he bought as a dual purpose cpu so as others have mentione

Your best bet

[eclectro]

Study the design of the "microwulf" [calvin.edu] and it's relatives. Considering that hardware prices has dropped since 2009, your task might be achievable.

Re:

[deadline]

Have a look at The Limulus Project [basement-s...puting.com]. You will be able to buy one of these real soon.

HP

[guttentag]

HP? Is that you?

Not doable

[darkjedi521]

Assuming a 1.5 to 1 correspondence with the USD, you're either getting a decent cpu box and no storage, or a reasonable amount of storage and no CPU. I build/run supercomputing clusters for molecular dynamics simulations at an university in upstate New York, and I wouldn't even consider attempting a cluster for less than $25,000.

Since the OP didn't specify if this was massively parallel or not, I'm going to assume this is so I can use AMD chips for cheapness.

First off, storage. Computational output ad

Re:

[cowboy76Spain]

First off, storage. Computational output adds up quick. You're looking at $7,000 USD for 24TB raw storage from the likes of IBM or HP or Dell. Yes, you can whitebox it for cheaper, but considering if you lose this box, nothing else matters (And I doubt you have the funds for proper backups), it pays to get hardware that's been tested and is from a vendor you can scream at when it breaks.

Having someone to scream off may help ease some pressure, but if you do not have proper backups you are fubar no matter who you scream to.

Re:

[imsabbel]

I build a cluster in a similar budget a while ago. It can be done easily.

3-5k for a node it pure bullshit and has nothing to do with reality.

For that budget, I got 8 machines with 2GHz core2 quad cores. 2 years ago.

Re:

[darkjedi521]

I've never been able to build dual socket 1U systems for less than $3K

Re:

[the eric conspiracy]

Don't buy rack mounted dual socket systems if you have this sort of budget. They are too expensive.

computer recyclers

[thedarknite]

Try a computer recycling centre, most tend to be short on storage and are happy to sell a large number of desktop machines at a lower than normal price per unit. Community operated ones tend to be more helpful than business ones though.

Buy a small chunk

[pigwiggle]

Buy a small chunk of something that looks like the big machines she will be using. As others have said, with that little money you aren't going to get legitimate computational resources. But she will certainly qualify - or already has - on some of the larger public machines. In my experience, it is really nice to have a small, i.e two or three nodes, cluster to test and benchmark code. You can look at things like parallel performance on a single node versus across nodes. If the code plays well with sha

BOINC Project?

[bradley13]

She could also consider creating a BOINC project [berkeley.edu]. She could then do some publicity locally and on forums, to get people to choose her project. I've never tried creating a BOINC project, so I don't know how hard this is. However, I do run the client as a background task, and I imagine many other people do as well.

Don't spend it on hardware.

[Jane Q. Public]

Spend the money on a programmer to parallelize the algorithm on standard CPUs, and put it out on BOINC. People volunteer their spare cycles for BOINC projects that are barely more interesting than the chemistry of aardvark snot. She would likely get volunteers if there's anything of even passing interest in her research.

Re:

[GameboyRMH]

Wouldn't it be possible to set up a local BOINC cluster connected directly to the BOINC job server to do the bulk of the work, and then optionally allow outside volunteers to help? Then you could take a "help or GTFO" attitude to volunteers. Worst case scenario, you're down to your local cluster (which could include all the desktop PCs in the labs as some have suggested.)

Boxes on shelves

[stevelinton]

If your friend doesn't want to do a lot of engineering work, then for this price I would just buy 10 or so PCs (depending on memory/CPU tradefoffs) from wherever has a special offer, plus a gigabit switch and put them on shelves. If you need a lot of memory, or can usefully share memory then that would be a bit different, but you can buy a usable headless PC for £300-£400. This will also not be terribly power efficient, nor will components like motherboards be of the highest quality, but you get

Re:

[stevelinton]

Just did a bit of checking. For £249 including VAT you can get a mini-tower, dual core midrange CPU and 2GB RAM. A dozen or so of these and a switch looks very appealing if there is space. 300W PSU, so cluster should be under 5kW.

Re:

[Vectormatic]

if you are open to custom building yourself, you can dispense with the optical drive and other crap you dont need (pick smaller drives if storage isnt a concern etc.. avoid windows tax, how ever small it may be on a 250 quid box) and dump that into stronger hardware

Late last year we had a crunch-intensive problem at work, and the internal IT department wouldnt even give us a price quote, just said they could do it (a problem which required six octo-core xeon machines, in a world wide company with 90k in peo

Re:

[imsabbel]

Also, dump video cards.

And HDs: Depending on the load, one might not need a big / fast local storage at all. It can be cost efficient to use a very small SSD.

And yeah to boxes in shelfs. A usable 19" rack case alone can be more expensive than a computer box..

Microwulf

[tqk]

Microwulf [calvin.edu].

Who is paying the electricity bill?

[Michael Woodhams]

And how much space and air conditioning do you have? Depending on the answers do these questions, the optimal* solution might be 'get a bunch of 5 year old computers nearly for free.'

* Optimal for your friend, not for her university.

Don't use EC2

[hawguy]

I don't think any of the posters recommending EC2 have ever looked at the economics of EC2 versus self-hosting.

If you have long-term compute needs (as opposed to needing to throw lots of cores at a problem to get fast results in a short time), you're better off buying a Dell.

An EC2 Quadruple Extra Large EC2 instance is $1.60/hour. You have around $6500 USD, so you could buy 169 days of computer time at EC2 (ignoring the cost of I/O and network bandwidth).

This instance has 23GB of RAM and is equivalent to 2 [amazon.com]

Re:

[PhrstBrn]

You can save a few bucks by building your own (or going to a custom whitebox builder), but the Dell comes with 3 years of next business day support. Last time I priced out a whitebox builder, they beat Dell's best discounted price by about 10% and only offered a 1 year warranty.

Rubbish.

• SYS-6026T-URF - $1200
• Xeon X5647 - $850 each
• 4GB ECC Registered - $50 each
• 1TB SATA 7200RPM - $120 each (WD RE4 or equiv)

Chassis + 2xXeon + 24GB RAM + 2x SATA drives =~ $3500. Drop the chassis down to 1U and you're looking at ~$3200. I'm assuming the $5k Dell is an H200 controller, and not a H700 card.

These prices are retail prices, not wholesale, and most are on the conservative side, so a system builder is going to have better wholesale prices on parts than what I quoted even. Unless your system

Re:

[hawguy]

These prices are retail prices, not wholesale, and most are on the conservative side, so a system builder is going to have better wholesale prices on parts than what I quoted even. Unless your system builder is making 50%+ margins an their hardware, your system builder is trying to rip you off, or you're not comparing apples to apples.

Yeah, admittedly, I'm comparing apples to oranges - last time I did a Dell comparison, i was using a Dell VAR that was able to secure good discounting from Dell, and it was a 4 socket server with 256GB of RAM and no disk.

The Dell was 10% higher and came with a 3 year warranty on-site warranty versus a 1 year repair shop warranty from the builder, so we went with Dell.

In 9 months we've had one problem with the servers, and Dell swapped out a motherboard the next day. Diagnostics pointed to the hard drive con

Re:

[hawguy]

I don't know where you get that $70/mo. figure from, but please note that rates vary wildly. UK rates are roughly 3 times higher than those in Hawaii which is the most expensive US state for electricity. The cheapest states are more than three times cheaper than Hawaii. Location matters.

Though, given the fact that this will run at a university, they're probably going to pay exactly zilch.

I used $0.15/KWh to run a 650W load 24x7, which is close to what someone would pay in California to host the server. This article [wikipedia.org] says that rates in the UK average around $0.185/KWh (which would raise the cost of electricity from $86/mo to $70/mo). Feel free to plug in your own local rate and see where the breakeven point is.

Re:

[hawguy]

Instead of just speculating that I'm overestimating the cost, why not do the math yourself and show us your work?

EC2 pricing is here: http://aws.amazon.com/ec2/pricing/ [amazon.com]

$1.60/hour * 24 hours/day * 169 days = $6489

You could buy a one year reserved instance:

$4290 + $0.56 * 24 hours/day * 164 days = $6494

I used 15 cents/Kwh (which is around what I pay in California). Your local rates will vary, feel free to use your local rate:
650W * 24 Hours/Day * 30 days * $0.15/Kwh = $70.20

The 650W figure is measured power u

Clusters on the cheap

[kawabago]

Mums are nice.

Re:

[John_Booty]

Yer mum's pretty nice, mate. Winkwink nudgenudge.

World Community Grid: Free if You Qualify

[BBCWatcher]

Somebody upthread mentioned BOINC [berkeley.edu], which is a great idea for many parallel-oriented compute-bound problems. However, while making your project compatible with BOINC is necessary, it's usually not sufficient. The problem is marketing, to convince enough people to run your work. World Community Grid [worldcommunitygrid.org], sponsored by IBM, is free and is an excellent way to solve that problem. You can submit a proposal [worldcommunitygrid.org], and if approved you'll quickly have lots of BOINC-powered computing working on your problem.

Many factors are involved ...

[MacTO]

First, see what's available. Many departments have computing options available that depend upon scheduling and departmental budgets.

If that doesn't work out, what are you doing? Serial processing or based upon existing software, then 'contract it out' (if that's an option). It's easier, and probably cheaper, especially in the early stages.

Parallel processing though needs more serious consideration. Cheap SIMD was being offloaded onto GPU's the last time I looked (which, admittedly, is probably too long

Anyone make a smartphone cluster yet ???

[Ex-MislTech]

I think it doable especially with Android based phones.

Data rates might be an issue but some things like SETI @home
didn't have real high data rates, with Wifi enabled phones this could be
mitigated to only work when Wifi was active.

100's of millions of phones moving in and out of a global cluster.

I think I just had a nerd moment.

Re:

[MimeticLie]

More like 100s of millions of people who try it once and then never run it again because their battery life went to hell.

In the US, we have Teragrid/XSEDE

[Orp]

Does the UK/Europe have federally funded, shared computational resources for researchers? In the US we have what used to be called the Teragrid (now XSEDE) which is a network of supercomputers that are available for researchers. You have to write a proposal for machine time, but they're not all that difficult to get. The main disadvantage is that you have to submit your jobs via a queueing system, so your jobs usually don't start right away (having your own hardware does have its advantages) but the big sha

Talk to this guy...

[argStyopa]

http://linux.slashdot.org/story/11/09/13/2111210/Ask-Slashdot-Best-Use-For-a-New-Supercomputing-Cluster [slashdot.org] ...who posted that he was allegedly just getting the budget to build "...the largest x86_64-based supercomputer on the east coast of the U.S. (at least until someone takes the title away from us). It's spec'ed to start with 1200 dual-socket six-core servers..." but apparently has no idea what he's going to use it for.

If true, he'll have lots of cycles to sell for cheap and/or his organization is clearly n

How many commenters have BUILT a cluster!?

[mprinkey]

OK, I won't be too hard on the discussions above, but I read enough to try to give some real help to the OP. I get that this is basically an embarrassingly parallel application. So, that means a gigabit network is fine. That also means that single core performance is the ONLY indicator of the speed of the application. That means investing in anything AMD is a mistake. The best bang for the buck is quad-core Sandy Bridge CPUs. 4000 pounds is about $6300. I can build a quad-core 2.8 GHz Sandy Bridge no

Wow ...

[jon3k]

never thought I'd see the day where the general consensus was "just rent it!". this is slashdot, how can we not do this better, cheaper, faster and "free" than amazong's ec2?

Who isn't on a shoestring budget really?

[BlueCoder]

Everyone wants to get the most for their money be it in good times or bad.

Team up with other departments or use a cloud

[angel'o'sphere]

As others pointed out: use cloud first.
If you want your "own" grid, try to team up with other departments. Likely anotherone either did the same and can share resources with you or there is demand and they wait for one to set up a grid.

Ask slashdot

[ThurstonMoore]

Just get with the guy who did the ask Slashdot the other day that didnt know what to do with his supercomputer.

Re:

[serviscope_minor]

The university has a few server rooms scattered around of various qualities, though #4k's worth of kit could probably be scattered around a bit if necessary. The department in question does not have much history of heavy computational demands.

Though it's interesting what you mention about a colo. I had a look around at colo's on the open market and they're very expensive, compared to the budget. Oddly, the density seemed lower than expected. These days, modern high density servers can easily reach 1kW/U, bu

Re:

[DrgnDancer]

Biggest problem you're going to have, and the reason I think some of the people suggesting renting outside resources rather than purchasing may be on target, is storage (both physical and logical). You can get a machines with 4-8 cores, and 4-8 GB of memory for around $1000 in a rack mount case, so around 6 boxes with your budget (since you don't have VAT on .edu gear I'm assuming around 1/1.5 dollar/pound ratio). That doesn't get you a rack to mount them in and doesn't get you any storage beyond the hard

Re:

[Gyorg_Lavode]

Find some place that collects e-waste in a business-heavy area (or military area). You can usually find racks for cheap.

Since there have been a lot of rack mount suggestions, I might throw in my personal experience. I bought got a half-height portable rack for $500 a few years ago. I put it in a closet with a free-standing portable AC unit ($400) and vented it into the dryer vent. (If you lack dryer vents, a closet with a window will do fine. I'd recommend talking to building maintenance to see make

Re:

[secondhand_Buddah]

I tried, and all I got was images of Wallmart....

Re:

[bhcompy]

This is a good idea. The Air Force has a PS3 cluster doing similar work.

Re:

[allenw]

Yes, but I don't think Yahoo! is adding any more universities to it. In fact, I don't think it ever expanded beyond CMU.

ObDisclosure: I was on the ops-side of that project for Yahoo!.