There's A Cluster of 750 Raspberry Pi's at Los Alamos National Lab (insidehpc.com) 128
Slashdot reader overheardinpdx shares a video from the SC17 supercomputing conference where Bruce Tulloch from BitScope "describes a low-cost Rasberry Pi cluster that Los Alamos National Lab is using to simulate large-scale supercomputers." Slashdot reader mspohr describes them as "five rack-mount Bitscope Cluster Modules, each with 150 Raspberry Pi boards with integrated network switches."
With each of the 750 chips packing four cores, it offers a 3,000-core highly parallelizable platform that emulates an ARM-based supercomputer, allowing researchers to test development code without requiring a power-hungry machine at significant cost to the taxpayer. The full 750-node cluster, running 2-3 W per processor, runs at 1000W idle, 3000W at typical and 4000W at peak (with the switches) and is substantially cheaper, if also computationally a lot slower. After development using the Pi clusters, frameworks can then be ported to the larger scale supercomputers available at Los Alamos National Lab, such as Trinity and Crossroads.
BitScope's Tulloch points out the cluster is fully integrated with the network switching infrastructure at Los Alamos National Lab, and applauds the Raspberry Bi cluster as "affordable, scalable, highly parallel testbed for high-performance-computing system-software developers."
BitScope's Tulloch points out the cluster is fully integrated with the network switching infrastructure at Los Alamos National Lab, and applauds the Raspberry Bi cluster as "affordable, scalable, highly parallel testbed for high-performance-computing system-software developers."
It's an older meme, sir, but it checks out... (Score:5, Funny)
Did they make a Beowulf cluster of those?
Re: (Score:1)
I christen thee a Finnesburg [wikipedia.org] cluster.
Re: (Score:2)
Now we just need a cluster of the clusters in the name of Science.
No wonder pi's are hard to buy (Score:2)
When somebody buys 750 all at once.
It was my experience that pi's are hard to buy so i gave up trying to get one. Mind you when people use ancient rasbian os and make 'secure' email servers on port 26 and then get called out for issues it is good to see that somebody is using them properly instead of poorly.
Re: (Score:2)
Re: (Score:2)
The PI is sold everywhere, at the site + amazon etc...
The RPi Zero has been sold out continuously everywhere for months.
Re: (Score:2)
Theyâ(TM)ve been regularly in stock at my local Micro Center for months.
Micro Center's website listed them as "out of stock" for at least the last 4 months. They are currently not listed at all. So apparently they no longer carry them, or at least are no longer taking new orders.
I buy one almost every trip.
A Zero? Without buying a $30 "development kit" that includes a $5 Pi? I don't think so.
Re: (Score:2)
It was my experience that pi's are hard to buy so i gave up trying to get one.
Maybe once upon a time...?
You can get them from many of the major vendors these days. I usually get mine from RS but I suspect the likes of Farnell and so on sell them too. I expect Amazon sells them too!
Re: (Score:2)
Mind you when people use ancient rasbian os and make 'secure' email servers on port 26 and then get called out for issues it is good to see that somebody is using them properly instead of poorly.
You know Raspbian is basically Debian? So it's pretty solid for the most part.
Re: (Score:2, Informative)
Element 14 has 80,000 in stock and ready to ship.
750 would be a small order.
Re: (Score:2)
Not a programmer, are you?
Re: Cost (Score:5, Insightful)
You are missing the point! The idea is not to have an super computer but to emulate one. Writing code for stuff like thus is hard and running it on the real deal is expensive. This way the can emulate 750 core system at an fraction of the cost.
Re: Cost (Score:5, Informative)
You get effect of network latency to induce concurrency paradoxes that wouldn't happen on a shared memory system.
ObCarAnalogy: a single bus can move a lot of people, but if you're modeling highway traffic, you want to use many independent cars.
Re: (Score:3)
Re: Cost (Score:5, Insightful)
You are missing the point! This way the can emulate 750 core system at an fraction of the cost.
So, what point am I missing? The Xeon phi 7290 is 4k$ and has 72 cores, you can get 10 of those and get way more speed, shared memory benefit etc...
Entirely different architecture. The point of this scale model is to have a cluster of compute nodes with TCP/IP communication between them.
Re: (Score:3)
Not useless if you're debugging queue systems, schedulers etc.
Re: (Score:2)
You could do all that by setting up a bunch of VMs on a shared memory machine.
That will give you different latencies and different bottlenecks. The point of this system is not to crunch data, but to serve as a testbed for parallel software development. It is possible that they also use VMs, but that would be in addition to this cluster rather than a replacement.
Re: (Score:2)
Interconnects don't matter much. Whether you use InfiniBand, GigE or serial, you're just pumping TCP packets.
Re: Cost (Score:5, Insightful)
So, what point am I missing? The Xeon phi 7290 is 4k$ and has 72 cores, you can get 10 of those and get way more speed, shared memory benefit etc...
The shared memory is a detrator not a benefit if you're trying to have something which emulates an expensive distributed architeture. The point isn't to get lots of speed, it's to get a bunch of cores distributed over a local network in order to get a cheap test bed emulation of a much larger machine.
Re: (Score:2)
Your analogies have zero relevance to how HPCs work.
They are relevant. What is zero is your willingness to learn, or at least accept that other people know what they are doing.
Re: (Score:1)
The only difference between a cluster and and a shared memory machine is you pay by the minute on the cluster. If you haven't realized that consumption fee is the same whether you are debugging or doing actual science, then you are the moron who can't see the point of using a less powerful less costly cluster to do mockups.
You mean 75% fewer cores, fewer connects, no RAM (Score:5, Informative)
10 CPUs with 72 cores each is 720 cores.
750 SOCs with 4 cores each is 3,000 cores (and RAM and motherboards included).
The point is to have a massive number of cores in a large number of machines, to simulate a large number of machines, at the budget point. Your idea would have 75% fewer cores.
> shared memory
Yep, that's another problem with your idea. It would no longer be an accurate simulation. Well except your plan doesn't include any RAM at all. Or motherboards, networking, etc. You're going to need to buy 750 network cards to simulate 750 machines, motherboards each capable of holding 18 cards, a number of storage devices, etc. So maybe FIVE 7290 CPUs with exotic motherboards plus RAM, network cards, storage, etc. Five 7290s would provide 360 cores, vs the 3,000 cores they got with the Pis.
Now AFTER the research yields fruit, in a couple years someone might want to put the ideas into production using fifty 72-core processors which may cost $2,000 each.
Re: (Score:1)
1440 is NOT 3000!!! AND
your power budget went to
hell in a hand basket.
One other thing, your being
a DICKWAD.
Re: (Score:1)
Or you could use a single Xeon Phi, emulate the 750 Raspberry Pi's and the networking and still consume less power with more performance for a lower price.
Re: You mean 75% fewer cores, fewer connects, no R (Score:2)
Re: (Score:2)
Re: (Score:2)
And running 1.2GHz 4 core STB processors over 10/100 ethernet is going to be similar to clusters of dual socket 3GHz 54 core processors with 25+ Gbps interconnects? (aka Cavium ThunderX2 CPUs. Nobody is planning on building an ARM based supercomputer with only one CPU per node, let alone with IO limited smartphone/tablet/set-top-box oriented SoCs)
Re: (Score:2)
Re: (Score:2)
Someone should invent software for emulating a CPU, that way you could use one machine to emulate many.
I'd call it a virtual machine.
Re: (Score:2)
Re: Cost (Score:2)
Re: (Score:2)
The RPi modules are 4-core, so the cluster is 3000 cores.
Re: (Score:2)
I wonder how many Commodore 64's I could emulate at once on a decent sized workstation. Maybe 500?
Re: (Score:2)
Re: (Score:3)
ROFL. We're not talking about debugging the scientific number crunching code that will run on the actual cluster, but the cluster management software. The actual jobs to run may very well just be doing sleep(10000*rand()); if rand()0.1 call WriteAllTheDiskSpace; else if rand() 0.2 then call segfault_horribly(); else return SUCCESS;. etc.;, one should probably add in a few more "bad things", MPI calls etc.
Re: (Score:2)
Oh yes, true. Now why on earth didn't the folks Los Alamos think of that!? They must be complete idiots. You should write to them and explain your ideas - they might give you a job as their chief architect, or maybe their Head of Cost Cutting.
Raspberry Bi? (Score:2)
As in bidirectional communication I assume!
Excellent way to learn parallel programming. (Score:1)
If you cannot get 1000's of slow cpu's to scale, then wasting debug time on the big fast server is really a waste. Today's programmers need to learn how it used to be. Even with using RPI's they have an advantage. The network is much faster than what we had 20 or 30 years years ago. Internal busses are faster, ram/memory is faster, caches are faster. This is a smart way to spend money for a bringup development environment on the cheap.
Re: (Score:3)
That was 20 years ago. Today we don't get those kinds of performance leaps anymore.
Attack of the Errant Apostrophe (Score:2)
You don't need a supercomputer to figure out that the headline is poor usage. The Chicago Manual of Style will do that for you.
Re: (Score:2)
Thereâ(TM)s A Cluster of 750 Raspberry Piâ(TM)s at Los Alamos National Lab
There. Happy now?
Re: (Score:1)
No, there isn't a square-boxed question mark.
I hope it's not idling. (Score:2)
1kW at idle is a lot. You could cut that down by shutting down Pis in banks as they went unused, and firing them up again as needed. It wouldn't require very much more hardware, just some microrelay boards which can be driven by some of the Pis themselves.
Re: (Score:2)
1 kW is not a lot of power, it's the equivalent of someone always using the hand dryer in the bathroom.
It's not a lot of power on their scale, it's true. But it's a lot of power to waste idling on a low-power project, when it's easily avoidable.
It's probably not idling much anyway.
Re: (Score:2)
1kW at idle is a lot. You could cut that down by shutting down Pis in banks as they went unused, and firing them up again as needed. It wouldn't require very much more hardware, just some microrelay boards which can be driven by some of the Pis themselves.
Electricity in New Mexico is $0.11 to $0.12 per kwh. So at maximum they would save $0.12 per hour. You would spend far more in labor/coding/hardware than you would ever save in power costs. Plus you may introduce bugs or other issues that would take even more time to fix or delay useful work.
Do you work in academia?
Big deal (Score:2)
There's A Cluster of 750 Raspberry Pi's at Los Alamos National Lab
I saw a bunch of them at the grocery store before Thanksgiving, next to the apple ones.
Speed (Score:4, Interesting)
Purely out of academic interest, how fast is this thing? How does it compete with, say, a 16 core Xeon or Threadripper workstation?
Re: (Score:2)
A Rasberry Pi 3 can do 6GFLOPs, if you can keep it cool enough to not immediately start throttling. 6x750= 4.5TFLOPs. A single NVidia GTX 1080Ti does 4TFLOPs double-precision and 11.5TFLOPs single-precision.
The academic interest is that this actually has 750 separate and independent CPUs and nodes, so one can see how tasks scale and bottleneck. You can't accurately virtualize all these parameters.
The Megaprocessor Still Laughs (Score:1)
https://robertmcgrath.wordpress.com/tag/the-megaprocessor-laughs-at-your-puny-integrated-circuits-stephen-cass/
Re: Obvious unimportant topic (Score:1)
Fuck Beta!
Re: Obvious unimportant topic (Score:2)
Re:Obvious unimportant topic (Score:5, Insightful)
Re: (Score:2, Funny)
Re: (Score:2)
This happens so often that I think we need a new mod:
Score: -1 Wrong topic, you idiot
Re: (Score:1)