Mini-ITX Clustering 348
NormalVisual writes "Add this cluster to the list of fun stuff you can do with those tiny little Mini-ITX motherboards. I especially like the bit about the peak 200W power dissipation. Look Ma, no fans!! You may now begin with the obligatory Beowulf comments...."
Some preliminary performance results (Score:5, Informative)
Re:Floating point performance (Score:5, Informative)
Test Text. (Score:4, Informative)
The construction is simple and inexpensive. The motherboards were stacked using threaded aluminum standoffs and then mounted on aluminum plates. Two stacks of three motherboards were assembled into each rack. Diagonal stiffeners were fabricated from aluminum angle stock to reduce flexing of the rack assembly.
The controlling node has a 160 GB ATA-133 HDD, and the computational nodes use 340 MB IBM microdrives in compact flash to IDE adapters. For file I/O, the computational nodes mount a partition on the controlling node's hard drive by means of a network file system mount point.
Each motherboard is powered by a Morex DC-DC converter, and the entire cluster is powered by a rather large 12V DC switching power supply.
With the exception of the metalwork, power wiring, and power/reset switching, everything is off the shelf.
At present, the idle power consumption is about 140 Watts (for 12 nodes) with peaks estimated at around 200 Watts. The machine runs cool and quiet. The controlling node has 256 MB RAM , and an 160 GB ATA 133 IDE hard disk drive. The computational nodes have 256 MB RAM, each and boot from 340 MB IBM microdrives by means of compact flash to IDE adapters. The computational nodes mount
Power and Cooling
Mini-ITX boards have very low power dissipation as compared to most motherboard/cpu combination in popular use today. This means that a Mini-ITX cluster with as many as 16 nodes won't need special air conditioning. Low power dissipation also means low power use, so you can use a single inexpensive UPS to provide clean AC power for the nodes.
In contrast, a 12-16 node cluster built with Intel or AMD processors will generate enough heat that you will likely need heavy duty air conditioning. Additionally, you will need adequate electrical power to deliver the 2-3 kilowatts peak load that your 12 node PC cluster will require. Plan on having higher than average utility bills if you use PC's...
Hardware Construction
The cluster is built in two nearly identical racks. Each rack has two stacks of three motherboards and dc-dc converters mounted on aluminum standoffs.
The compact flash adapters used to mount the microdrives are also in stacks of three. Each stack of boards is mounted on a 7 inch by 10 inch 0.0625 thick 6061-T6 aluminum plate as are the microdrive stacks. There are seven metal plates in all, in each rack.
The top cover plate has the mounting bracket for the 6 on/off/reset switches.
The plate below it is home to the power distribution terminal block. The power delivery cable for each rack is heavy duty 14 gauge stranded wire with pvc insulation. The power cabling from the terminal strip to each of the dc-dc converters is 18 gauge stranded pvc insulated hookup wire. The wiring for the power/reset switches is 24 gauge stranded, pvc insulated wire.
The top rack houses nodes one through six (node one is the controlling node). The bottom plate of the top rack also houses the 160 GB ATA-133 hard disk drive used by the controlling node. All other nodes make use of the IBM microdrives. Node number three has a spare compact flash adapter which can be used to duplicate microdrives for easy node maintenance.
The disk drive and power cabling to the motherboards was dressed as was sanely possible on the back panel. The liberal use of nylon cable ties helps reduce the ten
Re:FLASH... (Score:1, Informative)
Re:Floating point performance (Score:5, Informative)
But, most mini-itx systems are very small in size, and strive for quiet or silent operation. So, there are obvious problems with the P4's heat/power requirements. Perhaps a better solution is the Pentium-M in a mini-itx form factor. It has pretty good performance, at a low power/heat level: Pentium M [commell.com.tw]. But, most of the Pentium-M boards are intended for industrial or OEM use, so they are hard to find in retail, and are pretty expensive.
slashdotted already? (Score:5, Informative)
I managed to get it mirrored here:
page 1:
http://www.phule.net/mirrors/mini-itx-cluster.htm
page 2:
http://www.phule.net/mirrors/mini-itx-cluster2.ht
page 3:
http://www.phule.net/mirrors/mini-itx-cluster3.ht
more information ... (Score:4, Informative)
Re:Floating point performance (Score:5, Informative)
Coudn;t find a link though, sorry.
Re:Cool stuff ... (Score:5, Informative)
Yes, I guess that most current BIOSes of the newer boards do, especially the consumer-ish stuff. We just used the stock Shuttle XPC with its FlexATX-board.
Clickable link... (Score:1, Informative)
Re:Floating point performance (Score:2, Informative)
Re:Floating point performance (Score:5, Informative)
The floating point is just a convenience. Almost any algorithm can be modified to work with fixed point precision -- and without loss of performance.
Of course, many people will insist, they need FP to be able count dollars and cents -- they don't even think of counting cents (or any other fractions of the dollar) with integers, for example.
These are, usually, the same people, who have troubles defining bit...
Re:Inexpensive for testing purposes, (Score:3, Informative)
Mini-ITX? Bah! Nano-ITX!!! (Score:4, Informative)
MB, slim DVD and laptop HD in a case the size of a large paperback book!
It will make my "K-Mart Toolbox Mini-ITX PVR" look like a full tower in comparison!
Re:I built a fanless ITX system... (Score:4, Informative)
Re:Inexpensive for testing purposes, (Score:5, Informative)
A beowolf of mini-itx boards is probably the cheapest way to get bragging rights. As a practical way of fast and cheap parallel computation they are not.
However, I have purchased three (V10000 boards) thus far and intend to add more to my network as low power (as in Watts) servers.
I worked out that given the power of 10.78W (source: mini-itx.com's power comparison tool) for the V series (probably the one with the slowest CPU in the series, board only), I could save a fortune on electricity compared to a more regular computer.
The electricity company sells electricity at the rate of 0.63 ($1.18) per watt per year. Compared with a standard PC of 100W, I can regain the purchase costs (in savings) of the board and memory within two to three years.
Also, I found rack mount chassis [icp-epia.co.uk] available cheaper than one for a regular sized case. This influenced my decision a little - who doesn't want a network of rack mounted computers?
Overall, because of the low price and low power the mini-itx boards are a no brainer if and only if the CPU power of each computer isn't important.
Jonathan
Re:Inexpensive for testing purposes, (Score:2, Informative)
-- Brooks
Re:Floating point performance (Score:2, Informative)
Re:Floating point performance (Score:4, Informative)
I'm not aware of any Athlon-based boards, but mostly because I'm satisfied with my Via-based M10000 board.
Re:Floating point performance (Score:5, Informative)
Re:Why Microdrives? (Score:3, Informative)
Re:Cool stuff ... (Score:3, Informative)
Re:Seriously, though... (Score:4, Informative)
The answer to this is...
Yes! (2) [linitx.com] and Yes! (4) [linitx.com]
Re:Floating point performance (Score:3, Informative)
Whoops, I made a mistake. He actually said his 12-node VIA cluster has more power than "four 2.4 GHz pentium 4 mcahines used in parallel". Not SMP!
Sorry about the mistake.
steveha
Re:FLASH... (Score:3, Informative)
"CompactFlash(R) is a small, removable mass storage device."
So you are correct in noting that he is actually using HDDs, not flash, but in the same time, he is using CompactFlash (BTW the CF pinout is IDE compatible, so to hook up your CF to your IDE bus all you have to do is to manage to connect the wires of the IDE cable and the power cable to the card)
Re:Floating point performance (Score:2, Informative)
$49.95 with Celeron 366
Re:Imagine.. (Score:5, Informative)
The original links went to NASA/GSFC [nasa.gov], but the Beowulf project central site has moved.
Re:I built a fanless ITX system... (Score:4, Informative)
My requirements were essentially (1) no moving parts, (2) affordable if not cheap, and (3) small. I settles on one of these [transcend.com.tw]. Debian is fine on 128MB, with 512MB of ram and no swap. Performance, it should be said, sucks. The next step up, for slightly more performance, much more capacity, and a whole lot more cost, is here [m-systems.com]; but I wanted to avoid using a case that needed drive bays, plus I haven't pockets that deep.
Neither of those is likely to be what you want for a database system, though. You're probably more in the market for a bunch of ram and a battery, unless your primary concern is reliability. If speed is the goal, you want this [cenatek.com], or, for more capacity and more money, this [soliddata.com]. Note that I haven't used either extensively, and in playing around with the rocket a little, I was surprised just how much of a bottleneck PCI became. Also, the rocket doesn't have a battery... so really, unless you have a board with 8GB of memory, and you just need another 8GB of low latency space, it's not such a great deal today.
If you fit into any of the niches above, solid state is wonderful. It's always more expensive than you think, though. And for any database systems I've dealt with, a disk is without question the way to go, perhaps with more memory on board. But if you want any further tips, I'm glad to help.
Re:Floating point performance (Score:2, Informative)
If you use the Intel compiler and its vectorization stuff then you could easily do it in C. It will generate the SSE instructions.
Most compilers like GCC won't give you maximum performance without resorting to asm.
Re:Inexpensive for testing purposes, (Score:3, Informative)
RTFA... he compares performance to 4-6 P4s. He does clustering for a living so I'm assuming he knows how to measure and compare performance at this scale...
I'm surprised no one linked to these already (Score:1, Informative)
The pictures are cool.
Re:Floating point performance (Score:4, Informative)
I have the VIA EPIA 8000 (not sure what the V and A modifiers mean), with an Ezra core. FYI, Eden isn't a core, it's an initiative. The VIA Eden is aka VIA EPIA 5000, and was the first fanless Mini-ITX. Eden was the development product moniker, and came to refer to the motherboard that was first produced from that initiative. It can also refer to any C3 CPU made to run fanless.
Back onto the original topic; my EPIA 8000 with an Ezra core runs the FPU at half clock. This document [via.com.tw] on the differences between the Ezra/Ezra-T and Nehemiah cores indicates that one of the fundamental differences between the two is the full speed FPU. So I doubt that the article you quoted is accurate...
Just some more info... Nehemiah was manufactured at 933 MHz, 1 GHz, and speeds up to 2 GHz are planned. The Ezra was manufactured at 533 MHz and 800 MHz in its first run; the 533 is also known as the Eden. The Ezra-T (the second run of the Ezra) was made at 600 MHz (aka Eden), 800 MHz, 933 MHz, and 1 GHz.
Re:I wonder too (Score:4, Informative)
dyne:bolic [dynebolic.org] is a Live CD distribution, very small, that can be PXE boot, with full audio/video capture/editing/processing/streaming capabilities plus the usual suite of tools, a few games and whatnot... and is auto-clustering on a private network.
Re:Floating point performance (Score:3, Informative)
Here's an old review [tech-report.com]. The VIA processors aren't built for speed; they're built for low power consumption. In that department, they're great. They're also relatively cool, temperature-wise.
I've got a machine based on a 1GHz Ezra; it's really not as bad as it sounds. Not a stellar machine, but it's got better performance than the K6-2 it replaced (but not a whole helluva lot better, especially when there's lots of floating-point math involved, for the reason stated above) and it doesn't require insane cooling techniques.
Beowulf. The REAL deal. (Score:4, Informative)
of spear-armed Danes, in days long sped,
we have heard, and what honor the athelings won!
Oft Scyld the Scefing from squadroned foes,
from many a tribe, the mead-bench tore,
awing the earls. Since erst he lay
friendless, a foundling, fate repaid him:
for he waxed under welkin, in wealth he throve,
till before him the folk, both far and near,
who house by the whale-path, heard his mandate,
gave him gifts: a good king he!
To him an heir was afterward born,
a son in his halls, whom heaven sent
to favor the folk, feeling their woe
that erst they had lacked an earl for leader
so long a while; the Lord endowed him,
the Wielder of Wonder, with world's renown.
Famed was this Beowulf
Sample from the Project Gutenberg Text of Beowulf.
Why not do yourself a favour and download it. Classic stuff.
Re:Floating point performance (Score:2, Informative)
Re:Power. (Score:3, Informative)
Well, I just applied my, admittedly imprecise, clamp ammeter to the power cable, and got ~2 amps @ 120 V = 240 W. Which means, 240W/6Gflops = 40W/Gflop. That cluster has 200W/3.6 Gflops = 55.555... W/Gflop. Slightly worse...
I admit that hardware interfacing is getting to be a problem for us hobbyists, since the demise of the ISA bus, but I have been able to get along with the parallel interface. I just hope the USB interface doesn't get too popular too soon!
Re:Floating point performance (Score:3, Informative)
Think of FP as a lossy compression algorithm. It allows the use of orders of magnitude less number of bits because it alters the density distribution of the representable numbers to meet the above specificiations.
This makes sense. With integers the density is uniform, which is impediment in some cases, but of help in others. [Any attempt to quantify the number of cases in each group is silly and will reveal nothing, but the attempter's personal bias. With my bias, I'll insist you are underestimating the number of cases, where such uniform distribution of density is useful and desirable.] However, unless you carefully choose the basic unit, you don't have control over the precision distribution. If most of your computations involve quantities on far edges of (you claimed 20 orders of (decimal?) magnitude) -- you are less precise than you may realize and the (carefull) use of integers may improve your results.
My bias, having to do with this story and the great^n grandfather post is towards numerical cluster computing. Now I might be narrow sighted but having looked at the types of things which are generally done on supercomputing clusters (from books, web sites, and of course observing the supercomputing cluster which I work on, which is #50 in the world, so not exactly small) I see very few applications that do not have the requirements my previous post describes. Lots of multiplication and division by wildly different orders of magnitude... this can be found in almost all branches of science... only discrete math seems to have little of this.
Also it should be noted that through the use of interval arithmetic, you can quickly find the maximum bounds on the error in a given floating point program. This is something that, to my knowlege, fixed point arithmetic cannot support. So, the amount of error is actually quite evident to the experienced numerical computer, and not "more than I realize". Any kind of numerical processing requires people to be careful, fixed point most of all (because most languages don't have fixed point infix operator support, and all shifts etc must be done manually).
Also you should note that many applications do not have a "basic unit". Of course, they all have basic units! Usually, it will depend on the application's desired precision.
A basic unit is an absolute number, however most error bounds must be given as relative error, and if the range of inputs on the program is unknown, then it is impossible to derive a "basic unit". What's more, the numer is generally meaningless in the types of constrained numerical systems which I described. The granularity is usually only meaningful in terms of significant digits, not absolute differential units.
For instance, what is the "basic unit" of length? Depends on the application. In yours, it is, probably, some fraction of light year.
The unit length in our simulation is, I believe, the radius of the neutron star. This can be as little as 10-100 kilometers. It should also be noted that the "base unit" represents the minimum discernable difference in measurements, and therefore must be much much much smaller than that for accurate calculations after many many multiplications.
What if your "basic unit" of length is of a radicaly different exponent than your "basic unit" of energy or "basic unit" of time? Who cares? Even if my program operates internally on units as horrible as, say "pounds per square inch" (a.k.a. PSI) -- so be it. If 4.5 Newtons is my basic unit of force I want and the 0.025 meter is as precise as I want the length to be -- fine.
What do you mean who cares? The quintessential question here is the varying orders of magnitude involved in calcuations! not to mention the fact that to multiply fixed point numbers together you will have to find a common base unit,
Re:Floating point performance (Score:2, Informative)
I would suggest that before you criticize a large research effort that you take a bit of your precious time to research it yourself first.
Also, we happen to live in a democracy, so feel free to vote for candidates that are not interested in promiting science, technology, and a better understanding of the universe. You do vote, right? And of course if you feel that our government doesn't represent you and never will, you can feel free to choose another country of residence...
Speaking of which, I seem to remember this weird network project that was funded by money spent on high energy particle physics. You know, that area of science that is so far from practical that it doesn't produce anything usable... I seem to remember this project being called the World Wide Web... A curiosity at best, we'll never see a return on our investment!