Supercomputers - Does the Cabling Matter? 95
papaia asks: "Having watched, for a while, the development in the area of high-density server hardware solutions (i.e. blade servers), like IBM's 'top gun', and their increased presence in Data Centers, I have been wondering if anybody has had any experience (thus comments) in regards to how important - in such highly priced solutions - is (or could be) the [always neglected] cabling, connecting the servers. One such comment caught my attention, in this regard. Slashdot, how important is the server cabling infrastructure in your Data Centers, and how do you resolve the cable management aspect of it?"
Always neglected??? Speak for yourself... (Score:5, Informative)
Allow for larger racks so there's room for cables and PDUs. Really.
Keep things neat with a raised floor, cable tray underneath. A second tray overhead can be handy, but can interfere with moving equipment, so don't fit it unless you'll need it.
Be aware high speed cables, particularly fibre, have a minimum bend radius - go beyond it, and you have fucked up the cable. Also, be aware of sag due to weight of cables - compress the cable too much with the weight of the bundle against cable ties, and you can damage it.
Crosstalk is seldom an issue these days, but be wary of laying power and network too close - even if there is negligible interference, you're safer if they're separated by a decent amount.
Patch panels are useful - use them. Run cables to patch panels and patch panels ->switches, don't go machine -> switch directly (unless you're doing really high-end stuff (for 1GigE copper or fibre, patch panels are fine).
Re:sentence (flow) (Score:1, Informative)
print("hello"); in C is (print "hello") in lisp. The name of the function moves inside the left paren when compared to C, that's all.
A Lisp program is like a c program with lots of function calls within function calls:
funca( funcb( 1,y ), funcc( 3 ) ); in C would be (funca (funcb 1 y) (funcc 3)) in Lisp. Yes, they are different. But one is not particularly easier or harder than the other, though deeply nested structures are IMHO easier in the lisp syntax.
Re:literally speaking, no (Score:3, Informative)
As you probably know, standard twisted pair cable comes in several grades, or categories. Cat 3 is the minimum acceptable for 10baseT, Cat 5 for 100baseT, and Cat 5e for 1000baseT. Cables are sorted into categories depending on the highest signal frequency they can pass reliably - 16 MHz for Cat 3, for example. Unfortunately, this means if you were to wire a cluster with Cat 3 wiring and try to run a 100baseT network (at 100 MHz), you would probably get errors in the datastream, resulting in lost frames; this can cause subtle errors that will spawn many headaches down the road.
The bottom line is, don't skimp on your cables. While Ethernet can be quite resilient (I've seen 10 meg go over a barbed wire fence, albiet slowly and with about 80% dropped packets), skimping on cable to save a few bucks can really cause serious problems down the road. The money you "saved" by installing cables that aren't up to spec will be obliterated tenfold when you factor in the cost of ripping out all the old cable and running new stuff that's up to spec.
Re:You don't want to raise the ire... (Score:3, Informative)
Re:Always neglected??? Speak for yourself... (Score:2, Informative)
Re:You don't want to raise the ire... (Score:3, Informative)
Under the floor? Reconsider! (Score:3, Informative)
The under-floor space can be used for AC power, if you use AC, but that's usually just for convenience outlets. Downflow air handling units that use the floor for air distribution are good too. I've even seen installations where the DC power cabling was run under the floor, and it simplified things greatly. But please, don't put your signal cables down there.
For one, it's easy to drop a tile into the floor while trying to remove it. One bad suction cup can cause the crushing or cutting of a cable. For another, it's awkward to feed cables through the little cutouts at the bottom of a cabinet. I've seen a lot of dirty or damaged connectors because of this.
If you're not bolting your cabinets to the floor, it also creates a shear point if the cabinet shifts. Please do bolt down your racks and cabinets, because they can tip.
Hiding the cabling also encourages poor workmanship. When someone has 20 feet of slack to store, and they throw it in a clump under the floor, it's a nightmare when another cable in the same area has to be pulled out. The initial infraction would've been noticed immediately if it'd been overhead, in plain view.
A well-designed overhead cable rack system is superior to any floor system. It's cleaner, because there's literally less dust colleting on it. Running cables overhead doesn't involve dragging them through a pile of connection-ruining crud. It's easier to install, because you don't have to contend with tile supports. It's easier to expand, because you can visualize the whole layout easily, and see where the congested areas are.
Furthermore, overhead rack is a natural companion to fiber trough systems, most of which are intended to be overhead. If you have a mix of fiber and copper, and most of us do, you owe it to yourself to plan a system that accomodates large amounts of both. As equipment density rises, the amount of cabling you'll need to bring to each rack also rises. Plan for that.
Also, plan for slack runout areas. The cables are never the exact length you need. Running them back and forth in the rack can create all sorts of tangle problems. Having a designated path to run your slack loop down can really make tearouts less dangerous.
Also, don't underestimate the sheer size of the cabling you're dealing with. I saw one particularly bad example, where a company had laid out their aisle of patch panels very carefully. The bottom of each bay was for panels that went to transport equipment, and the top was panels for other equipment. That way, most cross-connects could be made without leaving the bay. There were cable management rings to accomodate the occasional jumper that had to go between bays. It worked great.
Then they merged with another company, and the recordkeeping system changed. The new system made port assignments automatically, and it didn't respect the physical layout of what was where. Now the majority of jumpers were long, inter-bay runs. Over time as circuits got moved around, the management rings got filled, overoaded, and eventually stuffed to the point that the mass of wire was essentially solid. You could punch the bundle and it would go "thud". Pulling out a jumper was likely to burn through its neighbors simply due to friction, so they stopped pulling old ones out.
Eventually they added a dedicated piece of cable rack, and run all interbay jumpers up there. It was ugly, and awkward, but it worked. The initial system was much better, but relied on a level of care and planning that the new owners weren't willing to provide. Consider this: Will your successor's successor curse your name, or laud you for laying out a comfortable, expandable environment?
Re:Always neglected??? Speak for yourself... (Score:3, Informative)
Another tip - where copper signal cabling (ethernet, serial, scsi, whatever it may be) has to come near power cabling, always cross them at right angles instead of running them parallel to each other, this greatly reduces the chances of inductive interference.
Re:literally speaking, no (Score:3, Informative)
I've seen many people fooled by listening.
Take two stereo's. One has 0.005% THD and the other has 0.1% THD. Do a side by side test, but have the 0.1% reciever set 3 DB louder. Guess which sounds better?
The Audioholics article shows many gross misunderstandings
What I don't understand people who spend $10/foot for speaker cables, then don't want to damage an expensive cable so they have a 30 foot cable to go 5 feet to the speaker.. If you are measuring results of cable inductance, A cheap 5 foot cable of the same conductor size as an expensive cable, is better. Cable length is important. This is more important the more the load impedance does not match the transmission line impedance. A 75 ohm SPDIF cable feeding a 75 ohm SPDIF receiver is a good match and will have minimum distortion and thus few data errors. A 120 ohm impedance $10/foot speaker cable feeding an 8 ohm impedance speaker is going to see high frequency attenuation due to the inductive reactance presented by the speaker cable. So many cable manufactures try to go with low capacitance cables to reduce the shunt capacitance. Low capacitance means a higher impedance because the inductance per foot remains the same. This makes a larger impedance mismatch between the cable and speaker. A high capacitance cable will have the capacitive reactance canceled by the inductive reactance resulting in a lower impedance cable. This would be a better match to the speaker. Any good radio tech knows the best response with the least reflected power is when the load impedance matches the transmission line impedance. The insulation used is important. Low dilectric loss is important to reduce high frequency attenuation.
Just for grins, visit your local wire and cable outlet and try to find sweep tested 8 ohm impedance wire...
Where it counts to have quality known impedance cable, it comes sweep tested. They don't sweep test speaker cable because the cable impedance is such a bad match to the load. In this case, shorter is better.
That's why in the days of RG58 ethernet, the cable was terminated with 50 ohm terminations. The termination did not overload the network cable, but eliminated unwanted reflections that would contaminate the data.
This is also why pro audio (stage) uses low impedance microphones. The long haul from the mike to the mixer is done using a low impedance microphone feeding a shielded twisted pair cable feeding a low impedance mixer. Typical impedances are 100-250 ohm. Plugging in a high impedance microphone will overload it resulting in poor sound. Plugging in a low impedance microphone on a long cable into a high impedance mixer also results in poor sound unless a matching transformer is used. When matching the cable impedance to the source and load, you get the most power transmission with the least distortion.
When you go very short distances such as the one you mentioned, (SPDIF) then neither the capacitive or inductive components of a cable are very significant, and as mentioned, a bad match may work OK for a short run. In a short run, cable capacitance and inductance impacts on the signal are small.
In theatrical lighting, the DMX512 standard specifies the need to terminate the data cable into it's impedance which is near 120 ohm. Many manufactures mention the termination is not needed for short runs. This is in violation of the spec, but they have found it still funtions for runs of less than about 30 feet. Longer runs must be terminated because data errors will cause problems. DMX512 operates at 250K Baud. The 6Mhz SPDIF signal runs at 24X that rate. Cable problems will show up at shorter distances on the higher frequency applications.
Re:literally speaking, no (Score:5, Informative)
Happy you got modded funny, however Long Reach Ethernet [cisco.com] (LRE) does exactly what it says with very good throughput (we're nowhere near the alledged 80% packets loss of the parent post).
Oh, and the video clip which shows Ethernet over barbed wire is at the same url on the right-hand side where it says "Video: Charlie Giancarlo Demonstrates LRE Technology". It's nice to see it once for the "Wow!" effect. You'll also see the demo go over Cat3, Cat5, speaker cable, coax and lamp cord...
Re:WARNING: In wall stereo speaker must be insulat (Score:3, Informative)
For in wall applications, you can't use plain old insulated copper anymore, you require something with a fire-resistant outer jacket.
Most companies accomplish it with PVC or PVC and foil.