Forgot your password?
typodupeerror
Data Storage It's funny.  Laugh. Sun Microsystems IT

Why Not To Shout At Your Disk Array 125

Posted by timothy
from the that's-what-your-subordinates-are-for dept.
Brendan Gregg of Sun's Fishworks lab has an interesting video demo up at YouTube demonstrating just how bad vibes, if expressed with sufficient volume in front of a rack full of disks, can cause a spike in disk latency. White noise, evidently, doesn't do them much harm. (Maybe they just feel awkward to get yelled at on camera.)
This discussion has been archived. No new comments can be posted.

Why Not To Shout At Your Disk Array

Comments Filter:
  • by slugtastic (1437569) on Friday January 02, 2009 @05:03AM (#26297927)
    ...always made me laugh.

    he's like the crocodile hunter of loud server rooms

  • by Anonymous Coward on Friday January 02, 2009 @05:05AM (#26297937)

    People yelling too much at their computers

  • by rolfwind (528248) on Friday January 02, 2009 @05:06AM (#26297943)

    It's been known for a long time vibrations are not good for discs (see notebooks). Even by early 90s music CDs had skip protection. If a disc skips, latency will of course momentarily increase. And with tolerances down even further, it's probably worse than back then.

    In 10-15 years it won't matter anyway, almost everything will have SSD by then.

    • Re: (Score:1, Insightful)

      by Anonymous Coward

      On a bright side, you can read all the crap without having to tolerate that god aweful stylesheet!

    • by Chris Snook (872473) on Friday January 02, 2009 @06:34AM (#26298263)

      Prior to the advent of skip protection in portable CD players, you could make them skip for several seconds just by shouting at them briefly, because it took much longer to recover from the vibration than the duration of the shock itself.

      • by rolfwind (528248)

        Yep. Even on my modern CD sound system with skip protection (cache), if I crank the volume up loud enough, the speakers will eventually vibrate the mechanism for so long as to cause the player to shudder and skip.

    • Are you sure about this?

      Let's say that I have my phone in my pocket, will that not affect?

      Yes I agree that plain vanilla yelling will probably have no effect whatsoever. BUT what about radio waves?

      Not to say it will stop me from buying SSD, just wondering out loud.

      • by X0563511 (793323)

        SSD is digital. Radio is analog.

        Your radio waves will need to cause at least 1.5v (or more, depending on the logic level of the device in question) to have a chance of flipping a line.

        If the waves coming out of your phone can do that, call your doctor. You probably have at least 3 types of cancer.

    • by trolltalk.com (1108067) on Friday January 02, 2009 @08:44AM (#26298793) Homepage Journal

      It's been known for a long time vibrations are not good for discs (see notebooks). Even by early 90s music CDs had skip protection. If a disc skips, latency will of course momentarily increase. And with tolerances down even further, it's probably worse than back then.

      There's BAD vibrations, and then there's GOOD Vibrations. [youtube.com]

    • by RobinH (124750) on Friday January 02, 2009 @10:59AM (#26299743) Homepage

      "Skip protection" on a hard drive is pointless. This is a fundamentally different scenario. With a CD, you can read the data *much* faster than you really need to read it, because you only need the data fast enough to convert it into sound. Plus you almost always know which piece of data needs to be read next, because the song is linear.

      On the contrary, with a hard drive, read speed is (usually) the bottleneck, so you want the data sent to the processor as soon as you can pull it off the disk. Also, hard drives are much more random access, so you can't guess the location of the next read and read it before the CPU requests it. The only thing you can do is cache frequently accessed data in memory, which the operating system already does.

      • by ultranova (717540)

        Also, hard drives are much more random access, so you can't guess the location of the next read and read it before the CPU requests it.

        Bullshit. Most files are read sequentially. Which is why most hard drive OS-level caches have read precaching.

        • When you load one of my webpages, it usually reads about 300 different tiny files to process the request.

          A drive like the one in TFA (if used for a webserver) is likely to be reading a few thousand different files every second, each one is on a different location on the disk and no way to determine which file will be read next.

          SSD drives have shown to be multiple orders of magnitude faster than the most advanced hard drives available today for web servers.

          • by setagllib (753300)

            Try to run a webserver with caching built in. Chances are good 295 of those 300 files are going to be reloaded per page and can be stored in memory indefinitely. Last I heard, memory is still faster than SSD.

  • Interesting... (Score:5, Insightful)

    by TFer_Atvar (857303) on Friday January 02, 2009 @05:08AM (#26297949) Homepage
    I wonder if the latency would vary by the pitch and tone of the person yelling. If that's the case, I'd wonder if that could be extrapolated into reconstructing whatever was being said. Granted, if you're yelling that loud, the person in the next county is more likely to hear you first.
    • by sakdoctor (1087155) on Friday January 02, 2009 @05:39AM (#26298043) Homepage

      I would say yes. When I was a teen my mother walked into my room and started moaning about the mess.
      Right then, windows blue screened and later I found the hard drive was completely dead. (Think it was a 15GB Maxtor or thereabouts) That cost me some pocket money to replace at the time.

      If you have women living in the house, factor this into your backup procedure.

      • Re: (Score:3, Funny)

        by apoupc (569200)
        "I would say yes. When I was a teen my mother walked into my room and started moaning..." I thought this was going to go somewhere else...
    • by Thanshin (1188877) on Friday January 02, 2009 @05:42AM (#26298057)

      You made me wonder; if the the effect could be detected and "read", a you say, it would be possible to use it as a way of transmitting information to the computer by shouting at it.

      I then remembered microphones.

      • Re: (Score:3, Interesting)

        by MichaelSmith (789609)
        My first computer had a 6502 CPU with BASIC and a machine code monitor in ROM. I found that the cassette interface could be used as a sound card if I configured a tape player to record and play back at the same time. For the output channel any AM radio would do because the CPU only ran a 1Mhz and it was leaky as hell.
      • by yanyan (302849)

        With a very loud set of speakers blaring at your drives, you'd do very well to step out of the room and close the door securely before shouting. Since most machine rooms i've been in were soundproof, i can't help but ask the question,

        If you yell at your disk array but are not there to hear it, do you make a sound?

      • by sshir (623215)
        Actually, that's not a bad idea for a covert channel [wikipedia.org].
    • by TheLink (130905)
      Drive seek times are typically in the order of 1-10ms (depending on whether to adjacent track or full sweep), that means it's 100-1000Hz which at least overlaps human vocal range.
  • by nitsnipe (1332543) on Friday January 02, 2009 @05:14AM (#26297969)
    It bothers me,
    How this guy actually made the discovery.

    He must have let off quite a bit of steam towards that rack.
    • Just like have you ever wondered what the first person to discover you could get milk from a cow was doing at the time?
      • by bytesex (112972)

        Not to be a dry pedantic killjoy, but he was probably watching a young calf drinking.

      • Years back I broke my ankle and had it screwed together for awhile. The screws are out now, but I've got them saved in a container and - other than the material - they really don't seem much different from those in my shop.

        So was there some doctor who moonlighted as a carpenter, and one day looked at a broken, out-of-place bone... then at his workbench wood-projects ... then back at the bone.

        Even today, looking at the X-ray of those screws firmly drilled through my bones gives me a bit of a creepy feeling,

    • Re: (Score:3, Funny)

      by thermian (1267986)

      'The very existence of flame-throwers proves that some time, somewhere, someone said to themselves, You know, I want to set those people over there on fire, but I'm just not close enough to get the job done.'

      George Carlin

      • by Dun Malg (230075)

        'The very existence of flame-throwers proves that some time, somewhere, someone said to themselves, You know, I want to set those people over there on fire, but I'm just not close enough to get the job done.'

        One of my favorite bits, to be sure, but one which becomes a lot less funny when you're being shot at by a couple [Germans|Japanese|*] with a machine gun through a tiny slit in a concrete bunker. That's about when being able to squirt a stream of flaming gasoline some arbitrary distance with some degree of accuracy begins to sound mighty useful...

        • by xenocide2 (231786)

          So you might say, there are people... and you want them on fire... but you just can't get close enough?

    • The guy is a sysadmin. No surprises at all from this quarter as to how he came across such a discovery...

    • I rediscovered it when my PC slowed down while I was playing the youtube clip
  • ...why?

    I'm sure there's a good reason, but... He's using dtrace, right? Thus implying Solaris? Thus implying ZFS?

    If you've got ZFS, why would you do JBOD?

    Or did I just mis-hear him?

    • Re:JBODs? (Score:4, Informative)

      by Anonymous Coward on Friday January 02, 2009 @05:41AM (#26298053)
      If you've got ZFS, why would you do JBOD?

      A few reasons.

      • If anything, I'd have figured you're more likely to setup a disk array as a JBOD with ZFS than (say) UFS. After all, you can get ZFS to do RAID0 for you (ZFS can probably also RAID1, but it's better to do that in hardware).
      • Solaris-10 still supports non-ZFS filesystems (VxFS) which I imagine you'd still want to use in some circumstances.
      • Their customers might be running an earlier version of SunOS/Solaris. He might simply be using DTrace to look at a customer bug-report from another prespective.
      • Indeed, even though he's debugging in Solaris, the customers might not even be running Solaris. Sun hardware is really sweet and is supported on both Microsoft Windows and various distributions of Linux.
      • And even if all the above reasons don't apply, using a JBOD is a good way of eliminating variables if you're trying to isolate/trace a potential hardware issue.
      • by setagllib (753300)

        ZFS RAID1 (and RAID-Z and RAID-Z2) can heal over corrupted blocks from the redundant copy. This is not possible with hardware RAID since it doesn't know which copy is valid.

    • Re:JBODs? (Score:5, Informative)

      by paulz42 (638751) on Friday January 02, 2009 @05:59AM (#26298125)
      Sure, with ZFS JBODs are the preferred storage. Let ZFS do end-to-end management of the storage, from the file level to the raw disk blocks. That way it can do it's end-to-end error checking and possible correction. If you do RAID1 in hardware ( really just firmware in the storage box) you trust that software to detect all problems and correct them or report them. That software may not do checking to see if both branches of a mirror are correct and pass on bad data upstream. ZFS will detect this because of it's checksums, but it will not be able to correct this. If ZFS is doing the mirroring it will detect it and read the other mirror, if that checksum is ok, it will correct the error and continue.
    • Re:JBODs? (Score:5, Interesting)

      by Chris Snook (872473) on Friday January 02, 2009 @06:03AM (#26298147)

      ZFS implements software RAID on top of JBOD. The box full of disks itself need not have any RAID controller, and if you're using RAID-Z, it would probably be a waste of money to spring for one, unless you go for the super-high-end for performance reasons.

    • Re: (Score:3, Insightful)

      by drsmithy (35869)

      Or did I just mis-hear him?

      "JBOD" in this context will be a reference to the style of disk array (eg: vs one with a RAID controller like the Dell MD3000), not the ZFS RAID level.

      • Right. Question is whether the disk array is concatenating the drives somehow, or if ZFS can actually see each individual drive. I would think the latter would be much more useful.

        • by drsmithy (35869)

          Right. Question is whether the disk array is concatenating the drives somehow, or if ZFS can actually see each individual drive. I would think the latter would be much more useful.

          Given it's the ZFS folks at Sun, it will be the latter.

  • Might the the drives themselves be sensing the induced vibration via an embedded accelerometer and momentarily parking the heads to avoid damage? It seems like the marketing folks shouldn't have too hard of a time putting a positive spin on this behavior.

  • if expressed with sufficient volume in front of a rack full of disks

    I wonder about the results of eliminating the superfluous "full of disks" part.

    Interestingly enough, it's precisely here, where the omission would both be understood and not bring unwanted connotations.

  • Great.... (Score:4, Funny)

    by Whillowhim (1408725) on Friday January 02, 2009 @05:33AM (#26298027)
    Now when Skynet finally goes sentient, it'll sue for emotional abuse. I thought metal death machines were bad, but now Lawyer-bots? We're doomed.
  • I doubt that white noise vs. voice has anything to do with it. He's yelling *right* in front of the disks - his voice is going to induce a lot more vibration just because he's so much closer than the equipment. Inverse-square decay and all that ;-)
    • by Bengie (1121981) on Friday January 02, 2009 @10:28AM (#26299463)

      I'm not an engineer or absolutely sure about how the brain works with white noise, but I had a job that I worked at that when I entered the freezer section, it didn't seem loud at all. Actually, it so much didn't seem loud that the few times I had to enter it, I forgot my ear plugs until I saw someone else using them.

      Anyway, even though you couldn't really hear anything 'loud', if you tried to talk to anyway, you could barely hear them.

      On to my question. If you have enough high amplitude random noise that is effectively destructive interference, would this make an enviorment where low amplitude sound could not be hear or even mechanically sensed easily?

      I know using 'heard' may be incorrect in this context because perceived sound usually has no direct relation with what's mechanically going on with the sound waves.

      • by f00Dave (251755)
        On to my question. If you have enough high amplitude random noise that is effectively destructive interference, would this make an enviorment where low amplitude sound could not be hear or even mechanically sensed easily?

        The effect you've described is called sound masking [wikipedia.org] (more generally, auditory masking [wikipedia.org]), so yes, but the cause isn't quite what you inferred (destructive interference). :-)
  • Nice, now I can use this to detect if people were loud in my server-room.

    Do they also react to smell?

    • Re: (Score:1, Interesting)

      by Anonymous Coward

      Well, you can also use HDs as speakers:

      http://www.afrotechmods.com/cheap/hdspeakers/hdspeakers.htm

  • I heard... (Score:1, Offtopic)

    by fluch (126140)

    I heard that there is a place where they throw chairs at things...

  • you need to play classical music to your disks, and they'll perform even better.
    • Re: (Score:3, Interesting)

      The plants publications never seems to die.

      Plants don't react to music, they react to the tiny shifts in air just above their stomata. The publication which reported this compared plants with music (read: vibrating air above the stomata) with plants in an enclosure without air vibrating (read:refreshing) above the stomata.
      The experiment shows a difference, even if there's air-movement simply because air "sticks" to the surface of plant's leaves in close proximity - behaving like a fluid. Normal air ventila
  • Well, partly at least. It's no secret that disk drives are sensitive to vibration as this video showed an extreme case. Keep in mind, since disk drives are spinning at 7200-15000RPM, they themselves create vibration that can affect adjacent drives. The drive enclosure can help reduce the problem with use of shock absorbers and vibration dampeners. Most drive enclosures nowadays, for cost reasons, are no more than just sheet metal wrapped around power supplies, fans and drives, which contribute to the pr
  • by Anonymous Coward on Friday January 02, 2009 @06:59AM (#26298367)

    Secret Fact : Ultrasonic noise at low volumes is WORSE !

    It took weeks to testing to get to the root issue of WD Raptors dropping in head seeks on very high end raid cards in tiny head movement seek benchmarks, but padding each JBOD drive in acoustic foam (shooting range foam), or testing one drive at a time, instead of 4 or 8, (either method works) increased I/O per second by 40% in a rack chassis.

    40% more head movements per second if no ultrasonic noise entering drives !!!!!

    This is VERY VERY RARE INFO, and only I, the head of Gigabyte in Asia, and two engineers in california know of this discovery.

    And because I know no one on Slashdot will mod this up, and no one reads at 0 anymore, I can trust my astounding well researched secret shall remain secret.

    Its sadly 100% factual.

    • >And because I know no one on Slashdot will mod this up, and no one reads at 0 anymore, I can trust my astounding well researched secret shall remain secret.

      For starters, make an account.

      • Re: (Score:1, Insightful)

        by Anonymous Coward

        Accounts are overrated. What happened to judging the content rather than the person?

    • by thogard (43403) on Friday January 02, 2009 @09:37AM (#26299023) Homepage

      You don't know too many greybeards do you? I'm surprised that modern drives are susceptible to ultrasonic under 80 khz but real old drives and drums were known to have problems with low audible frequency harmonics. A simple solution to this problem is stamp a butterfly like pattern in the arm of the head. The same thing works for power lines (which is what the small dumbbell looking things are near the insulators)

      • by NeutronCowboy (896098) on Friday January 02, 2009 @01:51PM (#26301693)

        Informative? Seriously? I hope this is some metamod effort at providing Karma... but just in case someone does take this seriously, I should take out a patent on this. That way, when Monster sells their butterfly-patterned head arms for 20K to audiophiles who don't like the lack of warmth in SSDs, I can get in on the racket.

        • by thogard (43403)

          Sorry, there is prior art in the audio field too but feel free to send in cash to the patent office since they will rubber stamp it. I just want my cut or I can let them know about the prior art.

  • I think that this discovery is just the beginning of a new and harmonious spiritual relationship between man and machine. Just imagine, DC engineers across the world singing beautiful harmonies to their storage arrays!!

    Wonderful. ;-)

  • White noise is just like white light: An even distribution of energy across the spectrum.

    It makes perfect sense that white noise is less of a problem than an equal amount of noise at a specific frequency. Given a suitable frequency the material absorbing the energy will vibrate and even resonate. (That particular engineer's yelling, apparently, resonates well with the disks in the array of that video...)

    It's the same reason why a bullet-proof west can make the impact of a bullet non-lethal: Spread out distr

  • ... would increase my latency too.
  • Having seen this, I wonder what effect the 'anti-vibration' rubber grommets that are used on most modern desktop PC hard drive bays have on disk latency. After all, they stop vibrations being transmitted into the case my allowing the HDD itself to vibrate more and damping the movement as it reaches the case. Of course, having the HDD vibrate of its own accord is much better than having it resonate with another component in the case, so perhaps in some cases, the damping is beneficial to latency aswell.
  • Warbirds? (Score:1, Interesting)

    by t0ny747 (849486)
    I was thinking ( more like dreaming ) about buying a HD video camera that uses a hard drive. I wonder how much the noise of flying in a WW2 era Warbird would mess with it. I flew in a B-25 and it someone screwed with my camera's ability to focus.
  • by Gord (23773) on Friday January 02, 2009 @07:49AM (#26298571) Homepage

    Also from Brendan Gregg comes the always useful /usr/bin/maybe [brendangregg.com]. Other funnies from him here [brendangregg.com].

  • by mugurel (1424497) on Friday January 02, 2009 @08:23AM (#26298707)
    go and measure your own performance degradation while your hard disk does something mean to you
  • by BigMike (122378) on Friday January 02, 2009 @08:31AM (#26298739)

    I dub this guy the disk whisperer ...

  • I'm glad Sun have got enough free time and energy to find out useless nonsense like this.
    • Re: (Score:2, Insightful)

      by Bengie (1121981)

      because all server admins are busy 24/7?

      Server Admins are getting paid to 'watch' the servers. They have plenty of pseudo-free time. It's when stuff is breaking that they're busy. Not to mention a good admin in large server area will have software like that person had to watch drive latency.

  • That thing can shake my whole apartment. Is anyone testing what effect this thing has on my hard drives when listening to music/playing a game?

  • Seems like I have a lot to worry about, I have a couple of disk arrays inside a 40-ft trailer that tends to rock in the wind...

    The racks are shock-mounted, though.

  • you insensitive clod! Give them a bit to recoup!
  • I'm not being cynical in any way when I say I spent the last 1/2 hour checking out Fishworks due to the interesting analytics screen he used during the video. It occurred to me just now that's some (whether intentional or not) pretty effective viral marketing. LOL. The bad news is it appears Sun hasn't released Analytics as part of the base OpenSolaris distribution just yet. Too bad. I could just as easily use it to look at Veritas storage, assuming Veritas played nice with DTRACE. It'll be interesting

  • by thethibs (882667) on Friday January 02, 2009 @12:36PM (#26300617) Homepage

    Disk drives have a resonant frequency

    I've seen dramatic demonstrations of this over the years. One that stands out was a test of a Bryant drive sometime around 1970. In those days a 2 GB drive was at the edge of the envelope and Bryant was test-marketing just such a beast. It consisted of eight four-foot platters mounted four to a side on a shaft going through a monster of an electric motor. The heads were mounted on arms whose positioning was controlled by hydraulic cylinders big enough to be used as shocks on a pickup truck. The whole thing would not fit in the back of that pickup truck.

    We were testing the thing with a program called the "Leese Bomb". Leese can identify himself or remain anonymous--I won't turn him in. The "Bomb" part was the nature of the test.

    Basic tests in those days would involve writing a whole track and then reading it back and comparing what was read to what was written. You'd do this a number of times with different patterns to capture not only faults in the surface, but any sloppiness in the head control. The Leese bomb went one better.

    It would write to the outside track, write to the inside track, read the outside track, read the inside track, and then compare. If the comparison failed it would repeat the test, and keep repeating untl it succeeded, counting the failures. If the test succeeded it would index the test both inward and outward so that the tracks tested would move toward the middle, cross, and continue. This test was superior in that it would capture dynamic flaws in the system as the distance the heads moved, and the time to move varied from max to zero.

    In the case of the Bryant Drive (and, accidentally, an innocent Ramac drive at Caltech), the test found a resonant frequency. When the heads overshot their mark causing an error, the test stayed on the back and forth pattern, reinforcing the resonant motion with each cycle of the test. The drive started walking across the test floor in three-inch hops, but not for very long. In a few seconds, one of the shafts broke and one of the platters, a 500 pound disk rotating at 2400 rpm broke through the front of the unit and flew across the building until it was stopped, explosively, by one of the steel columns supporting the roof of the building. Miraculously, no one was hurt.

    We gave up on Bryant for that application. Not long after that, CDC introduced its 200MB drives, and they passed the Leese Bomb with flying colours. Ten of them didn't take up any more room, or cost more, than the big Bryant, so our client was happy to go with that solution.

    In any case the lesson is that, if it has moving parts, resonance is an issue.

    • by sshir (623215)
      That's so cool!

      If I remember correctly, Nikola Tesla invented an apparatus to wreck buildings from within by finding building's resonant frequency and pounding at it.

      Kinda like modern mass dampers, but active and, well, not damping much...
  • But is his name a killing word?

    /came for a Dune reference, left disappointed

TRANSACTION CANCELLED - FARECARD RETURNED

Working...