Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

Multiple Sites Down In SF Power Outage

Posted by kdawson on Tue Jul 24, 2007 05:12 PM
from the or-was-it-a-drunk dept.
Power IT
corewtfux writes with word of a major outage apparently centered on 365 Main, a datacenter on the edge of San Francisco's Financial District. Valleywag initially claimed that a drunken person had gotten in and damaged 40 racks, but an update from Technorati's Dave Sifry says the problem is a widespread power outage. Sites affected include Technorati, Netflix (these display nice "We're Dead" pages), Typepad, LiveJournal, Sun.com, and Craigslist (these just time out).
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

Multiple Sites Down In SF Power Outage 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.

  • I work in the Financial District (Score:5, Interesting)

    by slug_bait (118345) * on Tuesday July 24 2007, @05:18PM (#19976327)
    I can verify that it affected much of the Financial District here in SF. We had the power go out 3 times. Seems to be back now. Haven't heard any explanation yet.

  • Oblig.... (Score:5, Funny)

    by Anonymous Coward on Tuesday July 24 2007, @05:18PM (#19976329)
    im in ur datacentr
    trashin ur racks

  • Redundant? (Score:5, Insightful)

    by DogDude (805747) on Tuesday July 24 2007, @05:19PM (#19976335) Homepage
    Don't these large sites have failover capable, redundant servers in multiple physical locations? Why should a failure in one rack, one room, or heck, even one state for the giant sites, effect them?

    • Re:Redundant? (Score:5, Informative)

      by Anonymous Coward on Tuesday July 24 2007, @06:17PM (#19977053)
      They do, but one of the dirty little secrets of most data centers is that they don't have enough generator capacity for all the cooling. They'll woo you with the generator, the 2,000 gallons of diesel, and N+1 array of UPSes, but when utility power dies, it gets hot very quickly. And some racks must go down.

    • Re:Redundant? (Score:5, Interesting)

      by ryanisflyboy (202507) on Tuesday July 24 2007, @06:32PM (#19977247) Homepage Journal
      For some of these sites they are a lot more central than you might realize. If they failed to build their systems with a secondary site in mind it can be near impossible for the "CTO" types to pony up the dollars for it later. The biggest issue I have seen that affects this is storage. Either they aren't using suitable SAN technologies, or they didn't put enough money behind the storage initiative to set up secondary site replication. I agree with you though. This is a problem that has been solved. Perhaps netflix thought - wth - if we go out for a few hours and people can choose their movies that's just tough luck.

      Sun.com going down is a good example of someone totally screwing up. They have absolutely NO excuse. The others - maybe they can get away with it and we won't care. If Sun can't keep their own site up, how can I expect them to keep mine up?
      • Because those who bought colo services were in fact ripped off, and should now be proceeding to San Francisco to seek veangance upon those who can do little more than process credit card payments.

        Perhaps they could begin their vengeful wrath by hiring a few (more?) winos...

        • Re:Redundant? (Score:4, Insightful)

          by raehl (609729) <raehl311&yahoo,com> on Tuesday July 24 2007, @06:51PM (#19977469) Homepage
          I'm certainly forwarding this article to my boss, who abruptly decided to put an end to planning for a backup site on the basis of "aw, nothing is going to happen".

          The thing is, letting something happen may be a better decision than trying to stop it.

          If you're going to have a fully-redundant setup, it's going to cost you twice as much as having just one setup. And if you're not going to have a fully-redundant setup, your backup site is going to buckle under the full load of normal traffic anyway.

          The correct business decision might just be "I just saved a bunch of money on my data center insurance," and if you lose a day's business, oh well, that was still cheaper than keeping a backup data center around.

            • Re:Redundant? (Score:5, Informative)

              by Jeffrey Baker (6191) on Tuesday July 24 2007, @07:12PM (#19977689)
              365 Main has a long and ignominious history of frequent and prolonged power outages, yet it remains fully booked. Some people just can't learn a lesson.

              For what it's worth, the datacenter which is adjacent to 365 Main, called 360 Spear, did not suffer from this outage.

  • Other sites.. (Score:3, Informative)

    by king-manic (409855) on Tuesday July 24 2007, @05:19PM (#19976337)
    Gamefaqs/Gamespot is also down. I wonder if it's related.

    • Re: (Score:3, Informative)

      by nuzak (959558)
      Gamefaqs/Gamespot is C|Net, located on Rincon Hill in downtown SF, and their servers are probably in 365main. So yeah.

      Anyway, PG&E says it's over now, but they still don't have an explanation as to why. Shyeah (rolls eyes)

  • Redundent power supply? (Score:3, Interesting)

    by msimm (580077) on Tuesday July 24 2007, @05:19PM (#19976349) Homepage
    Does this mean backup generators have failed or is the fault somewhere outside the datacenter? Time to start shopping.

      • Re:Redundent power supply? (Score:4, Informative)

        by dextromulous (627459) on Tuesday July 24 2007, @05:33PM (#19976535) Homepage
        You mean that all 3 x 20,000 gallon tanks [365main.net] were empty? I find that hard to believe.

        • Re: (Score:3, Informative)

          by grumling (94709)
          I really doubt they were ever full. Diesel fuel goes bad after a few months. Unless SF has really, really crappy power*, the generators don't do much more than idle once a week for 20 minutes or so. The giant tanks are only there for the marketing department. And maybe for the employees to top off their tanks.

          *I live out in the middle of nowhere and I get a power failure exceeding 5 minutes about once per year. The longest I've had at my current location was just over 2 hours.

          • Re:Redundent power supply? (Score:5, Interesting)

            by aaarrrgggh (9205) on Tuesday July 24 2007, @06:00PM (#19976871)
            It takes Diesel a few years to go bad. That site has fuel polishing systems to prevent that. Because of earthquake risk, they contractually are obliged to have 24-48 hours of backup fuel with many of their clients.

            They have the HiTec rotary UPSs in all their facilities, which link a generator to a flywheel UPS. It's stupid to not have backup fuel for that type of system; you can only run for 13 seconds before the load crashes.

            It is possible that they got a number of small hits and the generators failed to re-start after a few. Good procedures are to stay on generator until utility stabilizes if you have more than one "hit."

            Be interesting to find out what happened.

  • how many data centers? (Score:4, Interesting)

    by riceboy50 (631755) on Tuesday July 24 2007, @05:23PM (#19976393)
    It's interesting that so many major sites would go down in a local power outage? Are they all sharing one data center in SF? If so, why don't they have co-locations in other cities?

  • The Scoop from SFGate.com (Score:3, Informative)

    by fromtheblueline (717915) on Tuesday July 24 2007, @05:28PM (#19976447)
    At least 20,000 without power in downtown S.F. Marisa Lagos and Demian Bulwa, Chronicle Staff Writers Tuesday, July 24, 2007 (07-24) 15:12 PDT SAN FRANCISCO -- At least 20,000 customers of Pacific Gas and Electric Co. in downtown San Francisco lost power this afternoon, the utility said. Brian Swanson, a spokesman for the utility, said outages have been reported throughout downtown and along the Embarcadero, including at PG&E's office on Beale Street near the Ferry Building. It was unclear initially how many customers who lost power remained without it for a sustained period. Power outages were also reported in the South of Market neighborhood, the Outer Mission and down the 3rd Street corridor south of Mission Bay. PG&E officials said they did not know why power had gone out, but most customers appeared to be back online by 3 p.m. The outage has prompted Muni to run shuttles in the place of cable cars, a spokeswoman said. The T-Third Metro line was unable to cross the 4th Street Bridge for a short time, but power was restored to the drawbridge by 3 p.m. Muni bus lines 14, 49, 30, 41 and 45 were without power for about 30 minutes following the outage, but are now working, spokeswoman Maggie Lynch said. Parking Control officers were deployed to the Outer Mission, 3rd Street and Monterey Avenue for traffic control, she added. Power first went offline around 1:50 p.m. and came back at least three times in the downtown area before shutting off again. The same problems were reported in South of Market all the way to AT&T Park and the Caltrain station at Fourth and King streets, and traffic lights were out as far south as Monterey Boulevard. At the Westfield Center at Market and Fifth streets, only one of six Nordstrom elevators was working while the shopping mall ran on a backup generator. Shoppers milled around as the lights flickered on and off. BART is still running trains but the lights at its downtown stations have flickered on and off several times, said spokesman Linton Johnson. The transit agency also has concerns about the ventilation system, which is on the same grid as the lights, he said, but will keep its downtown stations open so long as the lights and ventilation continue to work. Workers at several downtown and South of Market offices were reportedly sent home for the day following the outage. Additionally, the datacenter 365 Main -- which hosts Web sites including Craigslist and Yelp -- lost power.

    • Re:The Scoop from SFGate.com (Score:5, Funny)

      by RealGrouchy (943109) on Tuesday July 24 2007, @06:25PM (#19977169)
      It was hard to read through that block of text, but looking closely, it explains why:

      "Officials say the power outage may affect some websites, including the site that hosts Slashdot.org's preview button."

      It all seems to be back up now.

      - RG>
  • We are working with our co-location facility managers to assess why it is back-up power generators failed to provide the necessary back-up power to prevent our site going down. We apologize for any inconvenience caused by our site being unavailable this afternoon.

    I think that's admin speak for:

    I warned these idiots eight months ago during my review that the datacenter had outgrown its generator capacity. But did they listen? Fuck no, they just kept counting money and worrying about the bottom line. The beancounters looked at me like I'd asked them for a blowjob from their grandmothers when I submitted the workup for additional generator capacity. And now that the shit's hit the fan, whose ass are they screaming for? Screw this, I'm applying at Taco Bell.

    • Re: (Score:3, Insightful)

      by Cervantes (612861)
      Where's the +1 "100% fucking right" mod option?

      Whaddya bet some poor mid-level admin gets blamed and tossed for this? And the upper-management guy who ignored the recommendations for testing or redundancy still gets his bonus for good fiscal performance.

    • Re:From Technocrati: (Score:5, Funny)

      by RealGrouchy (943109) on Tuesday July 24 2007, @06:28PM (#19977203)

      "... to assess why it is back-up power generators failed ..."
      I've been a grammar nazi for many years, but it looks like the enemy has unleashed new weapons.

      Tell my family I loved them.

      - RG>

  • LOLcurrent (Score:3, Funny)

    by carou (88501) on Tuesday July 24 2007, @05:51PM (#19976745) Homepage Journal
    I is not in ur datacenter, 2 power ur servers.

  • Kiss of Death? (Score:5, Funny)

    by Honig the Apothecary (515163) on Tuesday July 24 2007, @05:53PM (#19976773)
    Press Release on Red Envelope having 2 years of uptime at 365 Main - San Francisco from today: http://365main.com/press_releases/pr_7_24_07_red_e nvelope.html [365main.com]

    • UPS system - it's a Hytec flywheel/diesel combo (Score:4, Interesting)

      by Animats (122034) on Tuesday July 24 2007, @06:16PM (#19977049) Homepage

      Data sheet for 365 Main [365main.com]:

      The company's San Francisco facility includes two complete back-up systems for electrical power to protect against a power loss. In the unlikely event of a cut to a primary power feed, the state-of-the-art electrical system instantly switches to live back-up generators, avoiding costly downtime for tenants and keeping the data center continuously running.

      They use a Hytec Continuous Power System [pageprocessor.nl], which is a motor, generator, flywheel, clutch, and Diesel engine all on the same shaft. They don't use batteries.

      With this type of equipment, if for some reason you lose power and the generator doesn't start before the flywheel runs down, you're dead. There's no way to start the thing without external power. Unless you buy the optional Black Start feature [pageprocessor.nl], which has an extra battery pack for starting the Diesel. "Usually the black start facility will not be often needed but it won't hurt to consider installing one. Just imagine if you were unable to start up your UPS system because the mains supply is not available.". Did 365 Main buy that option?

        • Re:UPS system - it's a Hytec flywheel/diesel combo (Score:5, Interesting)

          by Animats (122034) on Tuesday July 24 2007, @08:18PM (#19978257) Homepage
          The classic Bell System policy on emergency generators, in the electromechanical switching era, was as follows:
          • Generators are started once a week.
          • Once a month, generators are started and run for an hour.
          • Once a year, generators are started and the entire facility run without external power for 24 hours.

          And this was in addition to the 48VDC battery backup.

          In the entire history of electromechanical switching in the Bell System, no central office was ever down for more than 30 minutes for any reason other than a natural disaster. That record has not been maintained in the computer era.

          If you have to build reliable systems, it's worth understanding electromechanical telephone switching. Because the components weren't that reliable, the systems had to be engineered so that the system as a whole was far more reliable than the components. Read up on Number Five Crossbar. [wikipedia.org] The Wikipedia article isn't really enough to understand the architecture, but other references are available.

          • Re:UPS system - it's a Hytec flywheel/diesel combo (Score:4, Informative)

            by Anonymous Coward on Wednesday July 25 2007, @04:00AM (#19980725)
            I was a hardware engineer about 10 years back on the battery backup systems. We were developing new technology to try and stretch the life of the batteries. We worked together with some of the top minds in battery technology in the US.

            The battery systems that are installed in the "Bomb cages" as we called because the larger ones were often underground and appeared similar to a 3 person bomb-shelter where quite impressive. Typically, they were two full banks of twenty four, 2 Volt, 375 AMP batteries. Each of them physically twice the size of a truck battery. They were most often lead-acid mammoths at the time since lead acid was reliable for a measurable period of time and inexpensive in comparison to the lithium-ion variety in the same capacity.

            The batteries were always rated at 10 years life from the manufacturer, but the telephone companies had tested in real-world environments and would rotate the cells out at 4 year intervals instead since down-time on the network to replace power systems was far more expensive then being prepared instead. After all, each one of these cabinets would typically handle as many as 15,000 telephone lines and would often contain fibre repeaters for higher speed lines connecting the boxes all together and then to the central.

            The biggest problem with these installments was that a single battery in a shipment would show signs of early fatigue, most typically visible from the appearance of bubbling in the plastic walls, then it was policy to replace the entire batch of cells immediately, not just the single battery displaying fatigue. This was because it was clear that if a single battery in the group showed fatigue then all the cells in the bank would probably be susceptable to the same issue. It could be something as simple as a manufacturing screw up or it could be due to a cooling system problem in the box, or any of a lot of other environmentally related issues.

            It's really quite impressive the cost and efforts the telephone company would go through just to maintain and prevent issues with the UPS system which thankfully, rarely ever gets exercised in places where people are intelligent enough not to live on fault lines or high risk hurricane paths.

            The greatest flaw in the design of the batteries systems was that they were always trickle-charged. The chargers were unintelligent and simply kept the batteries topped off. This caused "memory issues" as we're all familiar with, especially thanks to notebook batteries.

            What we learned about the cells where I was engineering was that, if a cell could physically survive as long as 7 years without environmentally related damage (bubbles), then it should be possible to detect early stages of design related fatigue within a single cell.

            We also found that if a weekly or monthly power cycle of a bank of cells were to be performed, the batteries would last substantially longer than the 4 years expectancy. So, in the case of Bomb Cages where at least two full banks of cells were available (that's pretty much a minimum configuration), on a proper schedule, using a huge-ass resistor bank, we would fully drain a bank of cells until we could detect nearly 0 current across the resistor. Then we would perform a full charge on the cells again, monitoring each cell more than 10 times per second. Batteries that failed to charge in sync with the other cells were typically early replacement candidates.

            Well, all that being said, one thing I'm 100% confident of is that data centers lack the experience and the interest to budget this kind of research for their systems. The telephone companies are amazingly well prepared in comparison.

            On a side note, just last week, I installed my first 48V DC powered RAID rack. I designed a high efficiency hard drive case that contained no fans. Each case was 1U and shallow enough to install two back-to-back in a rack. We installed 96 units in a single rack with 4 drives each and no-air conditioning in the room. The design was extremely simple.

            1) Use Telco
  • This has got to be some type of joke: RedEnvelope Reports Two Years of Continuous Uptime at 365 Main's San Francisco's Datacenter [365main.com].

    It was released today....

  • About Emergency Power (Score:5, Informative)

    by linuxwrangler (582055) on Tuesday July 24 2007, @06:13PM (#19977021)
    It's been a long time since I went on a tour of several data centers to locate a new facility for our dot-com. I believe that 365 Main was a facility that does not use a battery UPS. Instead, they have engine-backed flywheel UPS system (see http://www.enterprisenetworksandservers.com/monthl y/art.php?2813 [enterprise...ervers.com] for a description). At the time, they have 10 2-megawatt generators on the roof in a N+2 configuration. The engines are kept heated and are spec'd to go from stop to engage-clutch/deliver-power in 3 seconds. The flywheel can deliver 11 seconds of power so they can fail through a couple of bad engines before running out of flywheel power. They periodidally do a 20-hour load test into a pair of 500,000 watt heat-sinks. Time will tell if this outage was a failure of design, failure of maintenance, or outright malfeasance. But it wasn't supposed to happen. They've got some 'splainin' to do.

    As to diesel storage, use of diesel is widespread for emergency use everywhere from hospitals to emergency-services to hospitals. Those systems are run regularly - typically weekly. The use of biocides, stabilizers, and mobile fuel-scrubbing services, and extra filtration systems can maintain the fuel quality. Our colo currently maintains a 1-week fuel-supply and has multiple quick-refuel contracts in place. I can't imagine any colo having less than 24-48 hours in-the-tank with quick-refill on-call.

    But one thing that is missing is cooling. Our colo has a typical contract that says something like blah-blah won't exceed 80F for more than 4 hours blah blah. OK, but a rack full of blade servers can crank out 15-20kW of heat load and a data center can heat up real quick without AC. By contract, 150F for 3.5 hours would be in-spec.

  • The word directly from 365 (Score:4, Informative)

    by Meridian Umbrios (1132753) on Wednesday July 25 2007, @03:46AM (#19980655)
    Here is the e-mail that 365 is sending out to their customers. The best is their tagline "the world's finest datacenters'.

    365 Main Customer,

    At 1:49 p.m. on Tuesday, July 24, 365 Main's San Francisco data center was effected by a power surge caused when a PG&E transformer failed in a manhole under 560 Mission St.

    An initial investigation has revealed that certain 365 Main back-up generators did not start when the initial power surge hit the building. On-site facility engineers responded and manually started effected generators allowing stable power to be restored at approximately 2:34 p.m. across the entire facility.

    As a result of the incident, continuous power was interrupted for up to 45 mins for certain customers. We're certain colo rooms 1, 3 and 4 were directly affected, though other colocation rooms are still being investigated. We are currently working with Hitec, Valley Power Systems, Cupertino Electric and PG&E to further investigate the incident and determine the root cause.

    All generators will continue to operate on diesel until the root cause of the event has been identified and corrected. Generators are currently fueled with over 4 days of fuel and additional fuel has already been ordered.

    We understand the seriousness of this issue and will provide full details once they come available. We sincerely apologize for the impact this has had on your operations.

    Regards,
    Vice President, Security
    365 Main
    "The World's Finest Data Centers"

    Just send me a big fat check and all is forgiven.

    • Re:No Generators? (Score:5, Informative)

      by grumling (94709) on Tuesday July 24 2007, @05:31PM (#19976493) Homepage
      Well, you test and test and test, and when something finally happens, nothing. Stuff happens.

      Brownouts sometimes fail to trigger generators, even though they should. If only one phase goes down, depending on the design, it may not trip (and would cause a somewhat random outage, like some drunk shutting down racks).

      If the generator runs on diesel, they usually only plan for a few hours of backup. If they didn't recalculate the generator runtime as they added equipment, the load may have caused the fuel consumption to go up higher than anticipated. Is it hot in SF today? Air handlers may be straining to keep the place cool, or maybe the generator got running too hot.

      Often times, as equipment is added, the load gets out of balance between phases. It is usually a good idea to keep the load as even as possible, but in a high traffic data center, I would imagine there would be a lot of stuff moving in and out, expanding and contracting, and it may become hard to keep track of the loads across phases. A good facilities manager should be able to tell you the current load off the top of his head, but too often these details get left out.

      This is just stuff I've seen in cable TV headends over the years. Granted, this facility should have a power manager/engineer on staff, but so often the power is one of the first things to get cut from the budget.

      • Re:No Generators? (Score:5, Interesting)

        by MichaelSmith (789609) on Tuesday July 24 2007, @06:14PM (#19977037) Homepage Journal

        Stuff happens

        No kidding. years ago in my former job on traffic systems we had a great UPS with a generator on site and the ability keep it fueled up indefinitely. A security contractor came in on the weekend to install something and tried to wire up a new circuit hot. He slipped with a screwdriver and shorted the white phase to the chasis of the breaker panel. I don't think the tip of the driver actually touched ground, but the burn mark is still there to show how close he got.

        The resuting current spike blew the 100A fuses (heavy metal strips) both going in to and out of the UPS. With the UPS effectively broken the generator set failed to start and the system gracefully shut down 40 minutes after the incident. Thats not bad. The batteries were only specified to work long enough for the genny to settle at 50Hz.

        In the process of blowing the fuses a spike got back into the power supply of one of our DEC Alphas and took out the power supply. The system was redundant at the software level so I didn't notice immediately.

        The UPS guy came out and didn't have enough fuses to replace the blown one, but we found that with a bit of brute force and filing attacks some others could be made to fit.

        Please type the word in this image: problems

    • zombies .... (Score:5, Funny)

      by taniwha (70410) on Tuesday July 24 2007, @05:31PM (#19976507) Homepage Journal
      There's a report here [sfgate.com] that "Flesh-eating zombies are prowling the streets"

      • Re:No Generators? (Score:5, Insightful)

        by eln (21727) * on Tuesday July 24 2007, @05:31PM (#19976503) Homepage
        Any data center that advertises high availability should be testing that sort of thing on a regular basis. It's possible that they could fail switchover even if they are being regularly tested, but it is unlikely.

        If the "power outage" theory is correct and the "drunken employee" theory is incorrect, as a customer I'd be pissed that the data center I pay tons of money to can't keep my site up in the event of a power outage, which is one of the main perks of hosting at a data center in the first place.

      • Re:No Generators? (Score:5, Informative)

        by eln (21727) * on Tuesday July 24 2007, @05:39PM (#19976603) Homepage
        This is a DATA CENTER, its whole purpose in life is to be available when things like this happen. It had better have generators and plenty of fuel on hand at all times. The data center I work at has the capability to run at full power with nothing coming in from the outside world for 36 hours. I don't know what the standard is for other data centers, but it seems like they should be capable of getting at least 12 hours of operation without incoming power from the grid.

      • Insane level of backup... (Score:5, Interesting)

        by SmoothTom (455688) <Tomas@TiJiL.org> on Tuesday July 24 2007, @07:03PM (#19977561) Homepage
        ...until the commercial power fails and doesn't come back for days.

        The only places I've actually seen the insane levels of backup that some would like is in some telco central offices. The one I was associated with the longest had eight-hour-plus battery backup and 8 days of fuel for the diesels. Some of our really remote microwave sites had 24 hour battery and 30 day diesel.

        Of course one of those sites failed high up in a mountain range in a mid-winter storm (Tieton, 1978) when the commercial power failed, and the starter battery for the diesel froze. When one of the techs finally got there (after burying his Sno-Cat and walking the last couple miles), he had to chip ice off the steel door to get inside, where he was able to get the diesel started with a little "rewire" of one of the backup battery sets. Oh, his two-way radio also failed during his hike, since it was outside his snowsuit, and the lack of communication caused the company to start two more Sno-Cats and a helicopter in that direction.

        The site was out for nearly six hours, IIRC.

        Even the BEST designs are subject to failure. :o(

        --
        Tomas

        • Re:Insane level of backup... (Score:5, Funny)

          by Technician (215283) on Tuesday July 24 2007, @09:55PM (#19978967)
          Of course one of those sites failed high up in a mountain range in a mid-winter storm (Tieton, 1978) when the commercial power failed, and the starter battery for the diesel froze.

          On Black Bute in Oregon, a communications site went out in the middle of winter following a power outage. The generator ran a short while and shut down because it overheated. The air intake 20 feet in the air was covered in snow.

    • Re:LiveJournal?? (Score:5, Funny)

      by eln (21727) * on Tuesday July 24 2007, @05:36PM (#19976573) Homepage
      Impossible, they would never commit suicide without posting a note in the form of bad angst-filled poetry to their blog first. There is no chance any of them will actually kill themselves until the site is back online.

    • I can hear it now, the sound of a million emos all finally committing suicide.
      Nah, they wouldn't commit suicide if they couldn't blog about it afterwards...

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.