Internet Archive Gets 4.5PB Data Center Upgrade 235
Lucas123 writes "The Internet Archive, the non-profit organization that scrapes the Web every two months in order to archive web page images, just cut the ribbon on a new 4.5 petabyte data center housed in a metal shipping container that sits outside. The data center supports the Wayback Machine, the Web site that offers the public a view of the 151 billion Web page images collected since 1997. The new data center houses 63 Sun Fire servers, each with 48 1TB hard drives running in parallel to support both the web crawling application and the 200,000 visitors to the site each day."
Re:Where do they store 4.5TB off site (Score:4, Informative)
Re:Story is meaningless without LOC measurement (Score:5, Informative)
4.5 petabytes = 4608 terabytes [google.com]
So, that's 460.8 LOCs.
Re:Story is meaningless without LOC measurement (Score:4, Informative)
83 terabyte in the LOC, so 4.5 petabytes == 54 Libraries of Congress
4.5 petabytes == 4500 terabyte hard drives, times $75 each == ~$340,000 == how much taxpayers spend, each hour, to maintain the LOC
In Other News (Score:5, Informative)
Incidentally: FileFront [filefront.com] is closing in five days, taking with it any files that aren't hosted elsewhere.
I am told that many of the Half-Life mods [filefront.com] hosted there are not available anywhere else, so get while the getting is good...
Re:What about 1996 and earlier? (Score:2, Informative)
Math (Score:3, Informative)
"Sun Fire" (Score:4, Informative)
The new data center houses 63 Sun Fire servers
That's not very specific. "Sun Fire" is a brand that for a while got applied to all of Sun's rack-mount servers (except for NEBS-compliant servers, which were and are called "Sun Netra"). A little confusing, of course, which is why they've started calling new SPARC boxes "Sun SPARC Enterprise" to differentiate them from those mangy x64 "Sun Fire" systems. Except that there are still SPARC systems called "Sun Fire", so I guess the confusion factor didn't get any better...
Anyway, the specific server being used here is the Sun Firex X4500 [sun.com], a system with no less than 48 1 TB disks in a 4U space. Notice that this model is EOLed; presumably iarchive got a deal on some remaindered machines.
The shipping container is something we've seen before [slashdot.org].
Re:What about 1996 and earlier? (Score:1, Informative)
Re:63 x 48 = 3024Tb (Score:5, Informative)
TFA says "...eight racks filled with 63 Sun Fire x4500 servers with dual- or quad-core x86 processors running Solaris 10 with ZFS. Each Sun server is combined with an array of 48 1TB hard drives." (emphasis mine)
I would guess this means there's a x4500 with 24TB in local disks, and 48TB in attached storage per machine. (24+48)*63 does give us the quoted number
Re:Where do they store 4.5TB off site (Score:2, Informative)
In Brewster Kahle's December 2007 TED talk he mentions a third mirror in the Netherlands.
http://www.ted.com/index.php/talks/brewster_kahle_builds_a_free_digital_library.html [ted.com]
As he puts it, the Archive is mirrored on 'a fault line, a flood plain, and in the Middle East'.
Funny thing is I can't find another reference to the Netherlands mirror. The Bibliotheca Alexandrina site mentions a plan to eventually have four sites (California, Alexandria, Europe, and Asia), but that's it. Anyone know what happened with the Netherlands site?
Re:"Sun Fire" (Score:3, Informative)
Since they're using one of Sun's modular datacenters that is actually on the Sun campus, I would imagine that they got some financial incentives / support from Sun for all of this.
The X4500 is EOL as you mention, although it was still sold a few months back. It lives on as the X4540, which really isn't that different; the main thing is it's moved to a newer Opteron processor type and is a fair bit cheaper. So they didn't really miss out on anything.
It's kind of interesting to me that they went this route, as opposed to a bunch of servers talking to a bunch of storage separately. This seems to be an exact use case for the X4500-type system, which as far as I'm aware is pretty unique.
Re:63 x 48 = 3024Tb (Score:3, Informative)
Sun has more information and an Interactive tour [sun.com] of the Internet Archive modular data center on their site.
The total raw capacity of the container is 3 peta bytes. In reality it's going to be less than that. First, 2 disks are likely to be setup in a mirrored pool for the system disks. I believe the root pool only supports mirrors, not raidz. Not sure if this has changed.
That leaves you with 46 disks for data. Maybe they partitioned part of the root pool to include in the data pools, not sure, but zfs works better with whole disks.
In the interactive tour, they weren't clear on how they set up the pools.
Side note. Maybe I'm cynical, but if this was the other way around, with linux servers replacing sun/solaris servers that probably would have been the headline.
Pretty neat to find out that the internet archive is powered by Java too. The wayback machine is java as well as the crawlers.
Re:"Sun Fire" (Score:2, Informative)
Anyway, the specific server being used here is the Sun Firex X4500 [sun.com], a system with no less than 48 1 TB disks in a 4U space. Notice that this model is EOLed; presumably iarchive got a deal on some remaindered machines.
There are newer X4540s which are mostly the same, but have newer CPUs, and can hold more memory (16 -> 64 GB).
Re:Where do they store 4.5TB off site (Score:2, Informative)
Their aim is to preserve the content found on the Web. They need the hardware for that. I assume they don't need much for the "serving users" part.