Internet Archive Gets 4.5PB Data Center Upgrade 235
Lucas123 writes "The Internet Archive, the non-profit organization that scrapes the Web every two months in order to archive web page images, just cut the ribbon on a new 4.5 petabyte data center housed in a metal shipping container that sits outside. The data center supports the Wayback Machine, the Web site that offers the public a view of the 151 billion Web page images collected since 1997. The new data center houses 63 Sun Fire servers, each with 48 1TB hard drives running in parallel to support both the web crawling application and the 200,000 visitors to the site each day."
Where do they store 4.5TB off site (Score:5, Interesting)
Re:Where do they store 4.5TB off site (Score:4, Informative)
Re:Where do they store 4.5TB off site (Score:4, Insightful)
i find it impressive they have all that hardware for a mere 200k users a day.
Re: (Score:2, Informative)
In Brewster Kahle's December 2007 TED talk he mentions a third mirror in the Netherlands.
http://www.ted.com/index.php/talks/brewster_kahle_builds_a_free_digital_library.html [ted.com]
As he puts it, the Archive is mirrored on 'a fault line, a flood plain, and in the Middle East'.
Funny thing is I can't find another reference to the Netherlands mirror. The Bibliotheca Alexandrina site mentions a plan to eventually have four sites (California, Alexandria, Europe, and Asia), but that's it. Anyone know what happened with t
The off-site backup IS the Internet. (Score:5, Funny)
They're keeping the offsite backup distributed around the Internet, using the World-Wide Web to store it in real time.
Part of it may even be on *your* machine! We've really got to stop Brewster from leaching all your storage and make him store his backup himself - this business of using the originals to back up the backup just isn't sustainable!
Re:Where do they store 4.5TB off site (Score:5, Funny)
one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system
floppy disks.
lots of floppy disks.
Re: (Score:2, Funny)
Not reliable enough.
I suggest that this important resource be backed up to punched cards.
This would also enable handy comparisons in units that us oldies understand, such as ELOCs
(Equivalent Library of Congress).
I'd calculate it myself, but seem to have mislaid my slide rule...
Re: (Score:3, Funny)
I'd suggest also using stone slabs. Water can do serious damage to paper, and don't get me started on fire hazards. Good old Stone Slabs resist both of those really well. I'm not sure what the write speed is, however, so you'll probably need to hire many stonecutters to work in parallel.
Re: (Score:2)
When you get right down to it, any hard-coded data on silicon is just data on a stone slab. Since you can compile SystemC into a hardware spec, you can write stone slabs as fast as you can generate C.
Re:Where do they store 4.5TB off site (Score:5, Funny)
I'd suggest also using stone slabs. Water can do serious damage to paper, and don't get me started on fire hazards. Good old Stone Slabs resist both of those really well. I'm not sure what the write speed is, however, so you'll probably need to hire many stonecutters to work in parallel.
A math problem. My favorite. I don't know much about stone cutters but lets assume they can write one bit every 2 seconds. Thats 1 byte in 16 seconds. The internet archive is (4.5 x 1,125,899,906,842,624) 5,066,549,580,791,808 (5 quadrillion) bytes. That works out to 81,064,793,292,668,928 (81 quadrillion) seconds or about 2,570,547,732 (2.5 billion) years. That is far to long for their stringent 2 month backup cycle. They would need 15,423,286,395 (15.4 billion) stone cutters to keep schedule assuming they had unlimited stone. Last time I checked there were only between 6 and 7 billion people with only a small fraction of them being stone cutters. That leaves but one solution. Force the web developers to become stone cutters. This would not only increase the work force but also reduce the amount needed to backup because fewer people will be making more web pages to backup.
Re:Where do they store 4.5TB off site (Score:4, Interesting)
Re: (Score:2)
Re: (Score:3, Funny)
Re:Where do they store 4.5TB off site (Score:5, Funny)
Comment removed (Score:5, Interesting)
Re: (Score:2)
Re:Where do they store 4.5TB off site (Score:5, Funny)
Can you say, Parallelism?
Parallelogram.... crap
Parallellellell... dammit
Parapalouza... >
Why did you have to point that out to everyone? :(
Re: (Score:2)
Re:Where do they store 4.5TB off site (Score:5, Funny)
It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.
Re: (Score:2, Funny)
It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.
in music today, there is a so-called 'loudness war' and I think I've discovered what it is: they're removing the zeroes, thinking that 'all ones' will make the music even louder!
I wonder if its reversable? where do the zeroes go? can they be unzeroed? we should try to find them.
Re: (Score:2)
Followed by run length encoding of the remaining ones.
Re: (Score:3, Funny)
It's simple, the backups are compressed -- they simply remove all those useless zeroes from the binary data.
Compressed with XML! Because XML makes everything better... right?
Right?
Re: (Score:3, Funny)
"XML is like violence. If it doesn't solve your problem, you're not using enough of it."
Re: (Score:2)
PB, not TB... hehe.
Re: (Score:2)
Re:Where do they store 4.5TB off site (Score:5, Funny)
They'd better have it backed-up. Last time the Alexandria library burned-down, we lost about one thousands years of collected information from ancient Greece and Rome. Ooopsie.
Re: (Score:2, Insightful)
I can't take anyone seriously who puts "truth" and a link to Fox news in the same signature.
Re: (Score:3, Insightful)
>>>I can't take anyone seriously who puts "truth" and a link to Fox news in the same signature.
Neither can I take seriously anyone who believes MSNBC or CNN are unbiased and/or better alternatives. Or is prejudiced (prejudges a report without ever watching it). For example I may think Rachel Maddow is a joke, but at least I listen to what she has to say before I laugh. And sometimes, she says something worthy of hearing... it's good to keep an open mind and listen to the opposition.
Re:Where do they store 4.5TB off site (Score:5, Interesting)
They store 4.5PB in Egypt! (Score:5, Funny)
The Internet Archive also works with about 100 physical libraries around the world whose curators help guide deep Internet crawls. The Internet Archive's massive database is mirrored to the Bibliotheca Alexandrina, the new Library of Alexandria in Egypt, for disaster recovery purposes.
Re:They store 4.5PB in Egypt! (Score:4, Funny)
Egypt could be a good choice. The area is fairly famous for reliable persistent storage. From papyrus scrolls to stone engravings, things tend to keep there better than most places. There really aren't many other geographical areas on earth that can claim the same kind of data retention rates over the time periods they've dealt with. Though despite their impeccable track record with avoiding hardware failures, they've done significantly worse when it comes to data loss due to theft and/or hackers/pirates.
The one curious part about that choice is that the library at Alexandria is the one notable case where mass amounts of data were irreparably lost. So it's odd that they'd choose to entrust their data to that specific institution. Perhaps they felt that since it's under new management, the previous problems will have been resolved.
However, had the choice been mine, I would have chosen to store my offsite data in Luxor. It's data retention was quite good, and included one data store that was preserved in its entirety for over 3000 years. As an added benefit, it seems that they've opened a second location [luxor.com] that's significantly more convenient for the IA since there's no overseas transmission to worry about.
Re: (Score:2)
I oughta start a new company called:
Off-Planet backups where I'd use the moon to store your precious data!
Only three things I'd have to worry about would be:
1) Aliens (if they are out there)
2) Meteors
3) Solar Flairs
Other than that pretty solid plan to me!
Re: (Score:2)
Don't forget:
4) Spelling Nazis
You can ship it over OC-192... (Score:5, Interesting)
... one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot,..
As I recall from one of Brewster's talks: Part of the idea was that you can install redundant copies of this data center around the world and keep 'em synced.
You can ship 4.5 petabytes over a single OC-192 link in about 71 days.
Re:You can ship it over OC-192... (Score:5, Funny)
You can ship 4.5 petabytes over a single OC-192 link in about 71 days.
yeah, but just at the 70th day, someone will pick up the phone and the whole thing will have to be resent.
Re:You can ship it over OC-192... (Score:5, Insightful)
Or, you can ship the 40' containers in just under two weeks!
Re: (Score:2)
one would assume that something like this does regular off-site back-ups
there are BIG fat cables you connect, wait 3 seconds, then do a massively parallel 'dd if= ...'
Re: (Score:2, Insightful)
4.5TB isn't that bad. Heck, we have 1TB tapes right now. 5 of them can be carried in a small bag.
It's the 4.5PB that the Internet Archive could use that's hard to store offsite. 4500 1TB tapes can be pretty unruly.
Re: (Score:3, Interesting)
one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system
Create snapshot of zpool (think LVM VG):
# zfs snapshot mydata@2009-03-24
Send snapshot to remote site:
# zfs send mydata@2009-03-24 | ssh remote "zfs recv mydata@2009-03-24"
Create a new snapshot the next day:
# zfs snapshot mydata@2009-03-25
Send only the incremental changes between the two:
# zfs send -i mydata@2009-03-24 mydata@2009-03-25 | ssh remote "zfs recv mydata@2009-03-25"
Now this looks a lot like rsync, but the difference is that rsync has to traverse the file system tree (directories and files), whil
Re: (Score:2)
one would assume that something like this does regular off-site back-ups, which must add up to a hell of a-lot, could someone with experiance in such matters shed a little insight into the logistics of backing up such a vast system
Dude, the Internet Archive IS the offsite backup.
At least mine anyway. Tape drives be damned.
Story is meaningless without LOC measurement (Score:5, Funny)
Re:Story is meaningless without LOC measurement (Score:5, Informative)
4.5 petabytes = 4608 terabytes [google.com]
So, that's 460.8 LOCs.
Re:Story is meaningless without LOC measurement (Score:4, Funny)
Re:Story is meaningless without LOC measurement (Score:5, Interesting)
from http://www.lesk.com/mlesk/ksg97/ksg.html [lesk.com] The 20-terabyte size of the Library of Congress is widely quoted and as far as I know is derived by assuming that LC has 20 million books and each requires 1 MB. Of course, LC has much other stuff besides printed text, and this other stuff would take much more space.
1. Thirteen million photographs, even if compressed to a 1 MB JPG each, would be 13 terabytes.
2. The 4 million maps in the Geography Division might scan to 200 TB.
3. LC has over five hundred thousand movies; at 1 GB each they would be 500 terabytes (most are not full-length color features).
4. Bulkiest might be the 3.5 million sound recordings, which at one audio CD each, would be almost 2,000 TB.
This makes the total size of the Library perhaps about 3 petabytes (3,000 terabytes).
so 230 libraries by the old standard or 1.5 by the new standard
Re: (Score:2, Insightful)
from http://www.lesk.com/mlesk/ksg97/ksg.html [lesk.com] The 20-terabyte size of the Library of Congress is widely quoted and as far as I know is derived by assuming that LC has 20 million books and each requires 1 MB. Of course, LC has much other stuff besides printed text, and this other stuff would take much more space.
1. Thirteen million photographs, even if compressed to a 1 MB JPG each, would be 13 terabytes. 2. The 4 million maps in the Geography Division might scan to 200 TB. 3. LC has over five hundred thousand movies; at 1 GB each they would be 500 terabytes (most are not full-length color features). 4. Bulkiest might be the 3.5 million sound recordings, which at one audio CD each, would be almost 2,000 TB.
This makes the total size of the Library perhaps about 3 petabytes (3,000 terabytes).
so 230 libraries by the old standard or 1.5 by the new standard
Compress each audio file to a 5 MB MP3. That's 17.5 TB. Total size would be 750 terabytes.
So the data would be 6 LOC.
Re: (Score:3, Insightful)
You compressed the video, and the photographs, but not the audio? And why do you need a full CD for every sound recording? Surely many of them are far shorter than a full CD?
Re: (Score:3, Interesting)
The CDs are already in digital format, so compressing them is a cardinal sin.
The photos, movies, and maps are in analog format to start with, so we don't feel so bad using lossy compression. Image files are really big. I think the 1GB estimate per movie is pretty good, considering shorts, black and white, and the standard (or lower) definition of most of them. That would allow for a very high detail scan of the movie in something like MPEG4.
And, since they started in analog formats, there's no fair way t
Re:Story is meaningless without LOC measurement (Score:4, Informative)
83 terabyte in the LOC, so 4.5 petabytes == 54 Libraries of Congress
4.5 petabytes == 4500 terabyte hard drives, times $75 each == ~$340,000 == how much taxpayers spend, each hour, to maintain the LOC
Re: (Score:2)
P.S. My "83 terabyte" quote comes directly from the Library of Congress statistics, mid-2008.
Re: (Score:2)
Re: (Score:2)
a new 4.5 petabyte data center
4.5 PB? Is that the best you can do? sheesh, amateurs....
Though it also did surprise me they only get 200,000 hits/day. I expected the WayBack Machine [archive.org] to get a lot more traffic than that.
Re: (Score:2)
Re: (Score:2)
"According to alexa, archive.org is the 386th most visited site on the internet"
386? I would think that with all those Sun boxes they already were 64 bits at least!
Mental images of libraries (Score:2)
Riiiight... because you happen to have a really really good mental image of exactly how many rooms/shelves/books/pages are stored in the Library of Congress!
(Which incidentally doesn't happen to be static, BTW; yo momma's LoC ain't the same size as my LoC.)
Re: (Score:2)
Feynman estimated the LOC at about 1 petabit, which would make the Internet Archive containing roughly 36 petabits a cube on the order of 1/50 inch wide.
So it should fit in your pocket.
Storage Envy (Score:5, Funny)
Re: (Score:2)
Yes.
Own the internet! (Score:5, Funny)
so all one need to do to "own the internet" is to drive a big rig and ... lift the container off their parking lot?
Re:Own the internet! (Score:5, Funny)
well if you plug in a laser printer you can print off a hard copy for your boss.
Slight problem? (Score:5, Funny)
I can now theoretically steal "the internet" with a flatbed truck and a lift. There's something to be said for conventional data centers: They're rather hard to load onto a truck and drive off with.
Not "THE" but "A" internet... (Score:2)
You would be stealing A backup copy of THE Internet. An incomplete one at that, but still quite extensive.
Now... If you were somehow able to steal that copy AND break [youtube.com] the internet [youtube.com]... your stolen internet may be considered THE internet.
Re:Slight problem? (Score:5, Interesting)
Here's a video tour [youtube.com] of one if you need it for reference.
Don't forget to turn off the water and unplug the ethernet cables. Just be very careful with the power cords.
Re: (Score:3, Funny)
take THAT, Ted Stevens!
Re: (Score:2)
Nothing is stopping you from putting it inside a building, cementing it into its foundation, or surrounding it with appropriately weaponized sharks.
Re: (Score:2)
Re: (Score:2)
"or surrounding it with appropriately weaponized sharks."
or defending it with other ISO containerized systems:
Super Sangar
http://www.mod.uk/DefenceInternet/Templates/NewsArticle.aspx?NRMODE=Published&NRNODEGUID= [www.mod.uk]{8F432B04-D3C9-419E-8365-5204663E648C}&NRORIGINALURL=%2FDefenceInternet%2FDefenceNews%2FEquipmentAndLogistics%2FSecuritySurveillanceAndsuperSangars.htm&NRCACHEHINT=Guest
Re: (Score:2)
There's something to be said for conventional data centers: They're rather hard to load onto a truck and drive off with.
Yes, but imagine the bandwidth!
Re: (Score:2)
Never underestimate the bandwidth of a... stationwagon filled with 4.5 petabyte shipping containers?
Housed in a metal shipping container.... (Score:2)
Nice use for a bunny-rabbit (Score:2)
Yes, "thumper" refers to the rabbit. I have a Sun Managed Storage slide somewhere about how data tends to, er, multiply...
--dave
What about 1996 and earlier? (Score:5, Interesting)
Are there any resources the let us see websites from 1996, 95, 94, or 93? I would love to revisit the web as it appeared when I first discovered it (1994 at psu.edu).
Re:What about 1996 and earlier? (Score:5, Funny)
I would love to revisit the web as it appeared when I first discovered it (1994 at psu.edu).
No, you wouldn't.
Re: (Score:2, Funny)
The entire internet prior to 1996 is archived on an old PC that I'm currently trying to get the 5GB disk restored on.. why I've kept all that old porn for so long completely escapes me tho. :)
Re: (Score:2, Informative)
That is a shit ton of space (Score:2)
Re: (Score:2)
Fortunately, Hell has now been upgraded to 2 mb/s, thanks to British Telecom.
Re: (Score:2)
Re: (Score:2)
One method of suspend-to-disk is to do a freeze/thaw. It has taken Hell over 16 billion years to do just the freeze. Two millibits per second should be able to do both in less than half the time.
In Other News (Score:5, Informative)
Incidentally: FileFront [filefront.com] is closing in five days, taking with it any files that aren't hosted elsewhere.
I am told that many of the Half-Life mods [filefront.com] hosted there are not available anywhere else, so get while the getting is good...
Never underestimate the bandwidth ... (Score:5, Insightful)
... of a 4.5 petabyte datacenter in a shipping container in transit.
Re: (Score:2)
Government economic stimulus: Treating a patient for anemia with an iron supplement made from his own extracted blood.
I can't resist replying to your Sig...
It's like treating a patient for anemia with iron supplements made from his own extracted blood from the future. We are taking on debt, not trying to push through a one year ballenced budget. I'm not sure it's a good idea, but it's a much better one than what your describing.
Re: (Score:2)
It's like treating a patient for anemia with iron supplements made from his own extracted blood from the future.
Unfortunately, when you finance with debt on an economy-wide basis you pay double - or more. There's the return payment. (Plus the interest - which is the "more".) But there's also the cost to the economy of whatever WOULD have been done with the "borrowed" resources but now is not done because the resources were diverted.
When they talk of how many jobs were created by the stimulus, ask how man
63 x 48 = 3024Tb (Score:3, Insightful)
Re: (Score:2)
Re:63 x 48 = 3024Tb (Score:5, Informative)
TFA says "...eight racks filled with 63 Sun Fire x4500 servers with dual- or quad-core x86 processors running Solaris 10 with ZFS. Each Sun server is combined with an array of 48 1TB hard drives." (emphasis mine)
I would guess this means there's a x4500 with 24TB in local disks, and 48TB in attached storage per machine. (24+48)*63 does give us the quoted number
Re: (Score:2)
I don't think that's right. Sun's site has a video tour of it. Haven't finished it yet but it's here [sun.com].
Re: (Score:2)
The new datacenter is only 3PB. I guess the total storage, with the old data centers is 4.5 PB.
So 48x63 gives you 3PB of raw storage. I'm guessing there using less because I can't imagine them running it in raid 0.
Re: (Score:2)
Re: (Score:3, Informative)
Sun has more information and an Interactive tour [sun.com] of the Internet Archive modular data center on their site.
The total raw capacity of the container is 3 peta bytes. In reality it's going to be less than that. First, 2 disks are likely to be setup in a mirrored pool for the system disks. I believe the root pool only supports mirrors, not raidz. Not sure if this has changed.
That leaves you with 46 disks for data. Maybe they partitioned part of the root pool to include in the data pools, not sure, but zfs works
Math (Score:3, Informative)
"Sun Fire" (Score:4, Informative)
The new data center houses 63 Sun Fire servers
That's not very specific. "Sun Fire" is a brand that for a while got applied to all of Sun's rack-mount servers (except for NEBS-compliant servers, which were and are called "Sun Netra"). A little confusing, of course, which is why they've started calling new SPARC boxes "Sun SPARC Enterprise" to differentiate them from those mangy x64 "Sun Fire" systems. Except that there are still SPARC systems called "Sun Fire", so I guess the confusion factor didn't get any better...
Anyway, the specific server being used here is the Sun Firex X4500 [sun.com], a system with no less than 48 1 TB disks in a 4U space. Notice that this model is EOLed; presumably iarchive got a deal on some remaindered machines.
The shipping container is something we've seen before [slashdot.org].
Re: (Score:3, Informative)
Since they're using one of Sun's modular datacenters that is actually on the Sun campus, I would imagine that they got some financial incentives / support from Sun for all of this.
The X4500 is EOL as you mention, although it was still sold a few months back. It lives on as the X4540, which really isn't that different; the main thing is it's moved to a newer Opteron processor type and is a fair bit cheaper. So they didn't really miss out on anything.
It's kind of interesting to me that they went this route, a
Re:"Sun Fire" (Score:5, Interesting)
This seems to be an exact use case for the X4500-type system, which as far as I'm aware is pretty unique.
Indeed. Sun is on a density kick. Check out the X4600, which does for processing power what the X4500 did for storage.
In both cases, there actually are competing products that are sort of the same. The most conspicuous difference is that the Sun versions cram the whole caboodle into 4 rack units per system, about half the space required by their competitors.
More absurdly-dense Sun products:
http://www.sun.com/servers/x64/x4240/ [sun.com]
http://www.sun.com/servers/x64/x4140/ [sun.com]
The point of these systems is that they take up less expensive rack space than equivalent competitors. They're also "greener": if you broke all that storage and computing power down into less dense systems, you'd need a lot more electricity to run them and keep them cool. That not only saves money, it gives the owner the ability to claim they're working on the carbon footprint.
Re: (Score:2, Informative)
Anyway, the specific server being used here is the Sun Firex X4500 [sun.com], a system with no less than 48 1 TB disks in a 4U space. Notice that this model is EOLed; presumably iarchive got a deal on some remaindered machines.
There are newer X4540s which are mostly the same, but have newer CPUs, and can hold more memory (16 -> 64 GB).
they cut the ribbon? (Score:2)
Re: (Score:2)
Easy. Ribbon's only good for short-distance parallel links. If they've got backups in Egypt, they must be using serial cables.
Load distributions (Score:2)
From TFA (yeah, I know):
So they get all 200,000 hits in a 7-minute window? I picture a sysadmin going insane for a few moments then napping in a hammock for the rest of the day.
Re: (Score:2)
hit != visitor
Ok, how do you backup this thing ? (Score:2)
Re: (Score:2)