Facebook Opens Their Data Center Infrastructure 90
gnu let us know about Facebook releasing specifications for their data center infrastructure as an open hardware project. They've released detailed electrical and mechanical data for everything from the server motherboards to the data center power distribution system. Digging further reveals that the specifications are licensed under the new Open Web Foundation Agreement which appears to be an actual open license. The breadth of data released really is quite amazing.
Re: (Score:1)
Re: (Score:1)
Not before you can get your hands on lots and lots of valuable personal information
Re: (Score:2)
Why for?
Re:Source Code (Score:5, Funny)
PHP?
Why for?
Indeed. FORTRAN IV was good for my father, it's good enough for me.
Re: (Score:3)
Building purely event-driven systems using the language is more and more appearing to be lipstick on a pig. I'm can't invest a tonne of energy like facebook into severely beating the language into some semblance of efficiency.
Ugh, I guess I should motivate myself to become more proficient in Python or Java.
Re: (Score:2)
Re: (Score:1)
Faceboook (Score:4, Interesting)
So open to our partners we'll even give them access to the servers themselves to poke around in your personal info directly.
On a serious note, the data center is pretty cool. Here's another source of pretty blue images that show better images regarding the evaporation cooling system.
http://www.technologyreview.com/computing/37295/?a=f [technologyreview.com]
You might have to 'skip' a couple HP ads but after about 2 or 3 they get the message that you're not interested.
Re: (Score:3)
A stark contrast to Google's nigh-on paranoid stance, the point of which I never really understood. It will be very interesting to see whether Facebook's open approach ultimately results in a lower infrastructure cost than Google's traditional secrecy. In this case my money is on open. Now how can I get some of that IPO? (Not a Facebook fanboy by any means, privacy issues are deeply disturbing.)
The cool thing about the Google/Facebook rivalry is, it's Linux vs Linux. I guess we'll be seeing more of that.
Re:Faceboook (Score:4, Funny)
The difference is that Google at least lubes up before performing the search...
Re: (Score:2)
While Web firms such as Google and Microsoft invest a lot in improving data-center efficiency, they keep their designs a closely guarded secret.
Actually Google made their data center designs public years ago (quite similar to this setup, but IIRC they were using containers for a modular data center and they were using plain air cooling w/ AC) but not as an open hardware project.
Re: (Score:2)
Actually Google tried to keep the container based design secret, only trouble was everybody else thought of the same (obvious) thing. See Sun's "black box".
Re: (Score:2)
Re: (Score:2)
I remember the cork-board server chassis and the fire hazard they presented. Facebook pulls way, way ahead of Google with these professional specifications. I'm especially fond of the omitted video processor, too.
Re: (Score:2)
Google has never made their *datacenter* designs, or even their locations public. They have shared their server design, or at least an outdated one.
From what I've heard from Ex-Googlers they never actually deployed the container concept beyond one half, of one of their many, many facilities.
Now if only... (Score:2)
They would stop pestering me about how I need to get back on Facebook and blather on about everything I've been doing lately.
i mean, that's what /. is for. ffs.
Re:Now if only... (Score:4, Funny)
Re: (Score:2)
by (1706743) and 4 others like this.
Actually... that's a good point. We can currently alter relationships with other users on here, but it'd be interesting if we could "like" posts -- and see how that rating compared to regular moderation. You could even go so far as to say "5 friends, 10 friends of friends, 30 foes and 500 friends of foes liked this." Take it even further, and add "If you liked this comment, you may also like..." and provide a widget that lists comments/submissions liked by friends and friends of friends.
Re: (Score:1)
Re: (Score:2)
the EVE Online forums recently added [eveonline.com] a "native" (i.e. not linked to Facebook) like button.
Did you see the ad for it? It made me choke on my beer. It was so much like OMG PONIES.
What a waste of time and space (Score:1)
Open source data centers? Can things really be that lame?
Where's the storage? Where's the fault tolerance? Where's the monitoring? Where's the fire suppression? Where are the diesel generators? Where's the physical security?
Where's....jeesh...there's so much more to a datacenter than just servers and racks.
Some yahoo just got the idea to apply "open source" to something random. Damned if it makes any sense at all, but hey, it's OPEN SOURCE, so it's got to be GREAT!
Meh. Idiots all.
Re:What a waste of time and space (Score:5, Insightful)
But didn't you just demonstrate the value by listing off the issues as you perceive them? Next step is discussion of your points to see if they are/are not addressed. Congratulations on your contribution to the open development process.
Re: (Score:1)
No. It's just dumb. Anyone with any smarts knows there are things such as blueprints, best practices, white papers, books, magazines, etc etc that cover this area in great detail. It's not like data center design is a big secret or has a lock-out to any interested parties.
Point is, this is just an ignorant application of a concept.
Re: (Score:2)
So you think that just because the subject has been written about extensively a data center will just assemble itself magically in an optimal configuration?
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
Replace "programmers" with "Microsoft" in that last sentance and you'd be closer. Seriously, I've seen OSS software take one -heck- of a lot more before breaking than Windows(Or, God forbid, some little piece of freeware/adware). OSS stuff generally works because when it breaks it gets /fixed/, and usually quickly too.
Volume Please.... (Score:1)
It's interesting, if they didn't didn't play the background music at twice the volume as the people talking.
Awesome (Score:2)
This is the sort of stuff everybody can benefit from. I wish more companies did this. And as an Oregon resident, this is doubly good for my state. BTW, what are large concrete security barriers doing there around the facebook data center? Is Facebook concerned about someone bombing them? Or do they serve a different purpose?
Re: (Score:2)
Oh, sometimes I crack myself up! In all seriousness though, props for releasing this information. (Also -- wanting to protect your data but not your privacy is not a real contradiction...just going for the cheap shot.)
Re:Awesome (Score:5, Funny)
Re: (Score:2)
Purely decorative. They are not "security barriers" they are just decorative concrete slabs.
Like (Score:1)
Slashdot Effect (Score:2)
Re: (Score:2)
No, but slashdot has been facebooked https://www.facebook.com/slashdot [facebook.com]
Using CentOS (Score:3)
Buried in the Intel Motheboard PDF on page 10 section 6.8 it says they're using CentOS 5.2 as the OS:
Also, in the chassis design it seems there are rubber passthrus to allow cables to go between servers above and below each other.
Re:Using CentOS (Score:4, Informative)
Facebook publicly says they use CentOS. They also have a large Mirror for lots of different Open Source tools: https://developers.facebook.com/opensource/ [facebook.com]
Kind of interesting (Score:2)
Re: (Score:1)
Commercial power (large buildings) is 3 phase 480 volt. (One leg of 3 phase 480 is 277 volts.) Normally motors (such as HVAC fans) are run directly from the 277/480 for highest efficiency and sometimes even the facility lighting will be fluorescent fixtures with 277 volt ballasts (again, highest possible efficiency.) There will be on premise transformers to knock the 480 down to 3 phase 208 volt. (One leg of 3 phase 208 is 120 volts.) This is used to power equipment (such as computers) that can not run
Re: (Score:2)
Re: (Score:2)
> Not sure how practical a PSU optimized for 277V input is for general use and the 450W max power is a bit tight for some Nehalem based configurations but overall it's pretty cool.
277V is perfectly fine. It gives about 400V DC, which maintains a decent safety margin to the absolute peak voltage of 600V for standard MOSFETs. And it should require only very minor changes from the standard 110-240V power supply.
450W are a decent amount, too, assuming you can actually load the PSU with 450W (and it does no
POE LED lighting (Score:1)
The lights are even powered by Power-Over-Ethernet. Slick. Anyone know who supplies these?
From http://opencompute.org/specs/Open_Compute_Project_Data_Center_v1.0.pdf :
4.11 LED Lighting Systems
Energy-efficient LED lighting is used throughout the data center interior.
Innovative power over Ethernet LED lighting system.
Each fixture has an occupancy sensor with local manual override.
Programmable alerts via flashing LEDs.
Re: (Score:2)
IIRC: http://www.redwoodsystems.com/products [redwoodsystems.com]
Re: (Score:2)
That's stupid. Putting your facility lighting into your data network?... There are many reasons software engineers aren't real, licensed engineers, and this is one of them.
Presumably the installation is up to code and includes backup lighting, as is required for commercial buildings. But keep up the good work!
Re: (Score:2)
I don't see how POE is inherently 'efficient' *if* it's a power-only connection. I can see it as convenience if you have gobs of ethernet ports and you don't want to run cable, but otherwise I'd think a more simplistic circuit would do the job as good or better.
Grammatical Error (Score:1)
Your title reads: "Facebook Opens Their Data Center Infrastructure 35"
Since the word Facebook is singular it requires signular verbs and singular possessive adjectives. You have have the singular verb but have a plural possessive adjective.
The title should read: "Facebook Opens ITS Data Center Infrastructure 35"
Switching infrastructure? (Score:1)
GNU? (Score:2)
Are they referring to the actual GNU organization or just some random /. user with username "gnu?"
Mostly meanignless.... (Score:3, Insightful)
Looking over the site, it's mostly warm fuzzies (look how green we are) and obvious (the system board specs are mostly bog standard reference designs). The chassis aren't particularly dense or make efficient use of the airflow, and no system vendor can ship implementations of this without running afoul of FCC regulations. There seems to be a lot of thought centered around a tech doing in-depth failure analysis of a failed board in person when even base boards come with IPMI implementations that allow all that to be done remotely. ROL is frankly a horribly dumb idea when you have IPMI capability in nearly every server board with acknowldgement and security. I know I'll get hit with people saying that IPMI costs extra, but the essentially free variants are sufficient to remove the RS232 connector and compete with 'ROL'. The free variants also tend to be flaky and sometimes need static arp tables, but so does WOL (in effect).
A rather direct dig at Google (Score:3)
Google perceives its datacenter know how as its major strength. This sort of removes a bit of that strength.
Go /. spin machine, go! (Score:1)
Re: (Score:2)
Easy, it reads less like a prescriptive howto and more of a blend between fluff about being green for the public and a requirements document for Tyan, SuperMicro, Asus, and any other board vendor that they might not have thought to explicitly include in their procurement process before. There isn't particularly much that is immediately actionable for datacenter builders.
Re: (Score:1)
Re: (Score:2)
I would have expected more in-depth techinical stuff (e.g. the expensive part of designing a system that facebook certainly outsourced) if it were a 'legitimate' open hardware project.
They may genuinely think they did something fancy though, I admit. Many customers don't go this in-depth on their requirements or mechanical designs, but they barely scratch the surface of the complexity of actually building any of the components. Of course, that's the case of most 'open hardware' involving complex things, y
The network? (Score:1)
So, nothing about the LAN / SAN?
Finally, no video system on a server (Score:5, Insightful)
Finally, it's so refreshing to see a server system specification that does not call for a video system, does not have onboard video, and properly directs console output to a serial port.
I've been disgusted with all the VGA crash carts, PS/2 keyboards and mice in server rooms, and all those video processors eating up system memory on servers. Servers should not have video.
Think of all the carbon dioxide and excess energy consumed by all the idle on-board video processors on most x86 and x64 servers out there. I shudder to think of all the planets resources being wasted displaying a graphical user interface that nobody will ever see, and, worse, reserving precious memory that should be used to serve users holding a useless frame buffer.
Have you ever smirked at a Linux server machine that is still running X and six virtual consoles? This news is really exciting that someone is honestly taking server hardware design seriously, just like Sun, HP, DEC, SGI, IBM, and others did in the 1980s and 1990s before all these x86 servers came about.
Bravo, Facebook, on a job well done.
Re:Finally, no video system on a server (Score:4, Interesting)
Server video cards embedded on motherboard don't use the system ram, they have an embedded 8 to 128 MB memory chip. Sure, they have a tiny frame buffer in the system ram but there are other things using more system memory than that frame buffer.
As for power usage, such plain vga video card embedded on the motherboard uses a couple of watts on idle - the chip doesn't even need a heatsink so it's not really a power saving feature if you remove it.
You would be saving much more power by using a power supply with high efficiency and wattage close to the actual server usage, instead of using (optionally redundant) 500-800 watts server power supplies.
Seriously, complaining about a few watts... some 1U servers have at least 4 x 40 mm high speed fans inside, each using 2-5 watts of power (because they run at max speed all the time) and you're complaining about a couple of watts on a video card.
Re: (Score:1)
They all take hardware design seriously, but if they put out a system without some sort of video, sadly, 98% of their customers won't buy it because they aren't confident in it.
MS users rarely ever know MS can be managed via serial and even those that do know there is a high chance some third-party software won't be manageable. Employers certainly know random tech off the street will need video on MS to get by.
Even amongst Linux users who likely would be using nothing more than a text interface, there are
Re: (Score:2)
Or Linux just defines its own method. Several BIOSes can redirect output to serial (ther
Re: (Score:2)
But if you want your server vendor to be replaceable, suddenly you have another system where the port you need is now ttyS1 when you had been using ttyS0. Or the BMC can only do 57600 wheras you have been doing 115200.
In short, yes there are several firmwares than all this can work. It would be a *lot* better if the board designers had a way of automatically describing the serial console capabilities to the kernel so that serial console would work after the kernel tears down frimware handling of IO.
Re: (Score:2)
Re: (Score:2)
I had not experienced that. As you know, a line break would go to the PROM monitor at which I just type "continue" or "go" and hit Enter a few times and we're all set. If I really somehow messed up the TTY, it's always recoverable, just turn off the TTY and turn it back on. Or maybe you just type CTRL-Q to release the paused output (which is likely what happened in your case).
In large installations I use a serial port concentrator. I sign into the concentrator and choose the system to log into. Alterna
What else can you do (Score:1)
SlugFest...a real bummer (Score:2)
Given a week? I could have at least made the case frame at least a pop-in-pop-out, no wires affair...
Give 1/10 of the time these loud mouths spent, I would have 4 up platters on bakery racks.
One easy hint! You are ordering at least 1000 motherboards, have the powersupply connector at a 90 degree angle so that it holds the powersupply connector to the side of the motherboard area, and holds the motherboard in. No screws. and for god sake... put som