Forgot your password?
typodupeerror
Facebook Data Storage Open Source Hardware

Facebook's Corona: When Hadoop MapReduce Wasn't Enough 42

Posted by Soulskill
from the named-after-a-desire-to-launch-hadoop-into-the-sun dept.
Nerval's Lobster writes "Facebook's engineers face a considerable challenge when it comes to managing the tidal wave of data flowing through the company's infrastructure. Its data warehouse, which handles over half a petabyte of information each day, has expanded some 2500x in the past four years — and that growth isn't going to end anytime soon. Until early 2011, those engineers relied on a MapReduce implementation from Apache Hadoop as the foundation of Facebook's data infrastructure. Still, despite Hadoop MapReduce's ability to handle large datasets, Facebook's scheduling framework (in which a large number of task trackers that handle duties assigned by a job tracker) began to reach its limits. So Facebook's engineers went to the whiteboard and designed a new scheduling framework named Corona." Facebook is continuing development on Corona, but they've also open-sourced the version they currently use.
This discussion has been archived. No new comments can be posted.

Facebook's Corona: When Hadoop MapReduce Wasn't Enough

Comments Filter:
  • Facebook (Score:4, Interesting)

    by gman003 (1693318) on Friday November 09, 2012 @03:58PM (#41934991)

    I have to admit, while I hate using Facebook, and hate most of their business practices, I like how they're not just writing new infrastructure software, but are open-sourcing it all. I don't think it quite makes up for everything else, but it helps.

We are Microsoft. Unix is irrelevant. Openness is futile. Prepare to be assimilated.

Working...