Behind the Scenes at Hotmail

mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."

High level of QC!

[Anonymous Coward]

From the article:
Hotmail relies on less than 100 system administrators to manage it all.

From the summary:
Between them, they process billions of emails per day and are overseen by hundreds of administrators.

Brought to you by the high quality control here at /.

Re:UNIX?

[Kraegar]

It used to be on FreeBSD w/Apache, now it runs on Windows w/IIS. It's not exchange based.

Read about it [microsoft.com]

Re:Does anyone know...

[jcaldwel]

[root@jboss html]# wget --save-headers -q -O- http://www.hotmail.com/ [hotmail.com] | grep "^Server:" 2>/dev/null Server: Microsoft-IIS/6.0

Re:UNIX?

[ThinkFr33ly]

While there were initial problems migrating to Windows, 100% of Hotmail now runs on Windows.

Also, Exchange was never involved in the migration. Hotmail is a combination of C++ ISAPI filters, COM+ (ATL) Enterprise Components, and SQL Server.

Coral Cache

[OctaneZ]

Looks like the site is down, it is however there is, however, a Coral Cache copy [nyud.net].

Re:Fairly Impressive

[mekkab]

Not only are the questions well picked

The interviewer is ACM Queue editorial baord member Ben Fried, who is the managing director of Morgan Stanley's worldwide IT deptartment.

Re:UNIX?

[Amoeba]

Yes. Hotmail was originally run on clusters of E3500 and E4500's running Solaris 2.5.1. After they got bought by Microsoft, a major initiative to migrate all boxes to Windows was undertaken in 2000. Hotmail has been 99.9% Windows for over 3 years now. The remaining 0.1% are some legacy solaris boxes used to handle backups for clusters... and even they are being phased out slowly.

--Amoeba (who no longer works there)

Re:Phil Smoot???

[Red Flayer]

Where do you think the unit of measurement came from? I visited my brother at MIT in '88 or '89, when the bridge (and the Smootlines) had been rebuilt... I thought it was the best thing about MIT (I learned differently later). on the origin of the Smoot. [lbl.gov]

Re:Spam improvment, but not perfect yet

[mccdyl001]

Yes it will. It will iterate through all your zip files. And if you password protect, gmail wont process it. So how do you send an exe or zip? Just give it some other arb extension (like bmp or dat or xxx or anything) and then attach it. On the other end give it back its proper extension, and you're done.

Re:Fairly Impressive

[xtracto]

Man, it is the Association for Computing Machinery magazine, I mean, it is not any PC-Weekly WalMart mag.

If you don't know about ACM publications, here [acm.org]are other interesting ones:

Ubiquity: IT opinion magazine and forum
TechNews: News Gathering Service for IT Professionals
eLearn: Distance learning magazine
MemberNet: Your Key to the World of ACM...and Beyond
Computers in Entertainment: New ACM online magazine

P.s. Sorry for the K.B.

Re:UNIX?

[shufler]

FreeBSD. Migrating Microsoft Hotmail from FreeBSD to Microsoft Windows 2000 Technical Case Study [microsoft.com]

Re:F**Kin Speak English !

[Tet]

The only possible advantage I can see in substituting the word "leverage" is that it sort of implies they are making the best use of these tools that they can

No, in fact it just makes no sense at all. The word "leverage" is a noun. The verb he was looking for is "lever", at which point it would at least have been grammatically correct. Of course, "use" would still have been a better option.

more Re:The SMOOT as unit of Length

[Alien54]

The Boston police have been known to use smoot markers to indicate accident locations on the bridge. Apparently Smoot's experience as a unit of measurement led to a life-long career; he eventually became Chairman of the Board of the American National Standards Institute, and later President of the International Organization for Standardization.

Re:Hundreds of administrators

[SpotTheCat]

From TFA:
What's interesting is that despite this enormous amount of traffic, Hotmail relies on less than 100 system administrators to manage it all.

Re:F**Kin Speak English !

[angelo]

Please don't "don't disagree" with people. It's like an abstract double negative. Instead use "I could agree with" or similar constructs.

Re:UNIX?

[enantiodromia]

This is only half true. The _front end_ runs on Windows with IIS. The _back end_, where the email data is stored (the User Stores), are Solaris. The front end machines dont mean much. If one or twenty go down, there are tons more to take their place. They are simply removed from the load balancing and marked as "admin plz fix this some day". The back end machines however, are super critical, as each user lives one one, and only one, user store. That machine goes down, and hundreds of thousands, to millions, of Hotmail users cant get to their mail. And thats why those machines run Solaris.

Re:Hundreds of administrators

[Anonymous Coward]

RTFA, not the /. summary. Hotmail runs on tens of administrators as well. The /. summary got it wrong.

Re:The article is fine...but

[MannyO]

You have to take into account less developed countries than the US.
I travel a lot to Mexico and it amazes me that *everyone* has a hotmail account there. They advertise it on fliers, on business cards, etc....
Some people will have (own) a domain like http://www.muchostacos.com.mx/ [muchostacos.com.mx] and *still* print their [e-mail] email.

It kills me....

I think this is because of the proliferation of internet cafes back when having internet (or a computer) at home was prohibitive.
All those machines with their homepages set to msn.com and nothing but windows messenger as the IM client...

When *I* worked there..

[EvilStein]

(just a little over a year ago) At a fairly new data center, Hotmail's backend still had a LOT of Sun Enterprise 4500 boxes running Solaris. None of them were being phased out at all. But, all of the boxes that were being brought online were HP/Compaq boxes running Windows.

Re:UNIX?

[Bing Tsher E]

And you know this, authoritatively?

The Hotmail service has changed considerably. Maybe the backend is still Solaris. But you didn't provide a cite.