Behind the Scenes at Hotmail 292
mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."
KISS my hotmail body (Score:4, Funny)
Don't try to tell me that the guys at Hotmail only want to Rock & Roll all night and party every day?!?
Re:KISS my hotmail body (Score:5, Funny)
Re:KISS my hotmail body (Score:2)
I think it's more "Psycho Circus"...
I wonder.... (Score:2, Funny)
Re:I wonder.... (Score:2, Funny)
Re:I wonder.... (Score:5, Funny)
Ghostbusters?
Re:I wonder.... (Score:2, Funny)
The French?
The SMOOT as unit of Length (Score:2)
This apparently appeared in "People Weekly", April 24, 1989, v. 31, p. 93+
Harvard Bridge spans the Charles River linking Boston and Cambridge. In 1958 Lambda Chi Alpha took 5' 7" MIT freshman pledge Oliver R. Smoot, Jr. and rolled him head over heels the entire length of the bridge. Every ten smoots they calibrated the bridge, painting marks. The bridge was found to be exactly 364.4 smo
more Re:The SMOOT as unit of Length (Score:2, Informative)
UNIX? (Score:5, Interesting)
Re:UNIX? (Score:2)
If I recall correctly, wasn't Hotmail originally run on UNIX boxes?
Yes, it was run on a combination of BSD & Solaris boxes, IIRC.
Re:UNIX? (Score:5, Informative)
Re:UNIX? (Score:3, Interesting)
Re:UNIX? (Score:2)
Re:UNIX? (Score:2, Interesting)
Re:UNIX? (Score:3, Informative)
Also, Exchange was never involved in the migration. Hotmail is a combination of C++ ISAPI filters, COM+ (ATL) Enterprise Components, and SQL Server.
Re:UNIX? (Score:2, Troll)
No wonder hotmail sucks. Then again I am a diehard gmail fan
Re:UNIX? (Score:3, Interesting)
The new Windows Live Mail beta is fairly good. Doesn't have the feature set of Gmail or Yahoo yet, but it's getting there.
If it wasn't for the near impossibility of migrating 20,000+ e-mails from Hotmail to Gmail, I probably would have jumped ship long ago... but Live Beta is keeping me interested.
Paul Graham on the importance of tools (Score:4, Insightful)
He submits, of course, that any program can be written in any reasonable language -- for they all are, after all Turing machine's equivalents. But the quality of the tools can make a difference between a feature being added next week and not at all.
If Hotmail's admins are back to command line and scripting anyway, maybe, they should've stuck with FreeBSD.
Look at how quickly Google is rolling new things out -- their platform allows them to.
Re:UNIX? (Score:2)
F**Kin Speak English ! (Score:5, Insightful)
Re:F**Kin Speak English ! (Score:2, Funny)
Re:F**Kin Speak English ! (Score:3, Informative)
No, in fact it just makes no sense at all. The word "leverage" is a noun. The verb he was looking for is "lever", at which point it would at least have been grammatically correct. Of course, "use" would still have been a better option.
Re:F**Kin Speak English ! (Score:2)
The only dictionary that matters [oed.com] lists leverage as a noun. Any other usage is incorrect.
Re:F**Kin Speak English ! (Score:2)
Re:F**Kin Speak English ! (Score:2, Informative)
Re:F**Kin Speak English ! (Score:2)
I don't disagree does not mean I could agree. It doesn't mean I do agree. I could possibly mean I have no opinion on the issue. It does however mean that I don't disagree.
"I don't disagree" is about as incorrect as "I didn't kill that man"
Re:F**Kin Speak English ! (Score:2, Funny)
Re:F**Kin Speak English ! (Score:5, Funny)
Because his team leverages best-of-breed systems to utilize the synergistic effects of the paradigm shift in relationships among stakeholders and the knowledge infrastructure, silly.
Re:F**Kin Speak English ! (Score:2)
I'm sure a time will come, if it hasn't already, when all those perfectly normal words that are replaced by multi-syllabic, often ungrammatical, usually awkward and always trendy buzzwords by the marketing types and other suits will simply cause the simple, clearer, and more precise words to simply fall out of use.
How's that for a new p
Re:UNIX? (Score:5, Informative)
Read about it [microsoft.com]
Re:UNIX? (Score:5, Informative)
Re:UNIX? (Score:5, Informative)
--Amoeba (who no longer works there)
Re:UNIX? (Score:4, Interesting)
When *I* worked there.. (Score:3, Informative)
Re:UNIX? (Score:2, Funny)
Re:UNIX? (Score:2)
Better subject... (Score:5, Funny)
And how does the NSA process all that email? Now THAT would be an interesting technical challenge!
Re:Better subject... (Score:2)
They use Microsoft MailSpy (version 1.0) with the Decrypt plugin.
Slick gui (although I hear it uses non standard widgets)
with the massive processing power of Microsoft Speed.
No need to be patched though as it has a very small userbase and isn't a virus vector target as such (like Linux/unix/OSX)
(removes tongue from cheek)
Re:Better subject... (Score:3, Interesting)
Has anyone ever considered that spam may actually help keep us all 'freer'? There's billions of spam messages everyday that add to the legitimate traffic. If all spam email magically disappeared, all that would be left is 'legitimate' correspondence.
Which would make the NSA's new job of spying on us much easier.
I used to know a guy who always went to the limit on doing his taxes - exploited every loophole, deducted everything that could even vaguely be dedu
Does anyone know... (Score:5, Funny)
Re:Does anyone know... (Score:2, Informative)
Re:Does anyone know... (Score:3, Insightful)
If it is responding in the headers IIS, it's probably being proxied by some kind of load balancer. In a modern setup, the proxy is a hardware device with a custom OS... probably originating in BSD, but the IP stack heavily modified. The system for delivery and transport of mail will also be differnt than that of the web interface.
I don't think an OS really matters anymore when you're getting to that scale. The architecture matters, and that's probably proprietary and protected by IP agreements with emp
Which cmd line? (Score:2)
SFU doesn't use bash. (Score:2)
Re:SFU doesn't use bash. (Score:2)
And not too well ;) [ornl.gov]
Re:SFU doesn't use bash. (Score:2)
Re:Which cmd line? (Score:2)
Fairly Impressive (Score:5, Interesting)
Not only are the questions well picked but the some of the answers are quite interesting. For instance Phil on scalability: Before reading this article, I always had hotmail pegged as a hacked together e-mail system less organized than a monkey sh*tfight but if Phil speaks the truth, I've underestimated them. They're a hacked togethor server mess with a lot of effort put into staying afloat--and they have been doing well for a long time.
I guess I've always taken my free Hotmail account for granted.
Re:Fairly Impressive (Score:3, Informative)
The interviewer is ACM Queue editorial baord member Ben Fried, who is the managing director of Morgan Stanley's worldwide IT deptartment.
Re:Fairly Impressive (Score:2, Informative)
If you don't know about ACM publications, here [acm.org]are other interesting ones:
Ubiquity: IT opinion magazine and forum
TechNews: News Gathering Service for IT Professionals
eLearn: Distance learning magazine
MemberNet: Your Key to the World of ACM...and Beyond
Computers in Entertainment: New ACM online magazine
P.s. Sorry for the K.B.
And yet.... (Score:3)
1.) Logging in. You would think that since I already typed hotmail.com in the address bar, I wouldn't have to type "@hotmail.com" in the log-in form, but alas, the solution has aluded them. In fact, it seems to have escaped them altogether, since it used to be that way. Apparently having seperate hotmail.com and msnmail.com, storing a cookie, or even just having a radio button is bey
Spam improvment, but not perfect yet (Score:3, Interesting)
Re:Spam improvment, but not perfect yet (Score:2)
I was getting a LOT of spam for a while, turned on most everything that could be turned on, I flag it all as junk...but I still get 20ish a day. Really, I just skim it fast to see if I have anything from people I know, then I go back to gmail or such (where I only get 1 or 2 spam emails a day).
If 20ish a day is their version of stable, I'd prefer it stabilize a little bit...lower.
Re:Spam improvment, but not perfect yet (Score:2)
"The probability that this file contains a virus is extremely high. Please confirm using direct communication with the sender, that the file is not generated by a virus."
That way, geeks can send attachments, and ordinary users might not run the
And preferably, when the virus scanner detects identically sized
Re:Spam improvment, but not perfect yet (Score:2)
Re:Spam improvment, but not perfect yet (Score:2, Informative)
Re:Spam improvment, but not perfect yet (Score:2)
Re:Spam improvment, but not perfect yet (Score:2)
High level of QC! (Score:5, Informative)
Hotmail relies on less than 100 system administrators to manage it all.
From the summary:
Between them, they process billions of emails per day and are overseen by hundreds of administrators.
Brought to you by the high quality control here at
Re:High level of QC! (Score:2)
The SPAM problem (Score:2, Insightful)
BF Can you quantify in some way the extent of the spam problem?
PS It is massive. Years ago we saw as many as 3 billion incoming messages. This has declined, but the estimates are that 75 percent of all e-mail is spam. Over the past couple of years our techniques have gotten better, and our partnerships with other major ISPs have improved. I would say spam is still gross and abusive, but it hasn't been getting worse lately.
We do continue to react to spam on a daily basis as spammers continue to seek
Re:The SPAM problem (Score:2)
They have installed spamfilters, but only on the input. Every Nigerian can create a hotmail account and start spamming, and their filters don't bother to act.
Receivers of those mails can complain at abuse@hotmail.com, but it will take two weeks for them to process the complaint and lock the account, at which time the spammer just opens a new one.
Is it really that difficult to scan outgoing mail, rate-limit the ma
Command line (Score:3, Insightful)
> To do that they have returned to the command line.
Absolutely.
I'm currently in the process of trying to change our company culture away from legacy GUI tools and toward command-line tools.
Scriptability is a highly under-rated goal. I'm not against GUI tools -- but they need to be built on top of scriptable utilities.
Interacting without any sort of user interface (Score:5, Funny)
I always thought that the command line was a user interface. You know, interfacing between a user and a computer.
It's hard to picture using a computer without any sort of user interface. I'm pretty sure that, in order to call it "using" a computer, some sort of interface must exist, be it keyboard mouse and monitor, binary switch, light gun, real gun, neural link, telekinesis, or whatever. Otherwise, you're not using it, are you?
On the other hand, maybe the article is correct- a lot of operations group probably don't want to use "any sort of user interface" to communicate with their computers. They want to be sitting on a beach in tahiti drinking daiquiris, thousands of miles away from the computers they're supposed to maintain.
Re:Interacting without any sort of user interface (Score:2)
Re:Interacting without any sort of user interface (Score:3, Funny)
No, this is Hotmail. They do not need any user interface. They managed to configure the servers so that they send each other billions of SPAM emails each day. Totally automatically. Then they deleted all user interfaces. That is also why the spam levels have sta
The article is fine...but (Score:2, Interesting)
Is this true? I thought Google might be the Everest. Anyway, speaking from personal experience, in my university every student has multiple yahoo/gmail accounts but just a handful use Hotmail. Can someone throw light on the actual number of users all over?
Re:The article is fine...but (Score:2, Informative)
I travel a lot to Mexico and it amazes me that *everyone* has a hotmail account there. They advertise it on fliers, on business cards, etc....
Some people will have (own) a domain like http://www.muchostacos.com.mx/ [muchostacos.com.mx] and *still* print their muchostacos@hotmail.com email.
It kills me....
I think this is because of the proliferation of internet cafes back when having internet (or a computer) at home was prohibitive.
All those machines with
From the immortal words of Henry Spencer (Score:5, Insightful)
"Those who don't understand UNIX are doomed to reinvent it, poorly."
From the article and elaborating on the
Q: Are there scaling reasons to think about the benefits of a command line for managing over a GUI, or are there other things to think about?
A: Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line. That's the only way you're going to be able to execute scripts and gather the results over thousands of machines.
Also, we all remember the scaling issues that MS had when they took over hotmail and initially tried to switch from freebsd to Windows.
MS had to port over cron jobs because its not something that is installed and used by default under windows like UNIX. They had to rewrite the "inefficient" perl code that ran fine on FreeBSD to C++. They had to redo the memory allocation to prevent memory leaks in the new C++ code. Read about it from the goat's mouth http://www.microsoft.com/technet/interopmigration
I can't wait until FreeBSD and other inferior OSes get tools to find memory leaks. One day....
(That last line was sarcasm and not a flame).
Re:From the immortal words of Henry Spencer (Score:2)
(That last line was sarcasm and not a flame).
Can't it be both?
Coral Cache (Score:4, Informative)
They have a special way of dealing with spam (Score:2)
So long and thanks for all the spam
So sad that it should come to sham
Could anyone suggest a better rhyme for spam?
Re:They have a special way of dealing with spam (Score:2)
Bam!
Clam?
Damn!
Ham?
Jam.
Ma'am
RAM
Sam
Yams?
Re:They have a special way of dealing with spam (Score:3, Funny)
There once was a young man name Sam,
Who spent his whole day sending spam.
But at night he went drinking,
And this led him to thinking,
That maybe he would end up being damned!
Windows (Score:2, Interesting)
Hundreds of administrators (Score:5, Interesting)
Heh. I used to work at Akamai [akamai.com] which provides content delivery services for many of the biggest sites on the web. They have somewhere over 15,000 servers that are managed by tens of administrators, not hundreds. In fact, a typical NOCC (yes, 2 'C's for Akamai) shift at Akamai is only staffed by 8 or so people, with only a couple of senior level admins on call. And they're delivering all sorts of web-based content, including streaming, not just e-mail.
But then Akamai runs them all on linux, whereas I belive Hotmail is all Windows based. You do the math.
Re:Hundreds of administrators (Score:2)
MSN Search has admin/server ratios similar to Akamai's.
Re:Hundreds of administrators (Score:2)
Does anyone know what mail server hotmail uses for smtp and imap or pop or whatever? I'm curious what scales up that well regardless of platform. If there's actually a decent
Re:Hundreds of administrators (Score:5, Insightful)
It sounds to me like you don't understand what it is that Akamai does. They're not just running web & streaming servers on their 15k machines. They're distributing content in real time in a way thtat vastly improves user access all around the world. You may have heard when Victorias Secret held their first video-streaming lingerie show. Well their servers couldn't handle the load because of all the people trying to watch it. They became an Akamai customer, and Akamai was able to redistribute their streams in real-time all over the globe. To be able to take video (or just web content) from a single source and distribute it quickly and efficiently to thousands of distributed users in real-time is a huge undertaking. Akamai has some very impressive technology to be able to do this.
I'm not saying that running a mail service like Hotmail is a piece of cake, but I do think that what Akamai does is a lot more difficult and impressive when you think about it. If Akamai's distributed environment were to drop off the net then you probably wouldn't be able to access any of the on-line services of most of their customers [akamai.com]. (And that's just a small subset of their customer base) The ability to keep websites like those of Microsoft, eBay, Fed Ex, Red Hat, etc. all highly responsive to end users is not a simple feat by any stretch of the imagination.
Re:Hundreds of administrators (Score:2)
Not at all. In fact, ads typically aren't served directly through Akamais caching technology, AFAIK. Most of the content that gets cached on Akamai servers are things like images (which virtually every major commercial site is chock full of these days), data files (Windows Update), etc. Typically ad serving and web caching don't work well together since the advertisers want different ads displayed each time you visit a particular site, not the exact same ad over and over and over
Re:Hundreds of administrators (Score:3, Informative)
What's interesting is that despite this enormous amount of traffic, Hotmail relies on less than 100 system administrators to manage it all.
Re:Hundreds of administrators (Score:2, Informative)
Re:NOCC? With 2 C's? (Score:2)
Hotmail was next to my cage at (Score:5, Funny)
"they process billions of emails per day" (Score:2)
Re:"they process billions of emails per day" (Score:2)
Spammers?
Hotmail Stinks (Score:3, Interesting)
Hotmail can no longer be used with Outlook because Microsoft admits it's easier to block everyone and charge for access rather than cancel the accounts of spammers. [infoworld.com]
This is why Google is on top now (Score:4, Interesting)
Compared to Google clusters [internetnews.com], they seem to be light years behind. As a software developer, I can tell you that the key to rolling out applications quickly, is to have a decent framework in place. Whatever that framework might be (from shell scripts to java monstrosities), once its in place, developing apps on top of it are easy. Similarly a well thought out app execution environment is golden.
If you ever check out Google's MapReduce [google.com], you'll see what I mean. It's just so well thought out and so elegant, that its easy to believe that they can scale outwards forever. You'd not be too far off if you thought that Microsoft were rethinking their whole production environment to compete with Google.
There's no way that Microsoft can quickly and easily roll out vast new applications that scale, because that whole clustering framework is completely opposite to what Windows provides.
For more information on Smoot... (Score:2)
Hotmail is so irrelevant (Score:4, Interesting)
It was a great service! One of the first, and probably the best.
Microsoft took it over and there was no advancement or innovation for years (a decade?). Spam ate up my tiny inbox while Microsoft just threw MSN graphics all over the place.
When Gmail came out, I gave it a try. It was everything Hotmail could have been years ago if it hadn't been bought by MS! (Well, it COULD have been out of business, so I've got to give them that I suppose).
They forced Microsoft to pay a little attention to features. They gave out a little more storage and started blocking some spam, but it was too little too late.
In order to write this I decided to visit my hotmail inbox, I haven't been there for a while. 136 emails, and 43 have been detected as junk. They are ALL junk--A party invite from "heather", a Cola Quiz, etc. 136 undetected junk emails out of 179.
And even at that, they still only give 1/8 the amount of storage that Google does.
Crap, on top of that I just looked at a spam with pictures in it and it didn't auto-block them like Google does. Now I'm probably infected.
Thanks Microsoft!
From,
The guy who used to argue the advantages of Microsoft to the Unix admins...
Re:Hotmail is so irrelevant (Score:4, Interesting)
Gmail didn't appear until much later on, but Yahoo were creating some fantastic portal features.
I have a Gamil, Yahoo & Hotmail accounts, but prefer to give out my Hotmail account for "free offers" and other junk, its a junkbox.
However Gmail & Yahoo are both solid email solutions, and as you say, Gmail fairs better than all of them in the spam war.
Gmail; From a geek perspective, I admire them for creating key mappings that mimick those of vi/vim.
There is features present in Yahoo I'd love to see in Gmail:
* Setup up one-time (or temporary) email addresses that are binded to your email address.
* A decent calendar that can sync to iCal and Sunbird. (I don't think Yahoo have this yet)
* Events management, setup birthday reminders and the like.
* A virtual notepad that you can scribble down notes
* Sharing your calender, its private by default.
* Check number of new messages without logging in or providing credentials (uses a cookie)
Yahoo is awesome, if you havent tried out their web portal, take a look. Its very impressive.
Oh, come on yourself. (Score:4, Insightful)
Right... it's always more interesting to read article after article about only unsuccessful operations run by people who aren't proud of what they do, and don't face huge, global challenges.
You're cranky because it's MS. If exactly the same article ran, substituting "gmail" and "google" for all of the other names, you'd say, "cool!"
Re:stabilised... (Score:5, Funny)
Re:Phil Smoot??? (Score:3, Informative)
Re:Phil Smoot??? (Score:2)
BTW. I thought the old SCC library was one of the best things (open 24/363). That and the Chille at 20 Chimneys (the old one).
Re:Phil Smoot??? (Score:2)
Re:Hotmail censors gmail (Score:2)
I'm not missing the point (Score:2)
The only exception to this is where local law requires us to do this as in some EU countries.
But WTF do I know, I get modded for trolling and I've only been in IT for 25 years
Re:speaking of hotmail and UIs (Score:2)