Behind the Scenes at Hotmail 292
mallumax writes "ACM Queue interviews Hotmail engineer Phil Smoot on how they manage more than 10,000 servers spread around the globe. Between them, they process billions of emails per day and are overseen by hundreds of administrators. To do that they have returned to the command line. From the article: 'Our operations group never wants to rely on any sort of user interface. Everything has to be scriptable and run from some sort of command line'. The overriding philosophy seems to be KISS. Also: tape backups are out and spam levels have stabilized."
UNIX? (Score:5, Interesting)
Fairly Impressive (Score:5, Interesting)
Not only are the questions well picked but the some of the answers are quite interesting. For instance Phil on scalability: Before reading this article, I always had hotmail pegged as a hacked together e-mail system less organized than a monkey sh*tfight but if Phil speaks the truth, I've underestimated them. They're a hacked togethor server mess with a lot of effort put into staying afloat--and they have been doing well for a long time.
I guess I've always taken my free Hotmail account for granted.
Spam improvment, but not perfect yet (Score:3, Interesting)
Re:UNIX? (Score:2, Interesting)
The article is fine...but (Score:2, Interesting)
Is this true? I thought Google might be the Everest. Anyway, speaking from personal experience, in my university every student has multiple yahoo/gmail accounts but just a handful use Hotmail. Can someone throw light on the actual number of users all over?
Spam spam spam! Argh. (Score:1, Interesting)
Re:UNIX? (Score:3, Interesting)
The new Windows Live Mail beta is fairly good. Doesn't have the feature set of Gmail or Yahoo yet, but it's getting there.
If it wasn't for the near impossibility of migrating 20,000+ e-mails from Hotmail to Gmail, I probably would have jumped ship long ago... but Live Beta is keeping me interested.
Re:UNIX? (Score:4, Interesting)
Windows (Score:2, Interesting)
Hundreds of administrators (Score:5, Interesting)
Heh. I used to work at Akamai [akamai.com] which provides content delivery services for many of the biggest sites on the web. They have somewhere over 15,000 servers that are managed by tens of administrators, not hundreds. In fact, a typical NOCC (yes, 2 'C's for Akamai) shift at Akamai is only staffed by 8 or so people, with only a couple of senior level admins on call. And they're delivering all sorts of web-based content, including streaming, not just e-mail.
But then Akamai runs them all on linux, whereas I belive Hotmail is all Windows based. You do the math.
Hotmail Stinks (Score:3, Interesting)
Hotmail can no longer be used with Outlook because Microsoft admits it's easier to block everyone and charge for access rather than cancel the accounts of spammers. [infoworld.com]
This is why Google is on top now (Score:4, Interesting)
Compared to Google clusters [internetnews.com], they seem to be light years behind. As a software developer, I can tell you that the key to rolling out applications quickly, is to have a decent framework in place. Whatever that framework might be (from shell scripts to java monstrosities), once its in place, developing apps on top of it are easy. Similarly a well thought out app execution environment is golden.
If you ever check out Google's MapReduce [google.com], you'll see what I mean. It's just so well thought out and so elegant, that its easy to believe that they can scale outwards forever. You'd not be too far off if you thought that Microsoft were rethinking their whole production environment to compete with Google.
There's no way that Microsoft can quickly and easily roll out vast new applications that scale, because that whole clustering framework is completely opposite to what Windows provides.
Hotmail is so irrelevant (Score:4, Interesting)
It was a great service! One of the first, and probably the best.
Microsoft took it over and there was no advancement or innovation for years (a decade?). Spam ate up my tiny inbox while Microsoft just threw MSN graphics all over the place.
When Gmail came out, I gave it a try. It was everything Hotmail could have been years ago if it hadn't been bought by MS! (Well, it COULD have been out of business, so I've got to give them that I suppose).
They forced Microsoft to pay a little attention to features. They gave out a little more storage and started blocking some spam, but it was too little too late.
In order to write this I decided to visit my hotmail inbox, I haven't been there for a while. 136 emails, and 43 have been detected as junk. They are ALL junk--A party invite from "heather", a Cola Quiz, etc. 136 undetected junk emails out of 179.
And even at that, they still only give 1/8 the amount of storage that Google does.
Crap, on top of that I just looked at a spam with pictures in it and it didn't auto-block them like Google does. Now I'm probably infected.
Thanks Microsoft!
From,
The guy who used to argue the advantages of Microsoft to the Unix admins...
Re:Better subject... (Score:3, Interesting)
Has anyone ever considered that spam may actually help keep us all 'freer'? There's billions of spam messages everyday that add to the legitimate traffic. If all spam email magically disappeared, all that would be left is 'legitimate' correspondence.
Which would make the NSA's new job of spying on us much easier.
I used to know a guy who always went to the limit on doing his taxes - exploited every loophole, deducted everything that could even vaguely be deductible, said he gave more to charity than he actually did. He mailed his forms in on April 13th. Said that he figured it was right in the middle of the heaviest flow - kind of like pissing into the Amazon. Figured that one of the reasons they never caught him was that everything 'seemed' right (and he always made sure there were no technical errors) and without a good reason to flag it, they just processed his return and gave him his money because, you know, they had about 30 million more returns to go through.
Wonder if he's still doing that? Jim, you out there?
Re:UNIX? (Score:3, Interesting)
Re:Hotmail is so irrelevant (Score:4, Interesting)
Gmail didn't appear until much later on, but Yahoo were creating some fantastic portal features.
I have a Gamil, Yahoo & Hotmail accounts, but prefer to give out my Hotmail account for "free offers" and other junk, its a junkbox.
However Gmail & Yahoo are both solid email solutions, and as you say, Gmail fairs better than all of them in the spam war.
Gmail; From a geek perspective, I admire them for creating key mappings that mimick those of vi/vim.
There is features present in Yahoo I'd love to see in Gmail:
* Setup up one-time (or temporary) email addresses that are binded to your email address.
* A decent calendar that can sync to iCal and Sunbird. (I don't think Yahoo have this yet)
* Events management, setup birthday reminders and the like.
* A virtual notepad that you can scribble down notes
* Sharing your calender, its private by default.
* Check number of new messages without logging in or providing credentials (uses a cookie)
Yahoo is awesome, if you havent tried out their web portal, take a look. Its very impressive.
Re:UNIX? (Score:2, Interesting)