Certificate Expiry Leads to Total Outage For Microsoft Azure Secured Storage 176
rtfa-troll writes "There has been a worldwide (all locations) total outage of storage in Microsoft's Azure cloud. Apparently, 'Microsoft unwittingly let an online security certificate expire Friday, triggering a worldwide outage in an online service that stores data for a wide range of business customers,' according to the San Francisco Chronicle (also Yahoo and the Register). Perhaps too much time has been spent sucking up to storage vendors and not enough looking after the customers? This comes directly after a week-long outage of one of Microsoft's SQL server components in Azure. This is not the first time that we have discussed major outages on Azure and probably won't be the last. It's certainly also not the first time we have discussed Microsoft cloud systems making users' data unavailable."
Re:Typical. (Score:5, Interesting)
Re:Somebody (Score:1, Interesting)
Caused by either corporate inadequacies in procedure, documentation, or checks of their systems. Or a corporate system designed to say "this isnt my problem".
Somehow I feel those worker visas are the issue here. Inexperience causes user errors, users administrate these large cloud systems, cloud configurations are complex and take years to master, yet cloud systems are a service provided by the lowest common denominator.
Corporate brilliance at its best.
Entwined failure loop... (Score:5, Interesting)
Re:Typical. (Score:5, Interesting)
You'd think that, but there's contract stuff. The thing is, you basically need a department in charge of renewing shit like this when you have enterprise level services. We've got a site with millions of hits daily and still manage to let it expire every couple of years. You try the credit card thing, but credit cards expire. You try recurring billing and then you get into a contractual nightmare with the registrar. The registrar isn't going to do you any favors, you might get millions of hits daily, but they still only get $5/year even from google.com so fuck you, figure out the billing yourself.
The only real way to do it effectively is build yourself a database of all the crap you need to renew regularly, then hire someone to renew that stuff. But who are you going to hire? It usually ends up being some assistant that doesn't know a damned thing about tech... and it's still going to cost you $60k a year in pay and bennifits to retain them. That's an expensive way of keeping track of such things... ah, the website admins can remember right?
Re:When you depend on other people ... (Score:4, Interesting)
Actually, there's a bit more to being "cloudy" than just virtual servers over the internet (indeed, they not even need be over the internet - you can have your own local cloud and many companies have internal clouds). Virtual servers over the internet is merely client/server. For a service to be "cloudy", generally it'll have attributes like HTTP (in other words, RESTful interfaces and each request being treated no different to the first request, in other words, the service doesn't hold state from request to request, just like with HTTP) and distributable. The main benefit of "cloudiness" is because of this you can easy scale up services when demand is high, and scale them back when demand is low. It makes it easier to make a resilient service than the traditional client/server type service where the server side has to keep state. Infrastructures like Amazon's EC2 allow you to scale things up and down easily and economically because you can turn on the "virtual server over the internet" part of it on and off very rapidly, and you only pay for the instances you've instatiated. But just using Amazon's EC2 doesn't automatically make your service "cloudy" if it does not have all the other necessary attributes.
Re:Somebody (Score:5, Interesting)
On the other hand, I've worked at places where the worst thing you could do is leave things that the company can't live without *in* the control of the company. Sometimes certain areas of expertise require specializations that the company just doesn't have and isn't interested in acquiring. Of course handing the responsibility of those things off to *Microsoft* is not necessarily any better.
Re:When you depend on other people ... (Score:2, Interesting)
Once you deploy to a vendor, you are stuck. From what I've seen, you can't easily move data and code from one vendor to another.
RHEL is CentOS is RHEL is Amazon Linux wherever you are. A basic of the cloud is that, as you migrate to it you migrate almost everything to Linux.
One of our clients is in the UK Azure cloud and we have to BCP about 6M rows from their server to our system every week. Takes over 90 minutes, and constantly fails because of losing the connection. We've looked at deploying systems to various clouds, and the costs were not worth it.
There have been outages in Amazon; almost nothing has ever crossed from one Availability zone to another. Multiple countries have never happened. At the same time there have been many total outages in Azure. Whilst Microsoft regularly loses data; every time a Google system fails totally, it turns out they have a tape backup. These are not "minor issues between very simlar services"; these are the fundamental differences which matter. If you attempt to use a Yugo as a form of armoured transport and then get shot, you shouldn't start saying "all armoured vehicles are terrible". Instead you should go and look for a system which fits your needs.
I will NEVER put any critical business system in someone else's cloud. At worst, I might put it in someone's data center on *MY* servers. The cloud seems to be fine for small business startups and non-important data for personal use. Businesses who no one would even notice if their site was down for a day.
You are asking the wrong question I think. It's not "Jakes cloud vs. mine". Instead the question is; what is the distribution; what is the split. Systems all in my own data centre are unlikely to survive if the data centre is flooded. Systems all in the cloud are going to be impossible to reach if the internet fails. A proper engineering decision probably uses some of your own servers backed up with multiple independent Linux based clouds each with a different technology base. E.g. Amazon +Rack space (OpenStack) +a private eucalyptus + a few dedicated servers.
BTW .. 'Cloud' computing is just remote virtual servers over the Internet. It's really not something new and original. People act like it's some amazing new 'thing'. Well .. it's not. It's just another way of letting companies with limited or no tech skills put up a web site or store data. It's expensive, proprietary, and I doubt very cost effective in the long run.
This is a total misunderstanding. It is "remote virtual servers over the Internet" which can be created, destroyed or modified in a small number of minutes or seconds using a clearly defined API. That makes it possible to do a whole bunch of interesting things (duplicate your entire Amazon system to Rack Space in five minutes if the whole of Amazon collapses for some reason) which are not possible with older fixed servers. It also makes a whole load of different potential problems (someone who has your master key can delete all your servers in one command).
In the end cloud computing is just another a tool. Just like a huge circular saw [wkfinetools.com], nothing it does is impossible with a hand saw. You could end up getting cut in half. However, if you need to regularly saw up huge number of trees, you may find you need one.
Microsoft has it's own internal CA (Score:4, Interesting)