Dropbox Moves Users' Data Off Amazon S3 to Its Own Infrastructure 45
Reader Richard_at_work writes: Dropbox today announced that it has been working on a "top secret" project called Magic Pocket for the past two and a half years to get data of more than 500 million users from Amazon S3 to its own custom-built infrastructure. The company says that it has migrated over 90% of its users' data so far. Dropbox's relationship with AWS isn't completely over, however, as they will continue to use AWS for specific regional data stores where there is a requirement.
Not surprised (Score:3)
I'd say there is no surprise to see them vertically integrate; they're large enough to leverage the economies of scale of running their own storage for themselves; rather than to pay someone else to do it.
Re: (Score:2)
This is the problem with "the cloud." It's not supposed to be cheaper to build a power plant in my backyard, than it is to buy power from the "the cloud" err I mean "the grid." The pendulum swings again.
I just read their blog post. As far as I can tell, they don't state anywhere that this was done to save money. I'm guessing this has more to do with being completely in control of the entire stack, and not being beholden to a third party (Amazon).
Which is great... as long as it works.
Re: (Score:2)
90s internet service over telephone lines was probably the closest we ever got to total ubiquity of service from many providers. We were covered by a plethora of providers enough to push the price into 'cant build it cheaper, can only enter under-served mark
Re: (Score:2)
Likewise, for some companies having their own network and computer infrastructure makes sense. For others, not. For lots of companies, it makes sense to host their own services in some instances and use cloud services in others.
Can I download my files as a .zip archive yet? (Score:2, Interesting)
Does this mean that I'll finally be able to download my files as a .zip archive? I have some directories in Dropbox with a lot of files in them, and I get some bullshit message about the folder being too large to download, or something like that, when I try to use the functionality that exports the directory a .zip archive. It's not even that much data. Maybe like 5 GB in total. But I always get that fucking message, and it never lets me download these directories as an archive. I even bought the pro subscr
ZIP has 4GB, 65535 file limit, needs extensions (Score:4, Informative)
Unless they get new software that supports the ZIP64 extension [wikipedia.org] probably not.
Re: (Score:3)
That's likely a limitation of the .zip format and/or how they generate them for export. It's hard to generate .zip data on-the-fly because the file has headers at the end as well as the beginning, so they are probably generating an actual file and storing it before sending. They don't want to store .zip files for large folders for long enough for you to download. Not to mention the CRC32 checksums required for the format are still computationally intensive at a large scale.
I'm sure it's possible to gener
Re: (Score:2)
Re: (Score:2)
Former Info-ZIP maintainer here!
Classic zip files are limited to ~2GB (or ~4GB if your implementation is using 32-bit unsigned int). I can't remember 100% but it might also be limited to ~65,000 files per archive, too.
If Dropbox attempts to build the entire .zip in memory before sending it over the wire, you could be hitting RAM limitations on the server. If Dropbox builds the .zip on disk before sending it, you might be running into a connection timeout.
A tarball would be a better solution in this use case
Re: (Score:2)
If Dropbox attempts to build the entire .zip in memory before sending it over the wire, you could be hitting RAM limitations on the server. If Dropbox builds the .zip on disk before sending it, you might be running into a connection timeout.
You don't have to hold it all in-memory. You can not enable HTTP byte-range requests and generate it on-the-fly - flushing it out of memory as it goes out the wire. Zip and tar are both the same in that you have to read in a whole file to calculate the checksum for each file's header before sending the file. This isn't as resource-friendly as a running checksum that can be computed and stuck in a head at the end of a file's data since you'd have to read every file twice if you don't want higher memory re
Re: (Score:2)
And yes, that would mean Dropbox would probably have to write their own implementation of ZIP. Or at least re-arrange an existing library so that its output is not a single, monolithic file.
Re:Can I download my files as a .zip archive yet? (Score:5, Insightful)
Isn't that the whole goddamn point of the cloud? I can just use my goddamn web browser to interact with it, instead of a custom native app?!
Uh, no. The cloud is whatever the people running the cloud want it to be, you just want it to be something different - there are no rules regarding what the cloud must do.
At the end of the day, Dropbox is a syncing platform - that "goddamn desktop client" is the entire thing Dropbox is built around. If you wanted a different feature set, you chose the wrong product to use - there's no shame in admitting that, just don't blame the tool.
Dropbox has issues creating zip files for huge data sets, because it doesn't want to commit a massive amount of resources to building that zip file, its as simple as that - if that's the way you are using Dropbox, then you are using it wrongly and not as its intended to be used.
Re: (Score:2)
Does this mean that I'll finally be able to download my files as a .zip archive? I have some directories in Dropbox with a lot of files in them, and I get some bullshit message about the folder being too large to download, or something like that, when I try to use the functionality that exports the directory a .zip archive. It's not even that much data. Maybe like 5 GB in total. But I always get that fucking message, and it never lets me download these directories as an archive. I even bought the pro subscription, and it still won't let me easily download an archive of my directories! I don't want to install the goddamn desktop client just to copy a few directories of files from Dropbox! Isn't that the whole goddamn point of the cloud? I can just use my goddamn web browser to interact with it, instead of a custom native app?! Holy fuck, all I want to do is download an archive of a directory in Dropbox. Why the fuck do they make it impossible to do that easily?! Does this move to their own infrastructure finally make it possible for them to let me download my directories as .zip archives?!
Standard Zip compression/file format limits archive sizes to 4GB.
Why use some one elses cloud when you can make (Score:2)
Rolling your own is great until it fails (Score:4, Insightful)
Because your own cloud server requires maintenance, and when your cloud server goes down you're SOL until you, personally, have the time to troubleshoot and fix it.
How do I know this? My server developed a tic in it's network card, corrupting about 1 bit in every 5,000,000,000 or so. Took me a year to find that I actually had a problem with the server, and then two weeks to narrow down what the problem actually was. As a side effect I also found that I had a dodgy drive cable (one of 6 in the system) which showed no outward sign of problems because CRCs were correcting those bit problems.
Could this happen to a cloud service? Sure. Are they likely to catch it? Faster than I am, in all likelihood. Will it take them less time to correct it? You're damn sure it will. And for the cost of the time I spent troubleshooting my server, I could have paid for a decade of service from two cloud services so that I had 100% redundancy, and still had money to go buy a kegerator so I could drink beer instead of chasing bit problems.
Re:Rolling your own is great until it fails (Score:5, Insightful)
Because your own cloud server requires maintenance, and when your cloud server goes down you're SOL until you, personally, have the time to troubleshoot and fix it.
How do I know this? My server developed a tic in it's network card, corrupting about 1 bit in every 5,000,000,000 or so. Took me a year to find that I actually had a problem with the server, and then two weeks to narrow down what the problem actually was. As a side effect I also found that I had a dodgy drive cable (one of 6 in the system) which showed no outward sign of problems because CRCs were correcting those bit problems.
Could this happen to a cloud service? Sure. Are they likely to catch it? Faster than I am, in all likelihood. Will it take them less time to correct it? You're damn sure it will. And for the cost of the time I spent troubleshooting my server, I could have paid for a decade of service from two cloud services so that I had 100% redundancy, and still had money to go buy a kegerator so I could drink beer instead of chasing bit problems.
Don't count on it being any easier to troubleshoot rare network glitches with a cloud provider. Admittedly most of the time you can just launch a new instance and the problem goes away, but not always.
The first thing they'll do is close your ticket with "can not reproduce", so it'll be up to you to provide a test case to reproduce the problem. Which may not be trivial since you have limited visibility into their systems. And you have to convince them that it's not a security group problem, and not a local configuration problem (like iptables). And even then they may dismiss your ticket because you're not running their officially supported kernel version, so you'll have to fight with them to accept that it is a real problem, or capitulate and try to repro on their supported software version.
It took me 6 months to convince AWS support that there was a rare bug in network setup (not all subnets were reachable) that only hit once ever 500 - 1000 instance launches. They finally admitted that it was some sort of rare convergence problem in their network stack and that they are not monitoring for such problems so it won't recur.
At least when you own the hardware, you have full visibility into the entire stack, and while you can sitll have different teams pointing the finger at each other, they all work for the same company so management can step in and tell them to stop pointing fingers and work together to find the solution.
Re: (Score:1)
Depends on how small your business is. Dropbox is pretty cheap and effective as off-site collaboration spaces go, across multiple operating systems. It can double as an off-site backup for non-critical stuff as well. I don't see why you should pay a premium for off-site storage that will rarely be critical. If it ever becomes critical you can backup the files to another area *shrug*.
My administration has a local backup, copied to a NAS not directly connected to the laptop I run it on, and replicated to a dr
Re: (Score:2)
Ha - exactly. I do run my business off a Dropbox personal account, with multiple (both on and off line) backups.
Interestingly, the only way to replicate a data failure was to transfer a file of at least 1GB in size. About 1/2 the time a 700-800MB file was bad. And my files are generally in the 5MB range, with almost everything but the photo images in the 1MB range. I'd had some odd files that had either an odd color cast, or something out of the ordinary, but with a million files on the server, the chance
Re: (Score:2)
Yeah, but that somebody else has people on call 24 hours a day to fix stuff, and they're (We hope) experts at it. For a personal or small business server, I have to fix or update things maybe once every 2 years (yeah slackware!) which means I have to pretty much re-learn everything I did last time I touched it. And compared to most people I know, I'm an hands-on guy when it comes to my home network.
TBH, I probably spent, all told, about 10-12 hours in lost time, one way or another over those two weeks. I bi
Re: (Score:2)
Tell that to Netflix (who, as far as I am aware, still run on AWS).
Re: (Score:2)
Somebody else's servers require maintenance too, and when their cloud servers go down you're SOL until they decide your stuff is next in line to get fixed--once they figure out what happened of course.
That's what SLAs are for.
Why? Capex vs Opex (Score:2)
Why use some one else cloud when you could make your own?
Have you ever tried to roll your own industrial strength, production quality cloud infrastructure? That shit gets expensive, and it requires you to do significant investment up front. At this point we are dealing with issues of capital flow, acquisition (or rental) of equipment, depreciation of said equipment, etc.
Renting cloud services, on the other hand, it changes the equation. What used to be a capital expense, it becomes an operational expense. It might sound more expensive down the line, but it stil
So, reinventing the wheel again (Score:3)
So they basically re-did everything that backblaze did for it's storage pods.
Re: (Score:2)
So they basically re-did everything that backblaze did for it's storage pods.
When you're big enough, reinventing the wheel is worth it because then the wheel is customized for your use case. Spending a million dollars to engineer a custom solution is worth it to eek out a few percent better performance when you're deploying $25M+ worth of hardware.
There's a reason why Boeing doesn't just use off the shelf automobile wheels on their aircraft even though it would save them ten's or hundreds of thousands of dollars per aircraft, even though an automotive wheel is proven technology that
Re: (Score:2)
Well, if you mean they are both round and made of rubber, then yes. An aircraft wheel has to go from 0 to whatever the landing speed of the aircraft is in just a second or so. They also need to handle fairly extreme temperature ranges, ie: 150 deg F tarmac on a hot day to freezing or sub-zero temps at altitude, and back again. And they need to be able to do both of those, and more, repeatedly
Re: (Score:2)
"even though an automotive wheel is proven technology that does pretty much the same thing as an aircraft wheel."
Well, if you mean they are both round and made of rubber, then yes. An aircraft wheel has to go from 0 to whatever the landing speed of the aircraft is in just a second or so. They also need to handle fairly extreme temperature ranges, ie: 150 deg F tarmac on a hot day to freezing or sub-zero temps at altitude, and back again. And they need to be able to do both of those, and more, repeatedly and with as low weight as practical and safe. I doubt most automotive tires would hold up well under same conditions.
See, that's my point exactly. A wheel is a wheel as long as you just want someone to roll smoothly across pavement, but when you start looking at application constraints, not every wheel is suitable.
If Dropbox just wants to store any old data, they can use Backblaze's design because data is data. But if they want it to work optimally with their application, then it's worth coming up with a custom design.
Re: (Score:2)
There's a reason why Boeing doesn't just use off the shelf automobile wheels on their aircraft even though it would save them ten's or hundreds of thousands of dollars per aircraft, even though an automotive wheel is proven technology that does pretty much the same thing as an aircraft wheel.
Other than the fact the load on each tire on an aircraft is orders of magnitude higher than that of a car, and it has to maintain compliance at -50 deg C. Oh, and handle going from ~0 RPM to ~1000 RPM in the span of a second or two, whilst being heavily compressed. But other than that, they really do the same thing.
Re: (Score:2)
"Hi, Backblaze? This is DropBox calling. We're sick of sucking on Amazon's tit and wanted to do our own storage. Mind sending us all your details so we can do it just like you did? I'm pretty sure we couldn't do it better than you do, and boy do we love reading your hard disk diagnostic reports -- we're already hitting all the Best Buys we can find for disks."
Even if Backblaze has "opened" their storage system so anyone can copy it, who says it's the optimal way to do anything? I'm guessing at Dropbox
Re: (Score:2)
That's the point Backblaze *HAVE* opened their storage system so anyone can copy it.
https://www.backblaze.com/blog... [backblaze.com]
Not For Everyone (Score:1)
I hope that they use these cost savings to offer.. (Score:3)
More storage with their free tier! Seriously, guys, 2GB doesn't cut it anymore. Your competitors like Google Drive and Microsoft Onedrive are offering five times more storage for their free tier customers.
Re: (Score:2)
...their free tier customers.
For me, as free tier customer is not a customer, it is a prospect.