Dropbox Open Sources DivANS: a Compression Algorithm In Rust Compiled To WASM (dropbox.com) 33
Slashdot reader danielrh writes: DivANS is a new compression algorithm developed at Dropbox that can be denser than Brotli, 7zip or zstd at the cost of compression and decompression speed. The code uses some of the new vector intrinsics in Rust and is multithreaded. It has a demo running in the browser.
One of the new ideas is that it has an Intermediate Representation, like a compiler, and that lets developers mashup different compression algorithms and build compression optimizers that run over the IR. The project is looking for community involvement and experimentation.
One of the new ideas is that it has an Intermediate Representation, like a compiler, and that lets developers mashup different compression algorithms and build compression optimizers that run over the IR. The project is looking for community involvement and experimentation.
Apache License (Score:4, Informative)
No such thing as better universal compression (Score:2)
The more a compression algorithm can compress one file, there has to be another file that it actually makes larger. Entropy is a bitch.
Re: (Score:2)
Its main upside appears to be being written in rust.
Yay for a language with API breaks that make its compiler unbuildable with previous minor versions of itself, and such a stellar portability [debian.org].
Re: (Score:2)
The language, its standard library and its one and only implementation all sucking are all minor problems compared to Rust's totalitarian community. I don't intend this as an insult, but in my opinion most Rust community members suffer from some degree of autism or Asperger's
I'd rather have that than have so party clowns or people who are obsessed with popularity. What you need for good quality is heavy focus and deep knowledge, and whether or not they have the social skills of a Kardashian is irrelevant.
Re: (Score:2)
I have no mod points and I must scream - yes. I am sick of new languages that are just toys for their development team to ticker with ad infinitum. A bit more thought before v0.0.1 and a lot less breaking v7.74.91 code with v7.74.91-r1 and I might bother my arse to look at these things.
Re: Dead project (Score:2)
Nah. Most of them are straight up astroturfers and disinformation operators.
Thats all well and good but... (Score:2, Funny)
What's the Weissman score?
ultimate compression- been there, done that (Score:1, Funny)
I like Dropbox and I'm sure they have a nice algorithm, but . . .
Does nobody remember the ultimate compression algorithm from 1995 that could scrunch any amount of data to less than 1024 bytes. The DataFiles/16 program got quite a lot of publicity for WEB Technologies.
As I recall there were some inconveniences; for instance for really serious compression one had to run the software multiple times- compress, then compress the resulting file, then compress that resulting file. Nevertheless that was a lot of c
one man's huge ... (Score:2)
Hard Drive Cost Per Gigabyte [backblaze.com] — July 2017
Looks like we're on track for $20/TB, if you purchase in bulk.
Let's monetize a "huge difference" at $1000 (which I regard as the smallest available value for a "huge difference").
Thus, your 1% extra compression needs to save 50 TB to make a "huge difference" of one large.
Correct me if I'm wrong, but I'm thinking your dataset needs to be on the order of 5 PB for a 1% compression improvemen
Re: (Score:2)
Here are the prices [amazon.com] for Amazon cloud storage. Depending on the type of storage, it ranges from $0.025 to $0.125 per GB per month. Yes, that's a lot more than buying a hard drive, but a huge stack of hard drives is pretty useless for storing lots of data. This gives immediate availability to all your data, backups, etc.
Let's say a company has 1 PB of data they need to store. Depending on the type of storage they need, that will cost between $25,000 and $125,000 per month. A 1% reduction in that cost cou
Re: (Score:2)
Let's say a company has 1 PB of data they need to store. Depending on the type of storage they need, that will cost between $25,000 and $125,000 per month. A 1% reduction in that cost could save them over $1000 per month, which is definitely meaningful.
Except 1PB is a lot of data. Walmart for example have 40 PB [forbes.com] in their data cloud, so they could save ~$40,000 on a $500,000,000,000 business. CERN has 200 PB so that'd save ~$200,000 compared to the $9,000,000,000 budget of the LHC. It's a rounding error and I think if you're working with that kind of data you've already worked on much more specific ways to compress it that won't leave much value in a general compression algorithm. Like Google working on a new video compression algorithm for YouTube makes se
Re: (Score:1)
Correct me if I'm wrong
You are not wrong, but you are missing the entire point of this.
If it was about disk storage space then Dropbox would be fine with just compressing it locally. There would be no need whatsoever to compile to WASM.
The point of having a WASM compressor is that you can compress the files on the client side without them having to install any programs for it.
The saving isn't in disk space, it is in bandwidth.
While compressing in the browser might be inefficient there is still an extra bonus for Dropbox here sinc