Digitizing Your Dead Trees? 367
smart2000 asks: "I'm tired of lugging around dead trees. I've just moved offices and had to move over 100 pounds of 'essential' technical books. It is clear to me that the dead tree industry is never going to supply the books I want in electronic form, so it's time to do it myself. What hardware and software should I use?"
"The Plan: Take the binding of each book and cut it off. Feed into a scanner with duplex and cut-sheet feeder. Scan as a 300 DPI jpeg with compression. Then OCR them overnight. I don't expect the OCR to be perfect, just good enough to use as a searchable index.
What are the suitable scanner choices for Linux? Any recommendations for OCR software that will write in an open format? Has anyone done this before?"
Re:Go To Kinko's!!!! (Score:4, Interesting)
Essential? (Score:4, Interesting)
Maybe you could donate the bulk of them to a school or something, follow the other suggestions about downloading fair-use versions where possible, digitize the few remaining ones, and start using ebooks or Safari [oreilly.com] (or similar) exclusively from now on.
Try one of these... (Score:3, Interesting)
Canon's 90ppm high speed scanner - only problem with high speed scanning is that they need loose leaves. Any decent books you have and want to copy will need a Stanley knife taking to the spine.
Please remember to make decent backups on a long lasting madium with a high chance of recoverability. Failing that place the loose leaf versions with a document recovery firm and take their insurance for the full purchase value of the originals.
Tech books shouldn't be dead tree only. (Score:1, Interesting)
People love books in dead tree format for the most part. You don't really want to curl up with a cup of coffee and a nice monitor. No, you want some good old dead tree.
But when you're coding, you don't want to curl up with a cup of coffee. You want to sit in a chair and hammer out code while quaffing coffee as if it were, well, coffee.
Most of the time when I look through books for reference, it's annoying. I'd rather be able to just grep for info.
Thankfully, at least O'Reilly's catching on to this.
contact your local school for the blind (Score:2, Interesting)
Re:Do you really need them? (Score:2, Interesting)
I own probably 500 computer books that completely cover an 6ft by 6ft section on my wall. No I haven't read all of them, but I have read 80% of them cover to cover, and I know the table of contents on the rest of the books. It's generally very useful to keep lots of reference material "grey matter indexed". That is, I know which book to find it in and roughly where it is in the book. I have found on-line documentation to be of very low quality personally, and I like to peruse it when I don't have a computer handy
The other consideration is it is nice to know the documentation isn't going to change, or move, or do anything weird. Of course it isn't going to get updated either so, cuts both ways.
PDF and OCR (Score:2, Interesting)
OCR sucks royally for large documents, documents with images or diagrams, handwritten comments, etc. However scanning the pages to an image and then creating a PDF of the images does not care about any of that.
So, scan all of your books as images that your OCR software can process. Use the OCR output to create an index of pages. If a specific word on a specific page doesn't OCR well who cares. With typed and professionally printed books your OCR software should be about 90% accurate. Take the images and create PDF files.
Now you have your nice clean images but you still have a searchable index. BTW, when you get this done post your procedures, problems, and solutions to a web site somewhere so that you can share your experiences with the rest of the world.
Somewhat on topic... Historical Papers (Score:3, Interesting)
Before doing this, though, we were thinking of scanning/copying all the documents to keep copies for ourselves. In doing so, though, we could use some advice:
What special steps must we take in scanning 150+ year old documents, some very yellowed and fragile?
What is the best format in which to store them (assuming we want them easilly readble in 20+ years for our kids)?
What is the best media upon which to store the data (again, hoping for readability in 20+ years)? (I'm thinking online storage to allow easy conversion to the media of the moment, but I still want something to stash in the safe deposit box)
Does anyone have experience with digital preservation/resoration of archival documents? Should I just try cleaning it up in photoshop or should I find a pro to help out? Maybe I can make it a term of the donation to the museum/library, for that matter.
Thanks in andvance for your advice.
Re:Safari is your friend (Score:2, Interesting)
for starters, I could only have access to three books at any givin time, I decided to just choose 3 books right when i signed up and later decided i wanted to trade one of the books in for another which they allowed me to do just fine. However, I then decided I wanted to check out another book and it said, sorry, you can only switch a selection once per month.. oh, isnt that handy, so
thanks oreilly, I love your books but you can keep your safari club.
You *need* to be aware of OpenDJVu (Score:5, Interesting)
It's truly a brilliant format. Go check it out.
Yours Truly,
Dan Kaminsky
DoxPara Research
http://www.doxpara.com
takes time, but is worth it (Score:2, Interesting)
A scanner with a reliable sheet feeder is essential. This doesn't necessarily mean expensive -- I've seen a lot of reasonable-looking scanners with ADFs on ebay for less than $100.
I cut the pages off the books using a single-edge razor blade -- non-ragged cuts are essential. Then I scan then into TIFF format at 300 DPI, greyscale. If I want searchable PDFs, I use OmniPage X on a Mac to create image-over-text PDF, it's quick and easy.
But most of the time, I these books are for Web viewing. So I use a graphics conversion program with batch capability (GraphicConverter on the Mac) to a) increase the contrast dramatically -- near 100%; b) trim the whitespace from the edge of the images; c) scale the pages as necessary. d) scale them more to create thumbnail versions. [atariarchives.org]
There are no hard-and-fast rules for choosing the final file type. Just got to balance file size and readability, and this varies from book to book. Sometimes I go with JPEG, sometimes 8-bit GIF, and sometimes 4-bit GIF. Sometimes I'll convert every page to GIF and also to JPG, then use a little script to select the smallest one for each page.
This is against the DCMA. (Score:2, Interesting)
By attempting to "recreate" the digital representation by using technological means to defeat the digital copy protection of a bound book, you are criminally liable to the owner of the copyright.
(Now if you were just copying this to another piece of paper, you may be ok under existing laws. But moving it to digital... Um, hands up scofflaw!)
dead trees to CDs (Score:2, Interesting)
Send the image directly from the camera to the OCR program. I find that the Xerox TextBridge program can do OCR on a page almost as fast as I could turn the page were I not using a scanner to input the text. TextBridge is quite ackward to use and not very customizable for new types of applications such as this.
Using a high resolution digital camera to input OCR text is also a good way to get around the question of whether or not to cut off the binding of the book.
By the way, I assume that you're wishing to scan european language text. Doing OCR on Japanese, Chinese, or Korean I would assume is much slower than recognizing ASCII. Does anyone know of an available program that will do OCR on Chinese?
With our friends in the middle east obsessed with blowing the shit out of us, it might be time to develop an open-source program that will do OCR on Arabic and Farsi, along with a translation program companion. Would Arabic be much more difficult to OCR because all of the phonetic symbols are joined together? I sometimes wonder about these things when I'm bumming about not having a life.
Well, YMMV, but that wasn't my experience. (Score:2, Interesting)
Safari also has a very good search engine, althought it's wierd that they coded it in MS ASP.
The spidering filter seems intent on inhibiting the casual copier. I thought this was lame, but there's actually a certain logic to it. If you go to all the trouble to download & reassemble the books, then you've put enough work into it not to not just throw the book out there on Gnutella.
At it's most expensive, Safari books cost $2 per month. So I'm not impeding anyone's education, and I'd like to see this service stick around. In fact, I can save people a bundle if I get them to use it the way it's meant to be used.
The one lame thing is that OReilly pads their selection with multiple editions of the same book and also with books that are available for free on the openbook site--ok, that's like five books, but still... They're really starting to get a good selection now.
In college, I used a free (as in stolen beer) html copy of a textbook for a class, and realized at the end of the year that someone had purposefully altered the book so that a lot of information was horribly incorrect. They'd basically cut out the word "not" all through the book, and inserted it after "is" in other places. Most people would not do that, but some a-hole did. Ah, college, what a hellhole.