Please create an account to participate in the Slashdot moderation system

 


Forgot your password?
Close
typodupeerror
×
Hardware

Digitizing Your Dead Trees? 367

Posted by Cliff from the when-you-can't-carry-them-with-you dept.
smart2000 asks: "I'm tired of lugging around dead trees. I've just moved offices and had to move over 100 pounds of 'essential' technical books. It is clear to me that the dead tree industry is never going to supply the books I want in electronic form, so it's time to do it myself. What hardware and software should I use?"

"The Plan: Take the binding of each book and cut it off. Feed into a scanner with duplex and cut-sheet feeder. Scan as a 300 DPI jpeg with compression. Then OCR them overnight. I don't expect the OCR to be perfect, just good enough to use as a searchable index.

What are the suitable scanner choices for Linux? Any recommendations for OCR software that will write in an open format? Has anyone done this before?"
This discussion has been archived. No new comments can be posted.

Digitizing Your Dead Trees? More

Digitizing Your Dead Trees?

Comments Filter:

  • look online before you scan (Score:5, Informative)

    by cheesyfru ( 99893 ) on Thursday May 09, 2002 @04:48PM (#3492957) Homepage
    You can find a wealth of PDF/PS/HTML/etc copies of computer texts online. Kazaa is a good place to start. Obviously, only download the books you have physical copies of. :-)
    • Most of the stuff you find online is training stuff, like Learn Photoshop or Learn HTML in 21 days or whatever.

      There's a dearth of available electronic copies of programming-type texts, except for those where the author/publish creates their own version (like all of Bruce Eckel's books).

      • I've got about 30+ O'Reilly books, Design Patterns, Stroustrap C++, etc. They're out there if you look long enough. LimeWire has also been a big help in it as well.
        • O'Reilly actually sells electronic editions of their books, so please buy them! You can also subscribe and read many of their books online. Also a good idea.

          (I personally like my dead tree O'Reilly books, and will stick with them until I have a really hi-res lcd to read electronic versions with.)
      • There are hundreds of them here [theassayer.org]. Very few are the kind of dopey software manuals you're referring to. Is that a "dearth?"

    • already scanned (Score:2, Informative)

      by Anonymous Coward
      Yup. There is quite a lot already scanned. The best places to look are usenet (at alt.binaries.e-book, alt.binaries.e-book.technical, alt.binaries.e-books) and IRC at #bookwarez and #bookz on undernet, dalnet, and irc.nullus.net (and most likely other irc nets as well.)

      You could try making a request in abeb, but the biggest selection in one place is irc. So as long as you are not scared by the interface, that is where I would look first.
    • and consult your lawyer

      'cause Elcomsoft thought they could do the same (minus the scanning part) and they were wrong [slashdot.org]. I don't think you need to copy an electronic version to be a pirate. You can scan a paper copy and become one.

      But then again, IANAL...

    • Another place to look... (Score:2, Informative)

      by zaren ( 204877 )
      is http://docs.rinet.ru:8080/ - I ran across this site a few years back. It almost looks like an online library for a Russian ISP's technical support staff.

      They've got lots and lots of official books, all HTMLized a chapter or a section at a time. They're all a bit old or out of date, too - I know of one Perl book in particular that they have there was one edition behind what was being sold on the shelf at the time I saw it.

      -----
      Is Darwin an evolutionary OS? [cafepress.com]

    • "A wealth" of ebooks? Yeah right. If you're a total freakin' nerd. There's 1) Programming boooks 2) Sci Fi and Fiction (only from the most popular/oldest authors including Harry Potter) and 3) How to get laid for Dummmies (No joke). And there's absolutely nothing in Spanish (which is a thing of mine since I live here in Spain and want stuff to practice on).

      I've thought of doing EXACTLY what this guy is doing. I hope there's some good advice... I can't wait until ebooks are as popular on Gnutella as MP3s.

      -Russ

  • An easier solution. (Score:4, Funny)

    by SystemFork ( 578511 ) on Thursday May 09, 2002 @04:48PM (#3492959)
    Lots of college students at $5/hour.

  • Go To Kinko's!!!! (Score:4, Informative)

    by thedbp ( 443047 ) on Thursday May 09, 2002 @04:49PM (#3492963)
    Kinko's offers high-volume scan-to-PDF solutions ... at low volume, it is usually a 10 - 25 per page and the cost of the media to copy it to, but in large volume, sometimes the cost can go down to 1 per page.

    Call Kinko's. Ask for the Territory Representative. They'll help you out!!!

  • monkeys (Score:4, Funny)

    by blugecko ( 152079 ) on Thursday May 09, 2002 @04:49PM (#3492966)
    hire an infinite amount of monkeys on typewriters and... oh wait, that is for shakespeare

  • Safari is your friend (Score:5, Informative)

    by Dredd13 ( 14750 ) <dredd@megacity.org> on Thursday May 09, 2002 @04:50PM (#3492971) Homepage
    If you're like me, a good chunk of your collection is ORA books... in which case, you should check out O'Reilly's Safari [oreilly.com], which is their online book offering. It also includes non-ORA books as well, actually.

    Quite useful and handy.

    D

    • But unfortunately, owning the OReilly books doesn't entitle you to be able to access them online. You'd have to pay a subscription to access them.

      That being said, the $9.99/month (or so) would probably be worth it, considering all the work tearing apart and OCRing all the books would take, just to get somewhat inaccurate digital versions.
      • "But unfortunately, owning the OReilly books doesn't entitle you to be able to access them online."

        That is, access them on their web site. You can put them on your own private webspace, on a CD, etc. It's no different than mixing your own music CDs from CDs you legally own.

        But yes, O'Reilly's fees are much less than what you'll pay to scan it all yourself.
    • Perhaps the original poster should subscribe to the O'Reilly books they've purchased (for a month) and then save each chapter locally. Even at Safari's upper subscription levels of $100/mo you get access to 200 books. There's no way you could get a quality scanner with a feeder and OCR software for less than $100. Re-inventing the wheel is instructive, but silly. ------

    • Re:Safari is your friend (Score:5, Informative)

      by Wanker ( 17907 ) on Thursday May 09, 2002 @05:05PM (#3493091)
      I'll second this-- the O'Reilley Safari site is wonderful for anyone with a hoard of tech books.

      I bet about half of your books are already online.

      Also, for your compression you should NOT use JPEG. JPEG is optimized for smooth tones and will badly blur hard edges like text. On the other hand, JPEG performs relatively poorly at compressing large areas of the same color (i.e. white backgrounds.) [Note for the nit-pickers, both of these JPEG issues will be reduced/eliminated in JPEG2000.]

      I scan documents to either compressed TIFF (tend to be large), PNG, or (*shudder* [unisys.com]) GIF.

      From the Project Gutenberg "Making Etexts from Paper Originals" paper" [promo.net]: (You can bet these guys know how to scan...)

      A general rule is to store scanned images to JPEG and store computer-generated pictures (like diagrams etc.) to GIF. The exception is if you scan in grayscale, then use GIF. Never scan pictures as lineart. If acceptable from a file size perspective use the highest possible quality setting for JPEG.
      I suggest never using JPEG. The quality loss for printed words is just terrible relative to the compression you get. Also, just substitute PNG for GIF and the above works.

      • I'll second what you just said about formats.

        And the tragedy is, the National Geographic Magazine collection on CD-ROM consists entirely of JPEG pictures of the pages (well, plus some (Win/Mac) indexing software). Okay, the photos are probably what attracts most people to National G, but the articles are damn hard to read.

        The folks (Tinker's Guild [tinkersguild.com]) that did the complete collection of The Amateur Scientist columns from Scientific American (admittedly a less ambitious undertaking than National Geo.) converted all the articles to HTML (illustrations in GIF). And the indexing software is in Java. Kudos to them.

      • Use JBIG - not GIF (Score:3, Informative)

        by mangu ( 126918 )
        For bi-level images, the standard to use is JBIG, comes from an ISO group similar to those that created JPEG and MPEG.

        It generates much smaller files than GIF for printed text, with none of the inconveniences of JPEG. Grey scale pictures come reasonably well, if done at 300 dpi, dithered.

        I don't know exactly why JBIG never caught like those other standards. There doesn't seem to be many JBIG programs around, but, if you are handy with source code, there's jbigkit, a library for reading and writing JBIG files. I wrote my own software with that, and converted a half-ton of old magazines into a 20-pack caselogic of CD's.
    • I subscribed to the safari club shortly after they announced it and I was not pleased.

      for starters, I could only have access to three books at any givin time, I decided to just choose 3 books right when i signed up and later decided i wanted to trade one of the books in for another which they allowed me to do just fine. However, I then decided I wanted to check out another book and it said, sorry, you can only switch a selection once per month.. oh, isnt that handy, so .. do you really have access to all the books no matter where you are? no, you only get access to a few. then I thought, it would be nice if I could save a local copy and then put it in a nice searchable databse. no way, they stopped me in my tracks for turning the pages too fast because they detected that I was a spider.

      thanks oreilly, I love your books but you can keep your safari club.

  • Comment removed (Score:5, Funny)

    by account_deleted ( 4530225 ) on Thursday May 09, 2002 @04:50PM (#3492974)
    Comment removed based on user account deletion

  • Great (Score:2, Insightful)

    by Quill_28 ( 553921 )
    Now the bookseller's will join with the entertainment industry. Nexty we will be seeing books that can't be scanned easily.

    Remeber those passkeys for computer games in the 80's that were black on maroon paper? Or some dial thingy.

    • Re:Great (Score:3, Funny)

      by yintercept ( 517362 )
      Cool idea. You could sell special 3D glasses with an encrypted pattern that you would have to purchase to read a book. With the print on demand technologies, book seller might create a system where people have to get a special printing of the book that fits only their encrypted readers. That way you can guarantee that only one person reads the book. You could also create a pretty good database of what people read. This would give you a good idea on who are the subversive elements in society.
      • Sounds like AD&D Tomes of Enchanted whatevers... spend three months studying it and it disappears... Now THAT's something the copyright nazis would love.
    • Remeber those passkeys for computer games in the 80's that were black on maroon paper?

      Even back then, every photocopier I ever tried it on could adjust the contrast so that they could be copied legibly.

      I also remember trading copied templates of the dials that you could cut out and assemble.

  • 100 pounds? (Score:5, Funny)

    by NineNine ( 235196 ) on Thursday May 09, 2002 @04:51PM (#3492982)
    That's it? Jesus, what are you, a 12 year old girl? That's 2 armloads. Sounds like you need the exercise, fatass.

  • You're mad, surely? (Score:2, Insightful)

    by fractalus ( 322043 )
    Most of my technical books contain vast quantities of useful information in charts, diagrams, and illustrations... which are far more of a challenge to OCR than mere printed text.

    I suspect that even were this sort of thing really possible, it's a major time investment. I have several dozen technical books I'd like to scan, each with four hundred or so pages... and I'm not sure I want to spend a week's vacation time doing it.

    And even were it done... there is just something comforting about having a nice printed book that I can set on the desk next to the computer and consult, without having to read it on the screen. Print still looks way better than monitors.
    • It's a convenience issue. I'd love to have all my books on CD's so I can either 1) leave them at work and use the dead tree's at home, or 2) carry them back and forth each day. There have been plenty of times that I need a resource that I know I have at home ( "I think something out of the Dragon book would help here"), but no way to access it.

  • Do you really need them? (Score:4, Insightful)

    by alt.sex.fetish.jesus ( 542450 ) on Thursday May 09, 2002 @04:52PM (#3492991)
    I suppose this will be marked off-topic, since the poster is asking about digitization hardware. But whenever I see coworkers with tons of books on their desk shelf, I wonder to myself why they really need them. Do they actually have time to read them? Or are they more for show?

    Personally, I have about 3 books I consider _essential_, and I've read them cover to cover (mostly while in the crapper ;-) ). The rest of the time, I get what I need off the web or USENET.

    As far as I'm concerned, the most important quality in an engineer is not what you know but what search engine you use to look stuff up.
    • All depends. I have probably 8 C++ books that have lots of different useful information in them. Really, I probably only need 3 of them, the ISO standard (yes I own a copy), Strousup's C++ Language and Jossutis's book (big black book, can't remember the title).

      I own probably 500 computer books that completely cover an 6ft by 6ft section on my wall. No I haven't read all of them, but I have read 80% of them cover to cover, and I know the table of contents on the rest of the books. It's generally very useful to keep lots of reference material "grey matter indexed". That is, I know which book to find it in and roughly where it is in the book. I have found on-line documentation to be of very low quality personally, and I like to peruse it when I don't have a computer handy

      The other consideration is it is nice to know the documentation isn't going to change, or move, or do anything weird. Of course it isn't going to get updated either so, cuts both ways.

    • Re:Do you really need them? (Score:4, Insightful)

      by sphealey ( 2855 ) on Thursday May 09, 2002 @05:22PM (#3493195)
      I suppose this will be marked off-topic, since the poster is asking about digitization hardware. But whenever I see coworkers with tons of books on their desk shelf, I wonder to myself why they really need them. Do they actually have time to read them? Or are they more for show?
      Because once you have developed the skill of processing technical books/documentation, you can scan through them and pick up critical information rapidly - far faster than you could click through them as hypertext.

      Case in point: I recently took a position where I had to do some work with Oracle, which I had not used previously. After some skimming at B&N, I purchased 5 good texts. A lot of pages, but when you need to figure something out you can open 2 or 3 of them, mark multiple pages, and get the outline of what you need very quickly.

      sPh

      • Blockquoth the poster:

        Because once you have developed the skill of processing technical books/documentation, you can scan through them and pick up critical information rapidly - far faster than you could click through them as hypertext.

        .... at least, until you develope a comparable skill with hypertext. The manner of reading is different but not necessarily inferior. Why does everyone assume that what we've used simply due to technical limits will actually prove to be superior in a new context? You can't grep books -- that already limits them.
    • I have a couple dozen bookmarks to stuff on the net about html, cgi, php, etc. I also have a half dozen of the O'Reilly books on similar topics, as well as most of their Perl collection. Which one do I find a quicker way to get at what I know is there? The books, hands down. Web pages tend to be broken up into individual "pages" to "simulate" being books, but don't have good indexing. Google doesn't count as good indexing, except insofar as I can find information that I've never seen before, because if I *have* seen it before typically it's still tough to find the right magic words to get exactly what I saw.

      So the answer is yes, I really need them. And I bet the original poster does too. And see, that's the hard part. He can scan and download and so forth all he likes, but finding a good index replacement is not going to be so easy.

    • Re:Do you really need them? (Score:5, Insightful)

      by Waffle Iron ( 339739 ) on Thursday May 09, 2002 @05:44PM (#3493366)
      Do they actually have time to read them? Or are they more for show?

      Back before the Web when I was a hardware designer, books were a kind of currency that engineering salespeople used to entice you to meet with them. Each chip manufacturer printed stacks and stacks of data books covering their various product lines. They'd give these to the sales reps who would cart them in on dollies to hand out to the engineers who showed up to hear their latest pitch.

      In a way, huge bookshelves with hundreds of books was a status symbol, showing that you'd been around a while and a lot of people thought it was worthwile to give you books. It was useful to have all of that info available, but few people actually used more than 1% the data that was on their shelves.

      The instant the chip companies put their chip data on the web, all of those books became totally useless. Now I'm doing software, everything is online, and I can go for weeks on end without picking up a technical book.

      I do sometimes miss the office atmosphere you get from row after row of data books neatly segregated by the corporate logos and color schemes on their spines. It had an important look to it.

    • A mathematician, a physicist, and an engineer are asked to find the volume of a red rubber ball. The mathematician measures the diameter and calculates the ball's volume. The physicist submerges the ball in a full beaker, and measures the amount of water that spills out to get the volume. The engineer turns the ball over until he find's it's serial number, then looks up the volume for that model on his Red Rubber Ball Table.

      Half of the library in my office is catalogs and equipment data sheets for components. A lot of the rest is more generalized data like stress concentration factors for various object geometries and material characteristics; these are things that CANNOT be derived from theory. Only about 4 of my books (which, admittedly, I do use a great deal) are theoretical books. Physics, Advanced Math, Design of Experiments, and a Mech. Eng. Handbook. When you work with real objects, rather than just theory and pure numbers, you tend to need a lot more detailed reference materials. And I'm sure that at least one Engineer in the red rubber ball industry has himself a Red Rubber Ball Table.
    • Because the yellow highlighter looks like shit on my CRT.
    • Most data sheets and application notes are downloadable pdfs at the vendor site. If you know what chip vendors you need, who needs a search engine?
  • You could always look into those funky OCR pens that you see in some electronics catalogs. Basicaly it's a pen with an optical sensor that you scan over lines of text to digitize them, they can then be transferred to a computer or to a palm pilot (or like product)
    • This would actually be quite useful -- basically scan the texts as you need them.

      When you need to look something up, you scan it as you go, almost like highlighting the text with a bright yellow marker.

      Let's face it -- out of most of these 500-page behemoth books often only use small chunks of them, especially when you're talking about using them as reference tools well after your first or second read. This way you wouldn't be wasting time, energy, electricity and disk space with all of the voluminous words you don't really need.

      I think this could be the best advice I've seen so far.
    • I'd complain that the code you scanned would come out formatted poorly, but your wrist would be so carpal-tunneled by the time you're done scanning 100 lbs of books line-by-line that you wouldn't be able to type code anyway!

  • Talk to the project Gutenberg guy (Score:3, Informative)

    by Anonymous Coward on Thursday May 09, 2002 @04:53PM (#3492995)
    Check out project gutenberg. I remember that they have a very nice how-to for scanning in texts

  • Question is: Free or Not Free (Score:4, Informative)

    by The Ape With No Name ( 213531 ) on Thursday May 09, 2002 @04:53PM (#3492996) Homepage
    You could scan it all into PDF/PS but I am not sure about making it all into a document with free tools after that but here is a go at a solution.

    Adobe Acrobat (read $$$$) does all of this and works well. But if you are *nix person you could pipe some ghostview tools together and put it all into LaTex then re-export it as a digital book in to PDF. Scanning: look no further than a HP scanner. It doesn't even have to be HQ unless you need the diagrams to be photoquality. After that burn it all to CD or, better, DVD.

  • Essential? (Score:4, Interesting)

    by daeley ( 126313 ) on Thursday May 09, 2002 @04:53PM (#3493005) Homepage
    If they're that 'essential' how can you justify cutting them up? (A 100 pounds of tech books is, what, three or four books? ;)

    Maybe you could donate the bulk of them to a school or something, follow the other suggestions about downloading fair-use versions where possible, digitize the few remaining ones, and start using ebooks or Safari [oreilly.com] (or similar) exclusively from now on.

  • I work in this field (Score:5, Informative)

    by JeanBaptiste ( 537955 ) on Thursday May 09, 2002 @04:54PM (#3493006)
    My company is a document imaging systems reseller. The drawback to siong this is that it is expensive. We work with many different libraries and we sell them book scanners. They do lots of neat things, including things like not breaking the binding of the book during scanning, binding curve compensation, masking/centering, and so on. Most of these customers then take the tiff images and upload them into a document imaging system, although you could easily make pdfs also.

    <plug>
    Let me recommend the PS7000 from minolta (www.minolta.com), that is the book scanner we sell the most of.

    If you are at all interested in document imaging, check out www.otg.com

    and if your in minnesota, wisconsin, or the dakotas, check out my companies web site at www.mid-america.com
    </plug>

  • check sane (Score:4, Informative)

    by walt-sjc ( 145127 ) on Thursday May 09, 2002 @04:54PM (#3493013)
    Check the hardware list for sane and then pick one of the fastest scanners you can afford. The DB on Sane's web site is your best bet. You will find that to get good scanning speed you will need scsi as USB is just too slow.

    jpeg also sucks for this. Jpeg is best for full color images like photographs. Better off using tiff or png. Most OCR software will require tiff. Don't know of any OCR software for linux although you might get some windows app to work under WINE. Textbridge from Xerox isn't bad for the money.

    • Re:check sane (Score:3, Informative)

      by josepha48 ( 13953 )
      There is gocr or jocr -> http://jocr.sourceforge.net/

      Also there are a few commercial ones. However scanned to text conversion needs at least 600dpi and is only goind to have about a 97% accuracy.

  • We do this all the time at the office...... (Score:4, Informative)

    by diorio ( 244324 ) on Thursday May 09, 2002 @04:54PM (#3493014)
    .....we use a xerox DC265ST. This digital photocopier scans pages at 65 per minute and posts them to an FTP server inhouse. It can scan at 300 or 600 DPI and you can apply OCR after the scans are done. The DC265 is a workhorse and there are about a million of them out there. The scan back feature is a additional price on the device so not everyone spent the money on that feature....but about 1000 Kinko's have these in house and a Kinko's with a good DTP department might actually even know how to use the feature. (Good Luck!)
    .
  • I dont know HOW many times i've looked at a tech manual(or other paper book for that matter)trying to find something I read a while ago and thought " i wish i could just do a text search to find the 3 or so words i remember seeing..." Sure theindex and table of contents gets you part of the way there, but if the author mentions something off-hand in an 'unrelated' section of the book...

  • Try one of these... (Score:3, Interesting)

    by matthew.thompson ( 44814 ) <matt@actualiNETBSDty.co.uk minus bsd> on Thursday May 09, 2002 @04:56PM (#3493023) Journal
    Canon DR-5020 [canon.com]

    Canon's 90ppm high speed scanner - only problem with high speed scanning is that they need loose leaves. Any decent books you have and want to copy will need a Stanley knife taking to the spine.

    Please remember to make decent backups on a long lasting madium with a high chance of recoverability. Failing that place the loose leaf versions with a document recovery firm and take their insurance for the full purchase value of the originals.

  • The first question you'll want to ask yourself is whether you want the result in searchable text form or scanned image form. Searchable text is achievable with OCR (optical character recognition) software, but has at least two issues:

    • OCR software isn't perfect, and so errors will occur that'll you'll either have to live with or correct manually. Good OCR software does some validating against a dictionary, but this doesn't help when the source is highly mathematical, etc.
    • You'll lose figures, diagrams and pictures.

    Scanned images solve these problems, but have two problems of their own:

    • They're not searchable.
    • They're bulky (perhaps 100x).

    Perhaps a hybrid solution exists, but I suspect such a solution will require a lot of manual intervention and tweaking, something you'll want to avoid if your goal is to digitize several books.

    • i seem to recall a product that adobe has which makes hybrid pdf files using ocr. Text where possible, graphics elsewhere. You get the benefits of both. Of course the software is expensive.
      • Acrobat can do this. Just scan it in with Acrobat, then "capture text." Works well with good, clear fonts, and a straight scan (not crooked) from a good scanner, though there's like a 0.05% fail rate per character. Yes, I know that sucks, it's one error a page, but it's survivable.
  • Electronic manuals are great, particularly because of the ability to search them. I certainly use plenty of them.

    Personally, however, I still like printed manuals. Using an online manual means either reducing some windows or switching desktops. With a paper manual I can keep the screen exactly as it is. Higher resolution screens, or the use of multiple screens, are making online manuals much more useful (anyone remember what a pain in the ass it was to try and figure out something with only an online manual on a 640x480 screen?). Occasionally I still manage to fill two 1600x1200 screens with a bunch of stuff I want to keep visible while still reading the manual.

  • if you are anything like the computer guys I know (myself included), you'd end up printing out
    portions of the text whenever you wanted to read them anyway!!!
    • Real men use the command shell and man() or google ;)

      Seriously, most of the hard-core computer folks I know either open their copy of the ORA book on the subject, steal their neighbors copy and flip it open, or use some form of online docs w/o printing said docs off. The only reason I've ever known anyone to print anything resembling a doc is when someone I knew had assembled binder full of pages on tech specs for a project.

      It's just a lot easier to sit at the screen arrowing up and down on the doc than it is to print it, reach over to the printer, pull it out, shuffle through it....and then eventually have to take it out with the trash. I've seen comments about paperless offices vis a vis paperless restrooms, but the fact is that for reference there really isn't a reason to print the online doc.

  • I want both (Score:2, Informative)

    by peterdaly ( 123554 )
    O'Rielly (sp?) has many of their java books available on CD-ROM, although I only own the dead tree versions of the ones I have in that series.

    On a regular basis, I haul 2188 pages worth, I just added them up, of QUE's Using Java2 Standard Edition, and Enterprise edition, between home an the office. (Speaking of which, go to the link in my .sig and buy some of my favorite books!) That a lot of weight for two books, and I usually haul around a couple smaller ones as well, O'Riely's perl book, and their EJB 3rd edition.

    Not only are all of these books heavy, but I have also yet to find an easy way to card them around, they don't all fit right in any of my bags.

    I want all of these books on CD-ROM, but not just CD-ROM. Half the books I have INCLUDED a cd-rom, it just doesn't contain the texxt of the book. With O-Riely, I'd buy the CD-ROM version, but I want to dead tree version too. I want to use the dead tree version, unless I am working from home, I want to haul home the CD's. I don't think I should have to pay any more for it either, I bought the IP (in the property sense), and I am already paying the price for the wood slices, which includes a silver disk.

    PUBLISHERS, GIVE ME THE BOOK ON THE CD TOO! I spend $100/month or so on tech books.

    -Pete

  • Let me get this straight... (Score:5, Insightful)

    by deacon ( 40533 ) on Thursday May 09, 2002 @05:03PM (#3493073) Journal
    You are going to cut up thousands of dollars worth of your "essential" books?

    And put them into an inferior visual format you cannot read without the computer being working and on?

    And you are going to spend about 100 hours to do this.. and the original books are going to be ruined.

    All this just so you don't have to make 3 trips to move your books?

    Mmmkayyy.. (backs away slowly)

    Have you ever heard of a dolly?

  • Schools for the blind have been doing this for years, especially with technical books. Many of my V.I. friends would remove the binding and feed them through a high-speed sheet feeder to a scanner. Then, the books are proofed by seeing people for OCR perfection. Contact your local school and ask if they already have some of your works in pdf/jpeg/tiff/WordPerfect (yes, lots of Word Perfect). They may be willing to give you some legal copies of your books in exchange for you converting some of the books you have that they don't into blind readable format (which means, you'd have to proof your own book for accuracy - but you're doing that anyway). Basically, you're donating your time for a good cause and bennifiting yourself.

  • are you sure you want to do this? (Score:4, Insightful)

    by binaryDigit ( 557647 ) on Thursday May 09, 2002 @05:05PM (#3493089)
    I think you may be underestimating the sheer enormity of your task. Getting sheets to all feed right (a little skew and you're skrewed) and in order (feeder issues, what happens when one page mis-scans/feeds, can you go back and insert it into it's proper location), handling front to back issues (though I would assume that decent scanning software would take care of this for you). Also, your plan to use jpg might be problematic. OCR is finicky enough as it is, back when we were scanning documents we always used 300dpi tiff (using group3 or group4 lossless compression) to get the maximum accuracy rates from the ocr package we were using. And speaking of accuracy, keep in mind that OCR software that has a 97% accuracy rate means that it will flub 3 out of every 100 words, in a book that might contain tens/hundreds of thousands or words, that is a whole lot of errors. Now it's been a few years (6-8) since I've done this kind of stuff, so who knows, maybe things are much better now?

    I've been wanting to do something similar for years, but with technical magazines, not books. But the sheer amount of manual labor involved has turned me off considerably (not to mention the thought of destroying the original source).

    Keep in mind that this is such a common need, that if it were pretty straight forward, much of it would be done already (perhaps someone out there has the time/hardware/software to have done some of this already?) Not to mention the issue that with the web, that much of the information contained in those books are now available online, makes you wonder if it's really worth the time and effort, esp. considering that a great many of the technical books are obsolete two weeks before they hit the shelves.
    • I've been wanting to do something similar for years, but with technical magazines, not books. But the sheer amount of manual labor involved has turned me off considerably (not to mention the thought of destroying the original source).

      Dr Dobbs (and I'm sure others) offers CDs full of all their articles from the past couple years for a pretty good price (less than $100, I believe). They also offer collections of books on CD for about the cost of one original.

      Just a thought,

      hgh
      • Yes, I was aware of DDJ online. It's cool that they are offering this (and have been for a several years now, pre-internet, well, pre-widespread-internet anyway). Two problems though, first is that this only represents a small portion of the stuff that I happen to have. Old Byte, Micro Cornucopia, PC Tech, Compute, etc, etc are probably not going to make it any time soon (though I guess Byte might, they already have some).

        The second and the one that many people don't really think of (and to be honest, care about) are the ad's. Both as a reference (for many old products, the ad can be the only source of information) and for entertainment value (hey, look at the 20MB MFM Seagate for $1200, not including controller). The ads always get lost when companies put their content online, sigh.

    • Re:are you sure you want to do this? (Score:4, Informative)

      by Hallow ( 2706 ) on Thursday May 09, 2002 @05:26PM (#3493236) Homepage
      What he's probably looking for is something like PDF. You can leave the image on the front (i.e., it's what shows up in acrobat reader), and adobe's ocr ocr's the document and and indexes it for searches. The problem with this is, you wind up with big pdf's with poor quality.

      Where I work we tried to turn a book into PDF that we no longer had an electronic copy of. Keeping the images up front with ocr text behind, about 300 pages alltogether. Even with max compression, and the lowest acceptable DPI (300 I think), the PDF came out to 95MB. It didn't help that we scanned the book page by page and generated the PDF by hand, on a slow hp general consumer model scanner, either. (the initial pdf took over 120hrs to produce, with rescans and ocr'ing and everything).

      We wound up taking the acrobat ocr'd text (it was better than the off the shelf ocr package we had at the time) via the adobe accessibility website, and fixing it up. It was a pretty big project.

      We recently hired a document imaging company to PDF a lot of smaller historical documents for us, and that has worked out well. It's kind of pricey, but we also paid them to proof the ocr behind the images, and to hand adjust the images for appearance. It's worked out rather well.

  • PDF and OCR (Score:2, Interesting)

    by 4/3PI*R^3 ( 102276 )
    If you really want to go through all this effort use both PDF and OCR.
    OCR sucks royally for large documents, documents with images or diagrams, handwritten comments, etc. However scanning the pages to an image and then creating a PDF of the images does not care about any of that.
    So, scan all of your books as images that your OCR software can process. Use the OCR output to create an index of pages. If a specific word on a specific page doesn't OCR well who cares. With typed and professionally printed books your OCR software should be about 90% accurate. Take the images and create PDF files.
    Now you have your nice clean images but you still have a searchable index. BTW, when you get this done post your procedures, problems, and solutions to a web site somewhere so that you can share your experiences with the rest of the world.

  • Start with google. There is a lot of technical information online, and google will find it. Not as good as those dead trees, but if you can find it and it is accurate, google is often easier than searching indexes. Best of all, dead trees are limited to the ones you own, while google is limited to whatever someone found useful to put online.

    Note the last line of the above: google is limited to what someone else finds useful to put online. So if you can't find it on google, take some time to put it online for the rest of us. If/when you find yourself going back to the same few sites often, link to them from your homepage so google knows you find them useful. In other words, google is interactive, make it work for you and it will work for everyone. The internet is not a one way street.

    Finially, some things are just plan eaiser to look up in dead tree format. I would strongly recomend you keep your books intact. Put the information you need on the web (what you can do legally), and keep the books for the rest. If you find you are not using a book anymore because all the information is on the web (including you put it there), then throw it out. My monitor is only 19 inches, not nearly enough to hold all the information I have scattered about my desk.

  • Blackmask.com [blackmask.com]

    Tons and tons of e-texts. In multiple formats: text, pdf, lit, HTML.

    Excellent resource!

    • First I thought about modding your post up. Then I went there and looked and afterwards I considered modding you down instead. (I have mod points right now.)

      Why? Because the Blackmask site you refer to has few or no books of the type referred to by the original post. There does seem to be a lot of cool content there, but most of it is stuff you can find just as easily on the Project Guttenburg site or elsewhere.

      So basically your post is somewhat off-topic, almost cool, but not really cool enough to merit a mod up despite the off-topicness of it. If I would have wasted a down-mod point on you someone else would have meta-modded it badly because they probably wouldn't know why I modded as I did. And, as I said, I just don't think the link is worth the mod up, despite the fact such a mod would probably survive a meta-mod.

      All this points out one interesting fact about meta-modding -- it may work better than its critics give it credit for! At the very least it makes a subset of the moderators (a subset with at least one member, me) think twice before bestowing mod points either way. Note that I often lose mod points when the time runs out because I just don't find anything truly worthy of moderating.

      Jack William Bell, who fully expects someone will mod this down as 'Off-topic'...
  • Anders Borg [torget.se] wrote this FAQ [promo.net] from Project Gutenberg [promo.net]. Lots of field-tested advice there, such as a suggestion to scan at 300dpi or better.

  • Somewhat on topic... Historical Papers (Score:3, Interesting)

    by Embedded Geek ( 532893 ) on Thursday May 09, 2002 @05:12PM (#3493135) Homepage
    My father passed on Sunday and we were going through all the family papers. We have lots of original documents from my family during the Civil War and earlier. My sister and I were thinking of donating them to a museum, so there would be no risk of their loss should my house get damaged (there's way too many documents to fit in my fire safe).

    Before doing this, though, we were thinking of scanning/copying all the documents to keep copies for ourselves. In doing so, though, we could use some advice:

    What special steps must we take in scanning 150+ year old documents, some very yellowed and fragile?

    What is the best format in which to store them (assuming we want them easilly readble in 20+ years for our kids)?

    What is the best media upon which to store the data (again, hoping for readability in 20+ years)? (I'm thinking online storage to allow easy conversion to the media of the moment, but I still want something to stash in the safe deposit box)

    Does anyone have experience with digital preservation/resoration of archival documents? Should I just try cleaning it up in photoshop or should I find a pro to help out? Maybe I can make it a term of the donation to the museum/library, for that matter.

    Thanks in andvance for your advice.

    • I highly suggest you consult an archivist or a librarian trained in archival management. Nineteenth century paper products are notorious fragile (a result of the switch from rag pulp to acidic, unstable wood pulp). If you don't have the facilities to store these properly, donating them to a local museum or archive is a wonderful idea.

      The National Archives and Records Administration [nara.gov] has a FAQ [nara.gov]. Their advice on preserving family papers? --

      Paper preservation requires proper storage and safe handling practices. Your family documents will last longer if they are stored in a stable environment, similar to that which we find comfortable for ourselves: 60-70 degrees F; 40-50% relative humidity (RH); with clean air and good circulation. High heat and moisture accelerate the chemical processes that result in embrittlement and discoloration to the paper. Damp environments may also result in mold growth and/or be conducive to pests that might use the documents for food or nesting material. Therefore, the central part of your home provides a safer storage environment than a hot attic or damp basement.

      Light is also damaging to paper, especially that which contains high proportions of ultra violet, i.e., fluorescent and natural day light. The effects of light exposure are cumulative and irreversible; they promote chemical degradation in the paper and fade inks. It is not recommended to permanently display valuable documents for this reason. Color photocopies or photographs work well as surrogates.

    • If you really want to do it right, do it on film. Either pay someone or beg/borrow/steal a medium format camera and try to do it yourself. Film and archive quality prints will probably last longer than CDs and you can get good scans from the negatives if you want digital, too.

      I beleive libraries use uncompressed TIFF files for digital archives.

      You might find some discussions of this on photo.net

  • I'm tired of lugging around dead trees

    Call Paul Bunyan. Cause he's a lumberjack and he's okay!

  • Have you tried contacting the publishers directly? Or maybe the companies that created any of your software documentation? I know that some companies have PDFs of their manuals and other books, but don't make it well known. They don't usually offer them for free download, but if you prove you have a hard copy some companies will tell you how to get a PDF version. This works especially well for lost instruction manuals, which you can always get for free.

    One good, but old, example is Oracle. Back in the day my company had megs of PDFs of all of Oracle's documentation. There was a main index PDF with links to basically every other possible document. I don't recall Oracle leaving them open for download on the internet. We got them on CD. But it was easy to get since they new we were a customer.
  • The Plan: Take the binding of each book and cut it off. Feed into a scanner with duplex and cut-sheet feeder. Scan as a 300 DPI jpeg with compression. Then OCR them overnight. I don't expect the OCR to be perfect, just good enough to use as a searchable index.

    Right then. In 1993/4, this is what I did for a living. The company I worked for [pindar.com] did quite a lot of this, and one contract in particular sticks in my mind - the digitising of all books in the French National Library.

    No doubt the equipment we used has moved on in the intervening decade however. We used Bell & Howell [bhscanners.com] scanners fitted with automatic document shredders. Err...feeders. Yes, automatic document feeders. Not shredders at all. No. Honest.

    You see, these were high-speed scanners, and some of the books we received were qute old. Me and the other coder on the project got really quite good at doing "pit stops", or changing the rubber wheels that drove the ADF. What I'm saying is no disrespect to the scanner company - it was the quality of the paper we had to put through it that caused the hassle. Some books, like the 18th century Academie Francais records, were so thin we had to photograph them and scan the photos.

    We then scaled, OCR'd, deskewed and indexed the results on decent machines - 25Mhz 486SX, 4Mb RAM and Kofax [kofax.com] graphics cards. Everything was then tarred up to DAT.

    Hardware moves on, but I'll bet the amount of work remains the same. Do not underestimate the preparation required, and also the ammount of QA.

    Oh, and don't use JPEG. Lossy compressionon text? Use TIFF - the image processing industry standard.

    Cheers,
    Ian

  • ..an index of the book on my system. just a table with all the words and which page they appear. Pretty useless without the book, since it would be practically impossible to create the book from it, and it would be damn convienant.

  • 4DigitalBooks 900 pages/hour - or do it yourself (Score:4, Informative)

    by jukal ( 523582 ) on Thursday May 09, 2002 @05:32PM (#3493268) Journal
    I do not have any experience with their products, but the solution offered by this company [4digitalbooks.com] seems simple and functional. Their system consists of an apparatus that turns pages of your book automatically, scans, turns, scans, turns. The result you can naturally pass to OCR.

    Now, if I was to digitize all my books, I would try to create te the 4DigitalBooks kind of solution myself. The only tricky part is to find a cheap enough way to turn pages automatically [mit.edu], see also Kris Mckenzie's automatic page turner [accesswave.ca], still the best start is this document [uconn.edu] which is a proposal and overview on how to create an automatic page turner from pieces, the total cost is $459.
  • Reading over these responses I realized what it is that bugs me most about having a reference manual in PDF or some other electronic format versus having a nice book in my lap: I don't have the screen real estate for both a document reader and whatever app it is I'm using the reference for.

    The endless jumping between windows gets old real fast, especially if I need to copy a code snippet out of a document (like a PDF) that won't let me select & copy text.

    But if I had a second monitor right there at eye level, I could just open up the reference doc there. No more switching between windows, and no more neck strain from constantly looking down at a book in my lap and then up at the screen.

  • Funny You should ask. (Score:3, Informative)

    by Fapestniegd ( 34586 ) <[james] [at] [jameswhite.org]> on Thursday May 09, 2002 @05:36PM (#3493298) Homepage
    My current setup consisits of:
    4 HP scanners with ADF ~$150 ea. (eBay)
    4 Sparc LXs from a property contol auction $50
    one flatbed scanner for covers and bad scans. $50 (eBay again)
    Barebones System/w scsi from Compgeeks $80

    (NFS server), An Amtren Device [amtren.com](courtesy of the office) and away you go. I've found the best way to cut off the binders is to use a box cutter and to use your previous cuts as a guide. Several shell scripts to scan various types of books. It's amazing the page numbering schemes some publisers use. With this setup I can scan approximately 2-3 college textbooks 1000 pgs.(grayscale) or 1 color in an 10 hour period. (including checking for bad scans, sane ain't perfect, so you better check em) also jpg isn't very good for OCR, I store as png, and convert a second set to jpg for web viewing. OCR under linux isn't quite there yet (unless you want to pay through the nose) So I am Archiving the pngs to CD until it is. This also allows me to regenerate the jpgs if I lose a webserver disk. Add a nifty little IMageMagick web viewer and viola! eBookshelf! Oh and a NSM CD changer is nice too get to the CDs nearline.You can pick these up on ebay for $200-$400

  • I've done this (Score:4, Insightful)

    by brad3378 ( 155304 ) on Thursday May 09, 2002 @06:01PM (#3493451)
    To do it, I purchased a used HP scanner with a 50 page Automatic Document Feeder (Search for ADF on Ebay).

    I started with the easiest books. - Books that could be removed from the binding. Scans go smoothly with the ADF, but it is not as easy as you might think. I find that I spend most of my time naming the files because the default naming comvention is *01.jpg , *02.jpg , *03.jpg, etc.

    It is a problem for two reasons:

    most of my books are double sided.
    My HP scanning software for windows does not let me name files with a 2,4,6,8 or 1,3,5,7 format.

    If books contain more pages than the ADF holds, The first page scanned will still be named page 1.

    If I knew a little perl, I'd write a script to rename the files between scan batches.

    For scanning full bound textbooks, there are two main problems:

    Scanning the side of the page along the binding requires carefully holding downward pressure on the book to keep it near the scanner glass.

    You cannot scan the book using ADF, so you should expect to spend A LOT of time scanning.

    Do not even consider manual scanning hundreds of pages with a parallel port scanner. WAY WAY too slow. USB scanners are cheap now, and will usually scan as fast as the scanner mechanism can move (assuming black & White scans).

    Lastly, be realistic.
    Know how much time you'll need to invest.
    Rule of thumb: If you need to scan manually, expect to scan about 200 pages per hour at top speed. Is it worth investing six hours to scan that 1200 page book of yours? If money allows, I'd suggest purchasing a second book that you can afford to destroy. Cut the binding off with something like a jigsaw, then insert the pages into an ADF scanner. Hope this helps somebody.

  • You *need* to be aware of OpenDJVu (Score:5, Interesting)

    by Effugas ( 2378 ) on Thursday May 09, 2002 @06:35PM (#3493625) Homepage
    Run, don't walk, to http://djvu.research.att.com/home.html . DJVu is a image-based competitor to PDF that is a feat of beautiful engineering -- 300DPI scans break down to about 10-30K a page, the viewer is about an order of magnitude faster than PDF, the format cleanly supports separate encoding of page texture/graphics vs. page text, there's significant amounts of open source for it, and more.

    It's truly a brilliant format. Go check it out.

    Yours Truly,

    Dan Kaminsky
    DoxPara Research
    http://www.doxpara.com

  • You are insane (Score:4, Insightful)

    by labradore ( 26729 ) on Thursday May 09, 2002 @09:09PM (#3494236)
    Ask yourself this question:
    What is the oldest file that I have?
    and ask:
    What is the oldest useful file that I have?
    For most people their papers and books are much older than the data they keep and the paper version is always available and easy to read.

    You are much more likely to lose or corrupt your data if it is on a disk or a tape than if it is in a book. Your electronic version is going to be of much lesser quality than the books you had and you will have a lot of "adventures" getting your ebooks to be as easy to read as your paper books. What happens to your portable ebook when your reader runs out of batteries? Ebooks have failed because ... THEY SUCK. Let us all know how much time you wasted tweaking your ebook setup and worrying about how to make them sustainable. Also, please tell us when you go back to the store and buy new "dead trees" copies of the ones you destroyed.

Slashdot Top Deals

"Buy land. They've stopped making it." -- Mark Twain

Close