What Desktop Search Engine For a Shared Volume? 232
kriston writes 'Searching data on a shared volume is tedious. If I try to use a Windows desktop search engine on a volume with hundreds of gigabytes the indexing process takes days and the search results are slow and unsatisfying. I'm thinking of an agent that runs on the server that regularly indexes and talks to the desktop machines running the search interface. How do you integrate your desktop search application with your remote file server without forcing each desktop to index the hundred gigabyte volume on its own?'
Call the NSA (Score:5, Funny)
They already have it indexed for you.
Re: (Score:2, Informative)
Or Google. They already wire-tap your voice mails, index them and post the results for public consumption [slashdot.org].
wow! (Score:5, Funny)
You've stumped Slashdot. Bravo!
Re: (Score:2, Informative)
I was wondering if "spotlight on OS X does that just fine" was too trollish... But, too late, I said it, and it's true, can't be trollish if it's true can it?
Re: (Score:2)
That's what we use... server version indexes servers, SAN volumes etc and makes them searchable from each desktop ;)
Re: (Score:2)
and it's faaaaast
It searches by content as well as by filename too for those that didn't know. Incredibly useful.
Spotlight (Score:2)
... seconded
Re: (Score:2)
I believe that the newer versions of OSX server will perform indexing on the volume so the clients do not have to. So searches are fast and the problems you described are gone.
Ok, just went over to apple.com to find the info page. It is here [apple.com].
Google Enterprise Search (Score:5, Interesting)
Everything (Score:2, Informative)
Re: (Score:2)
Well it sounds like it is. I should note that Everything only indexes filenames, so if you want to index file CONTENTS you're out of luck (that sort of thing is GOING to take a long time anyway, since you have to read every file on disk that the indexer knows how to parse, so "quicker" could well translate to "less complete search index").
But if you don't care about indexing contents then Everything should work fine for you.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
...it beats the native Outlook search speed handily...
Honestly? I beat native Outlook search on a fairly regular basis. I cried tears of happiness and joy when Lookout hit the scene. I choked up a bit when Microsoft bought them out, but I recovered when it looked like they integrated the engine into the engine Microsoft uses to do Desktop search on XP.
Seriously, ask slashdot? (Score:2)
http://en.wikipedia.org/wiki/List_of_enterprise_search_vendors [wikipedia.org]
Re: (Score:3, Insightful)
Maybe search, but didn't know what to search for.
If this is a guy who's used to doing home and small business support, with a handful of machines at best, and kind of got thrown into the deep end by management because he's "good with them thar cmpooterz" then he may not be thinking "enterprise" search.
After all, why would anybody need a search engine to find a starship?
Probably searched for "shared drive search engine" or something like that.
solution to hundreds of terabytes of docs (Score:3, Interesting)
how about using a program like Documentum? We generate several thousand technical documents and drawing a month, and use it for all our document management needs.
Re: (Score:2)
I assume you mean the Verity part. Documentum can be a bit pricy for the whole suite of DiskXtender, AppXtender, and Verity, but it does work great for this purpose.
A couple of options (Score:4, Informative)
NO! Try Alfresco (Score:5, Informative)
Re:NO! Try Alfresco (Score:5, Informative)
You mean the Document Management Alfresco [alfresco.com] and not the CMS software. The Community Edition is free but unsupported, and the Enterprise edition has a free 30 day trial. It looks like it won a government award for document management which is rare for open source document management software.
Re: (Score:2)
You could use Microsoft Enterprise Search Server Express [microsoft.com] which is free (if you have a Windows Server license laying around). It's the same search engine as MOSS without the CMS functionality and it can crawl just about everything either natively or with connectors. You can use MSSQL Express as the database engine which is also free.
Or you could go completely open source with Apache SOLR [apache.org], though I hear it's so featureful that it's very difficult to install and configure.
Re: (Score:3, Informative)
SQLServer is per CAL even though the app is a web app.
Re: (Score:3, Informative)
Re: (Score:3, Insightful)
Here's a few options you might want to consider: 1) Use Office SharePoint Server 2007 to index the share
First, MOSS isn't free.
Second, have you ever actually tried using SharePoint 2007 text search feature? I dunno what it indexes, but finding anything in that afterwards is about as convenient as searching for a needle in a haystack.
There have been claims of some huge improvements in the upcoming SP2010, which is not surprising in light of Bing, but that's not released yet.
Re:A couple of options (Score:5, Funny)
You are on /. and actually recommending an upgrade to Vista?
Brave man.
Re: (Score:3, Interesting)
Re: (Score:3, Informative)
Re: (Score:2)
Coming from a company that made heavy use of sharepoint (2003 and 2007 - to be fair 2007 is aeons ahead of 2003 but inherited out projects' terrible design) I can heartily say that the index tool sucks, and the relevance engine is piss poor; no one could ever, ever find anything on the company intranet, and one of the helldesk's most idiotic tasks was keeping a bookmark of all the important sub-sites and documents on the intranet... and all because devs fell for the idiocy of the "don't bother organising it
Everything (Search Engine) (Score:2, Informative)
Federated Search (Score:5, Informative)
MS does have a solution, it's called Windows Federated Search. Windows 7 with 2008R2 has it .. there might be a way to do with Windows Desktop Search 4.0. Here's some info on it - http://geekswithblogs.net/sdorman/archive/2009/05/14/windows-7-federated-search.aspx
Re: (Score:2)
Re: (Score:2)
Actually it's quite relevant. Windows 7 can federate queries to a SharePoint or Search Server index using OpenSearch.
Also, Windows Vista and Win7 (and even XP with WS4 to some extent) can query remote Windows Search indexes. I use this functionality along with my Windows Home Server (running WS4) for my personal needs.
Sharepoint Services (Score:2)
Enterprise Content Management with Alfresco (Score:5, Informative)
Yes, Google's Search Appliance (GSA) could be used, I have seen it used with limited success. The main problem was how to respect access control on documents: either you index them or you don't, and if you index them with GSA, sensitive data may show up in search results. Also, we had a lot of trouble "taming" GSA: it would regularly take down servers that were dimensioned for light loads.
I would suggest using Alfresco http://www.alfresco.com/ [alfresco.com] as a CIFS (Common Internet File System) or WebDav store for all those documents. This would give you the simplicity of a shared folder and the opportunity to enrich the documents with searchable metadata such as tags, etc. Each folder (or any item, in fact) could have the correct access control that would be respected by the search engine, Lucene. http://lucene.apache.org/java/docs/ [apache.org]
Alfresco comes in both Enterprise and Community Edition, it's very easy to try out -- even our non-techie project manager could install it on his PC within 10 minutes. Try that with Documentum, FileNet or IBM DB2 Content Manager!
Re: (Score:2)
There is some stuff you can do with GSA to try to implement document security - you can set up separate collections/indexes (I forget the GSA term) for different parts of your document repository, then restrict the search results to specific indexes based on the logged-in user's credentials. (That's assuming you roll your own interface)
Note that my one use of GSA was a couple of years ago, and we had an extremely simple security model with only 2 user types - one got access to everything, one got access to
Re: (Score:2)
The GSA works perfectly well with many security standards (AD, NTLM, Kerberos, SAML, forms, cookies) and it obscures secure search results from users who do not have read permissions to those documents. It's probably easier to configure the GSA's security settings than with any other enterprise search platform.
Furthermore, the GSA has a self-throttling feature called Host Load Scheduling which allows you to limit the number of connections opened per second. If that's not sufficient you can throttle the VL
Mirror it. (Score:5, Funny)
Re: (Score:2)
So you would rsync hundreds of gig to a local disk to index it? I think I would rethink that strategy.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Bingo, I guess Palegray has a much larger budget than most of us when it comes to storage. Afterall why have a single copy when you can have fifty.
Well, at least that way you don't have to screw with tape drives......
Re:Mirror it. (Score:5, Insightful)
Have you ever actually used rsyng on a decent sized file set? Determining the changed file set requires significant disk activity.
It's a certain win when compared to just blindly transferring everything. But if you think that rsyncing 20 changed files in a 100 file working set is the same as rsyncing 20 changed files out of a 2,000,000 file working set you are very very wrong.
Completely aside from the absolute insanity of suggesting that you replicate the full contents of the fileserver to every desktop, which has been covered by others.
Re: (Score:2, Funny)
Re: (Score:2)
Re: (Score:2)
Yeah, but what if your file system is 70TB with 20 million files?
Re: (Score:2)
It seems that the parent wants to merge a remote index with his desktop search so that he doesn't have to do this. Also, wouldn't giving each desktop its own copy of the data defeat the purpose of having a shared server?
Re: (Score:2)
Re: (Score:2)
Disk space is cheap when you're outfitting a single server. Outfitting even ten workstations with the same amount of disk can become quite expensive.
Re: (Score:2)
Trust me, I've learned from experience that the local disk space works out to be much cheaper for this sort of thing.
Re: (Score:2)
How about Spotlight? That works on shared volumes. (Score:4, Informative)
*ducks*
Re:How about Spotlight? That works on shared volum (Score:4, Informative)
Yeah, my thought exactly? I wasn't aware that it was a problem searching hundreds of gigabytes on shared volumes. We have a couple of terabytes shared by our Mac servers and I don't think I've had search times longer than ten seconds over a couple of million files.. MS Office files, PDFs, movies, audio, pictures, photographs, text, HTML, source code.. all indexed with metadata and contents.
Even the days before Spotlight, using AppleShare IP Servers in the 90s, finding stuff on the servers was never an issue. It has always been so fast that I have never even reflected over that it was fast. Maybe I should use some other operating system once in a while to experience what the majority experiences. Or not.. I'd rather stay care free and productive.
Don't call me when you figure this out.
Re: (Score:2, Insightful)
Spotlight is the obvious answer if you have OS X. Not everybody in the world is lucky enough to be in that
position, most are stuck on one of the inferior platforms. Your rubbing it in, is not helping it just
alienates people who already have been through enough and have it tough.
Re: (Score:3, Funny)
I've tried ducks, but they tend to nibble the occasional one or zero, and they leave an awful mess on the platters when they poop. Try Spotlight instead -- not as cute, but easier on the data, hardware, and the nose.
Re: (Score:3, Interesting)
Except then you have another terrible search solution which isn't meant for the amount of data you'd find on a large server. Worse, you have an operating system that is terrible as a server solution.
On the other hand, you could just use a unix/linux distro of your choice, and beagle (http://beagle-project.org) - which is meant for indexing large amounts of data and has many clients some of which can remotely access it.
Use Microsoft Indexing Service (Score:2)
One way is to set up Microsoft Indexing Service on the server with the shared drive. The MSC console app provides a search capability and one can also use the Indexing Service SDK for client apps.
Hmm (Score:2)
Re: (Score:2)
That's not really a "desktop search engine" anymore, but I agree with your suggestion. Desktop search is not the right tool to index shared content, lest you get a department full of PCs constantly indexing the same repository over slow ethernet wires. Microsoft Enterprise Search Server lets you separate the indexing and serving servers, but unfortunately the free Express version doesn't support this.
Use MSS 2008 Express, SharePoint, FAST (Score:3, Interesting)
Use Microsoft Search Server 2008 Express...its free, all you need is a free server box. Also Check out SharePoint Search and FAST enterprise search.
http://www.microsoft.com/enterprisesearch [microsoft.com]
Two words... (Score:2)
Google appliance...
Lucene is a great foundation for this (Score:2)
So I think I'd start by looking here [apache.org].
Re: (Score:2)
Cross-Platform (Score:2)
Re: (Score:2)
You want an enterprise search service. Microsoft and Google both make web-driven search solutions which work on any end user platform including mobile. There are many other companies that have web-based search services as well but those are the two goliaths.
Windows 7? (Score:2)
PC Docs Docs Open (Score:2)
Docs Open is a commercial document management system [pcdocs.com] but right now their web page doesn't seem to be working. We used it at a law firm I worked at. IIRC it was able to search through the billions of documents that the 300+ lawyers used in their cases.
You don't (Score:3, Interesting)
Your problem is obvious (Score:2)
Try Earth (Score:2, Interesting)
Re: (Score:2)
Earth requires a single database of search results. This is stupid because it creates an unnecessary single point of failure. When I execute a search, it should query all the remote machines for their own databases. This creates a few more ACKs but otherwise puts little more load on the system. It would be nice to have the OPTION to store the catalogs on a repository, but being forced to means that I need literally another server on my network; I don't have anything running 24/7 right now that is capable of
Desktop search is not the way to go (Score:2)
Seriously. You're probably going to want a separate server(s) for this job. You didn't specifiy what you're indexing, how often, or where, however I'll make some assumptions and point you towards an enterprise search appliance or product. Many will probably point you to Google Enterprise Search. I've worked with the search functionality withing Microsoft Sharepoint 2007, and it's (ostensibly) free spin-off, Microsoft Search Server. Again, you'll probably need to dedicate some hardware to this. In addition t
Share searching. (Score:2)
I wrote a web site/spider to do this for the whole network at uni. It was beautiful C++ all the way. After I left some silly CS people rewrote it in Python/PHP (ugh) here: http://code.google.com/p/trufflepig/ [google.com]
Portfolio Server (Digital Asset Management) (Score:2, Informative)
(Disclaimer: I work for Extensis)
Portfolio Server can continuously index files on SMB/CIFS (and AFP) volumes using a feature called "AutoSync". Web and Desktop (Windows/Mac) clients then search by folder name, file name, document text, or other metadata. Indexing and thumbnail creation takes place on the server, so clients are relieved of any cataloging workload and metadata is centralized.
http://www.extensis.com/en/products/portfolioserver9/overview.jsp
Use Windows Indexing Service (Score:2)
Combine that with the next article from that site and you have a solution: http://www.windowsnetworking.com/articles_tutorials/Making-Windows-Server-2003-Indexing-Service-Useful.html [windowsnetworking.com]
This article s
Re:Use Windows Indexing Service (Score:4, Informative)
For indexing files, you're better off using Windows Search 4, a free download for Windows Server 2003. The old content indexing service is deprecated and a much older technology. It's useful in some particular scenarios but for a smaller (100,000 - 250,000 items*) corpus of file content, WS4 will work much better. And for larger repositories, SharePoint and Microsoft Search Server are almost always better options.
* = Server 2008 R2 / Win7 has a newer version of the Windows Search indexer that scales better to even larger corpuses.
Try Xapian Omega (Score:2)
I had this same problem not too long ago - we have a shared documentation tree with tens of thousands of documents that I wanted to index. I tried dozens of search engines in my spare time, most of which were just horrible (Beagle), were a nightmare to install for someone like me who's not a full or even part time admin (Apache SOLR), wouldn't allow cross platform access (lots of Windows ones, obviously), store a complete separate copy of every document (Alfresco, which didn't seem to have an option to ) an
Re: (Score:2)
Re: (Score:2)
Good question. I'm doing it for my project team, being the lead software guy. The IT dept. is a small team and kept busy with other stuff - but they gave us hardware for a team Linux server when I asked for it and let me admin it, so I'm just grateful that they're helpful when so many would be obstructionist.
Rambling: I've also installed a wiki, which is getting rave reviews from the people using it; hoping to make that popular enough that IT will /want/ to take it over.
There's also an alternative to GSA and Sharepoint (Score:2)
From IBM and Yahoo called OmniFind. It runs on a desktop or server and can index multiple shares... and the basic version is free but offers a lot of functionality.
Although if your business is booming, a GSA is freakin' sweet.
Depends on the server, doesn't it? (Score:2)
Microsoft / Windows Search options (Score:2)
Microsoft has a few solutions you can consider depending on your specific needs.
With Windows XP/2003, Vista/2008, or Windows 7 - you can install Windows Search 4 (not necessary on Win7, but recommended for Vista) on the server side to index the content, and then if you have WS4 (or Win7) on the client, it will automatically query the remote index when you perform searches against that file share.
Alternatively, if you run the free Microsoft Search Server (the Express version is free) which is based on ShareP
Pfff... (Score:2)
Slackware is at version 13 which makes it much more advanced than a version 7.
Read at Slackware got to version 13 so quickly at this link:
http://en.wikipedia.org/wiki/Slackware [wikipedia.org] ;-)))
Access Rights? (Score:2)
I had a look at some solutions last year, and ran into one hell of a road block; most solutions I had a look at presume that all the information you're indexing should be searchable and/or available to anyone that can reach your search tool's client.
Has anyone had experience with something that will search the indexes for items based on your credentials? (Meaning that if you're not in accounting you can't get results for that data set)
Looking at the wrong problem? (Score:2)
Ever consider hiring a librarian? I've worked at 3 small companies that had one and they were far more profitable than most of their competition because there was someone in charge of organizing the data.
Re: (Score:2)
Re: (Score:2)
Hmmm... locate doesn't allow you to search within files. What about using rgrep or grep -r ?
find is great too (but slower on the first run before results get cached by the kernel, if you have enough spare memory) when you need to know which files have been modified in a given period of time, which files take more room on the disk, etc..
I usually disable locate for security reasons, at least use slocate ! ;-)
So I'd say I use find and rgrep ;-)
Re: (Score:2)
You could make a web interface to locate.
(Only searches files names.)
Re: (Score:2)
Well, that runs into the problem the OP has discussed. If the data is present as a network share, it'd take slocate forever to index the data on the remote server. Basically, he or she wants a way to run slocate once on the server and have that index file be merged with all of the individual desktops. That way, each desktop wouldn't have to go through the effort of duplicating work.
Re: (Score:2)
So use ssh and run slocate on the server, or share out the slocate.db file.
Re: (Score:2)
Yeah, that could work, but I don't think it'd be as seamless as the OP wants. The user would still have to select which db file to use. Still, its a solution.
Re: (Score:2)
or here is a real smart idea:
In each users path place a locate_on_server.sh script that just runs "locate -d $PATHTODBONSERVER".
-d can take multiple database filename arguments, so you could have one locate_on_server that searches all your fileservers.
Re: (Score:2)
Heck, you could skip the script and just alias locate
But you still run into the problem that it runs from the command line and the database is byte-order dependent.
Re: (Score:2)
Re: (Score:2)
And then you sit and wait for ages for find to finish, and then you realise that it only searches in the file names, and not the contents of the file. Of course, what I do, is ssh in, and then use mdfind, but yeh, find doesn't cut it on multi-terrabyte volumes, and especially not when you want to search on more than just the name.
Re: (Score:2)
Wrong! The "grep" part of that command searches the contents of the files.
But if you think you can get away with just grep on large amounts of data, you really ought to learn something about how indexing works and how much faster it can make your searches.
Re: (Score:3, Informative)
FYI, GNU find has xargs built in these days:
find -name '*.php*' -exec grep func {} +
the + instead of ';' makes it collect up multiple arguments to grep
like xargs instead of the traditional find -exec behaviour which is like xargs -n1. I use -exec {} + all the time, because it's less typing, and safe with
filenames with punctuation or whitespace, so you don't have to type -print0 | xargs -0 either. (BTW, if you have a list of filenames that you processes with something line oriented, you can use xargs -d'\n
Re: (Score:2)
ssh user@fileserver 'cd /shared/myproject;wcfind . -name _\* -prune -o -type f -! -name \*.o -a -! -name \*.a -a -! -name \*.so\* -a -! -name \*.d -print | while read a; do ( file "$a" | fgrep -q 'text' ) &>/dev/null && fgrep -H "" "$a"; done | gzip -c9' >~/index/myproject.gz
Re: (Score:2)
Oops, forgot to add
zfgrep goddamnfunctionname ~/index/myproject.gz|less
That's actually quite quick, for a couple MLoC.
Re: (Score:3, Funny)