Please create an account to participate in the Slashdot moderation system

 



Forgot your password?
typodupeerror
Get HideMyAss! VPN, PC Mag's Top 10 VPNs of 2016 for 55% off for a Limited Time ×
Google

Submission + - Google News gets a separate web crawler and index

markkezner writes: In response to complaints by Rupert Murdoch's News Corp. and others, Google has given Google News its own web crawler. The service had previously relied on the same web crawler that is used to index pages for regular web search. This change allows publishers to use robots.txt to finely control how Google News indexes their content without affecting how it is indexed for normal Google web searches. Before this change occurred, publishers could have contacted Google to specifically de-index content from Google News.
This discussion was created for logged-in users only, but now has been archived. No new comments can be posted.

Google News gets a separate web crawler and index

Comments Filter:

Congratulations! You are the one-millionth user to log into our system. If there's anything special we can do for you, anything at all, don't hesitate to ask!

Working...