The desire for greater control over how search engines index and display Web sites is driving an effort launched yesterday by leading news organizations and other publishers to revise a 13-year-old technology for restricting access.Currently, Google, Yahoo and other top search companies voluntarily respect a Web site’s wishes as declared in a text file known as robots.txt, which a search engine’s indexing software, called a crawler, knows to look for on a site.But as search engines expanded to offer services for displaying news and scanning printed books, news organizations and book publishers began to complain.News publishers said that Google was posting their news summaries, headlines and photos without permission. Google claimed that “fair use” provisions of copyright laws applied, though it eventually settled a lawsuit with Agence France-Presse and agreed to pay the Associated Press without a lawsuit filed. Financial terms haven’t been disclosed.
http://www.washingtonpost.com/wp-dyn/content/article/2007/11/29/AR2007112902207.html