A large-scale study of robots.txt.

Yang Sun, Ziming Zhuang, C. Lee Giles

Browse the full WWW paper archive.