tag:blogger.com,1999:blog-3417408127993138882.post8384642247115047928..comments2024-03-27T02:59:19.222+00:00Comments on Colnect, Connecting Collectors: When Web Crawlers AttackUnknownnoreply@blogger.comBlogger2125tag:blogger.com,1999:blog-3417408127993138882.post-42045810918177023452009-04-13T01:03:00.000+01:002009-04-13T01:03:00.000+01:00This is not much different in nature than using ro...This is not much different in nature than using robots.txt and in fact more complicated to perform. Too bad regular expressions are not supported in robots.txtcolnectorhttps://www.blogger.com/profile/00454900325413396837noreply@blogger.comtag:blogger.com,1999:blog-3417408127993138882.post-44579560609467842862009-04-12T15:54:00.000+01:002009-04-12T15:54:00.000+01:00You can use google's webmaster tools and remove th...You can use google's webmaster tools and remove th bad url from their index and crawlersOfer Waldhttps://www.blogger.com/profile/01717295623900006044noreply@blogger.com