Malicious Web Crawling

SiteAdvisor indeed cashed for evaluating the maliciosness of the web, and New Zealand feels that nation wide google hacking initiatives are a more feasible solution to the problem of google hacking, compared to the Catawba County Schools Board of Education who blamed Google for indexing student test scores & social security numbers. It's like having a just-moved, 25/30 years old neighbors next to your place, who didn't know you have thermal movement detection equipment and parabolic microphones, in order to seal the house by using robots.txt, or assigning the necessary permissions on the web server asap.

Tip to the Board of Education, don't bother Google but take care of the problem on your own, immediately, through Google's automatic URL removal system, by first "inserting the appropriate meta tags into the page's HTML code. Doing this and submitting via the automatic URL removal system will cause a temporary, 180-day removal of these pages from the Google index, regardless of whether you remove the robots.txt file or meta tags after processing your request."

Going back to the idea of malicious web crawling, the best "what if" analysis comes from Michal Zalewski, back in 2001's Phrack issue article on "The Rise of the Robots" -- nice starting quote! It tries to emphasize that "Others - Internet workers - hundreds of never sleeping, endlessly browsing information crawlers, intelligent agents, search engines... They come to pick this information, and - unknowingly - to attack victims. You can stop one of them, but can't stop them all. You can find out what their orders are, but you can't guess what these orders will be tomorrow, hidden somewhere in the abyss of not yet explored cyberspace. Your private army, close at hand, picking orders you left for them on their way. You exploit them without having to compromise them. They do what they are designed for, and they do their best to accomplish it. Welcome to the new reality, where our A.I. machines can rise against us."

That's a far more serious security issue to keep an eye on, instead of Google's crawlers eating your web site for breakfast.