Malicious Web Crawling

SiteAdvisor indeed cashed for evaluating the maliciosness of the web, and New Zealand feels that nation wide google hacking initiatives are a more feasible solution to the problem of google hacking, compared to the Catawba County Schools Board of Education who blamed Google for indexing student test scores & social security numbers. It's like having a just-moved, 25/30 years old neighbors next to your place, who didn't know you have thermal movement detection equipment and parabolic microphones, in order to seal the house by using robots.txt, or assigning the necessary permissions on the web server asap.

Tip to the Board of Education, don't bother Google but take care of the problem on your own, immediately, through Google's automatic URL removal system, by first "inserting the appropriate meta tags into the page's HTML code. Doing this and submitting via the automatic URL removal system will cause a temporary, 180-day removal of these pages from the Google index, regardless of whether you remove the robots.txt file or meta tags after processing your request."

Going back to the idea of malicious web crawling, the best "what if" analysis comes from Michal Zalewski, back in 2001's Phrack issue article on "The Rise of the Robots" -- nice starting quote! It tries to emphasize that "Others - Internet workers - hundreds of never sleeping, endlessly browsing information crawlers, intelligent agents, search engines... They come to pick this information, and - unknowingly - to attack victims. You can stop one of them, but can't stop them all. You can find out what their orders are, but you can't guess what these orders will be tomorrow, hidden somewhere in the abyss of not yet explored cyberspace. Your private army, close at hand, picking orders you left for them on their way. You exploit them without having to compromise them. They do what they are designed for, and they do their best to accomplish it. Welcome to the new reality, where our A.I. machines can rise against us."

That's a far more serious security issue to keep an eye on, instead of Google's crawlers eating your web site for breakfast. Continue reading →

Nation Wide Google Hacking Initiative

The idea of doing reconnaissance for the purpose of pen testing or malicious activity through google hacking, has already reached levels of automation -- the problem is how the threat gets often neglected by those that actually suffer from a breach later on. I came across to an article pointing out that :



"Anyone who wants to hack into sensitive information on New Zealand internet sites might be pleased to know it can be as easy as typing keywords into a Google search. Researchers at Massey University’s Albany campus say the country’s websites are more vulnerable to "Google hacking" than anywhere else in the world. University Information and Mathematical Sciences Institute senior lecturer Dr Ellen Rose and graduate student Natalia Nehring recently completed a study into the topic."



Not exactly a type of cyberterrorism exercise such as the most recent DigitalStorm, but it's logical to conclude that if someone takes the time and effort to data mine the web, localize the attack like in this case, a lot will be revealed. In a recent article, CSOonline goes in-depth into the security implications posed by Google. I once had a chat with Johnny Long on many topics, among the "few", of course, was google hacking. He made a good point on saying that it's whatever you actually do with the results that matters most, and how diverse is the threat -- by googling your lights off for instance.


What you should keep in mind is that it isn't Google to blame, the way "Improving the Security of Your Site by Breaking Into it" provoked awareness, and not damage. Think the problem isn't big of a shot -- gather some intelligence by yourself through the Google Hack Honeypot project. Continue reading →