Google uses several user-agents. Using the bot name, you can block access or track any user-agent easily.

Googlebot: crawl pages from web index and news index
Googlebot-Mobile: crawls pages for mobile index
Googlebot-Image: crawls pages for image index
Mediapartners-Google: crawls pages to determine AdSense content. Only used to crawl your site if AdSense ads are displayed on your site
Adsbot-Google: [...]

Continue reading...

Ever wanted to know when Google crawls your website? Here is a simple PHP script which emails you whenever GoogleBot crawls your site.
<?php
if(strpos($_SERVER['HTTP_USER_AGENT'], 'Googlebot') !== false)
{
$email = 'you@domain.com';
mail($email,'Googlebot Alert', 'Googlebot has crawled your page: '.$_SERVER['REQUEST_URI']);
}
?>

Continue reading...

Search engines use web crawler (spider/robot) which browses WWW in an automated manner to provide up-to-date data. But what if one doesn’t want to get listed? Yes, “robots.txt” comes into play when you have to set your own trend whether to allow bots or prevent your site from being crawled!
Place robots.txt in your root directory [...]

Continue reading...