New answers tagged robots.txt
3
From the Google Search Console help docs:
If you want Google to exclude the image-name-here.png that appears
on your site at www.example.com/images/image-name-here.png, add the
following to your robots.txt file:
User-agent: Googlebot-Image
Disallow: /images/image-name-here.png
The next time Google crawls your site, it will see this directive ...
3
It is already answered in Stackoverflow, you just need to define the useragent name there. Here we only specify Googlebot, so that rule is not applied to other search engines.
user-agent: Googlebot
Allow: /$
Disallow: /
As Google said (at the end of page). Use $ when you want to Match URLs that end with a specific string. Here /$ will say allowed to index ...
Top 50 recent answers are included
