Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
2answers
232 views

Will duplicate international (i18n) content hinder SEO rankings?

Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
0
votes
1answer
69 views

Clicks counting and crawler bots

I am currently running a small affiliate-program for Facebook users. We use an auto-poster to publish links to fan pages. Every hit is stored in our database and we have included a 24 hour reload ...
0
votes
1answer
102 views

Addon domains and SEO

I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page. Can I tell bots not to index the main domain WITHOUT ...
-1
votes
1answer
213 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...
0
votes
0answers
58 views

Bingbot requesting incorrect robots.txt path

A .NET site I am currently working on is being hit rather frequently by the Bingbot on the path: /:80/robots.txt This is obviously throwing the following exception: System.Web.HttpException: A ...
0
votes
0answers
152 views

Strange Google crawler 404s on Magento store involving “image” and “id” in URL

We're hit with over 10,000 404s on Google Webmaster all of a sudden. We've been getting them a few now and then, but not even close to this magnitude. Upon examining the items in this list of errors, ...
0
votes
0answers
289 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
0
votes
0answers
96 views

Disallow age gate/verification page in robots.txt?

Should I disallow the age gate page in robots.txt? I need to make sure the crawlers can get through the gate.