Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
2
votes
2answers
108 views
Will duplicate international (i18n) content hinder SEO rankings?
Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
0
votes
2answers
23 views
Google Webmaster Tools robots test not working
Within Webmaster Tools I have supplied my test content:
User-agent: *Disallow:/admin/
Disallow: /tag/
When I specify the URL to test against, for example:
http://www.site.com/tag/
It gives me ...
1
vote
1answer
19 views
To allow or disallow alphabetical content directory pages?
We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
1
vote
1answer
286 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
0
votes
1answer
43 views
Addon domains and SEO
I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page.
Can I tell bots not to index the main domain WITHOUT ...
0
votes
0answers
160 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
0
votes
0answers
50 views
Need to know how to configure robots.txt for an age gate/age verification website
is it the normal allow all in the file? I need to make sure the crawlers can get through the gate. Thanks!
