Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
2
votes
2answers
88 views
Will duplicate international (i18n) content hinder SEO rankings?
Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
1
vote
1answer
46 views
GoogleBot can't reach my site
For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
1
vote
1answer
285 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
1
vote
1answer
405 views
Robots.txt practices with .htaccess redirections (inherits)
I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place.
We have a hosting account that enacts primary and add-on domains. All of our domains ...
0
votes
1answer
39 views
Addon domains and SEO
I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page.
Can I tell bots not to index the main domain WITHOUT ...
0
votes
0answers
143 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
0
votes
0answers
48 views
Need to know how to configure robots.txt for an age gate/age verification website
is it the normal allow all in the file? I need to make sure the crawlers can get through the gate. Thanks!
