Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
1answer
106 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
2
votes
1answer
131 views

Robots.txt to block a parameter instead of a directory

It's my understanding that URLs are of the format example.com/something/somethingelse. It's my understanding that parameters follow the URL with a question mark example.com?l=fr_FR My CMS website ...
1
vote
1answer
58 views

Removed pages from website - do I need to tell Google or will they fall off in time anyway?

This is a follow up question from here: Duplicate title tags due to translations Since asking the question I have removed the duplicated content that was being generated by language/translations on ...
1
vote
1answer
33 views

Cascading matches in robots.txt

I was wondering if it's possible to let bots match two groups of User-Agent: blocks. I have this block of robots.txt User-Agent: AhrefsBot # https://ahrefs.com/robot/ Crawl-Delay: 5 User-agent: * ...
1
vote
2answers
129 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
2
votes
2answers
112 views

robots.txt: block all webpages except a few number of webpages?

I have a few doubts regarding robots.txt. Say, my domain is stackoverflow.com, A) Will the code below do the following for all the crawlers? User-agent: * Disallow: / Allow: /$ Allow: /a/$ Allow: ...
2
votes
2answers
883 views

Correct usage of robots.txt file (“Googlebot-Image” and “Mediapartners-Google”)

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress ...
0
votes
1answer
148 views

Sitemaps: rel=“alternate” hreflang=“x” naming

I'm creating a Sitemap for rel="alternate" hreflang="x", but no where in Google's guidelines do they mention how it should be named. Also can we reference it separately in the robots.txt file, or does ...
1
vote
3answers
104 views

robots.txt file Question

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google
-1
votes
1answer
230 views

Unreachable robots.txt error

I am getting Unreachable robots.txt error whenever I use fetch as Google. I know, on this site there is other queations related my issue. But I also use Google Page Speed test, site is working ...
2
votes
1answer
233 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
0
votes
2answers
2k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
0
votes
1answer
158 views

How to prevent webpages translated with Google Translate Plugin from being indexed?

I've been reading the information here https://support.google.com/translate/#2534535 about the Google Translate plugin. At first I thought that if I used the plugin my site wouldn't be indexed, but ...
3
votes
1answer
795 views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...
0
votes
1answer
50 views

Do I need disallow other websites directory in robots.txt?

I have a hosting which has several websites in www folder. I set up a WordPress website in root directory for main domain. Do I need disallow other websites directory in robots.txt? I think these ...
0
votes
1answer
93 views

How to block certain URL from robots.txt file? [closed]

This is simple question but also I have a confusion. Below is the code which I wrote in my robots.txt: User-Agent: * Disallow: /phpmyadmin/ Disallow: /login/ Disallow: /newsletter/ Disallow: ...
4
votes
5answers
372 views

With Robots.txt disallow all, why was my site still getting traffic?

In failing to check my robots.txt on a community site, it had: User-agent: * Disallow: / I only clued into this because of a message in my AdSense account saying that five URL's couldn't be ...
1
vote
2answers
114 views

Handling duplicate content caused by 'new' smart-url structure

I used to have an URL structure like this: mydomain.com/search.php?key=value which I've since changed to: mydomain.com/key/value. I've changed all the internal links in my site, but Google ...
0
votes
3answers
2k views

How to fix robots.txt file error in my GWT?

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file. My blog’s robots.txt file ...
0
votes
1answer
94 views

Searches for “more information about text formats”

I have a site using Drupal as CMS, and got "infected" badly with spammer bots. Moreover, recently I signed up for Google's Webmaster Tools, and I see that one of the top search queries that result in ...
1
vote
2answers
171 views

How does robots.txt handle links to disallowed pages?

We have affiliates that link to our site with a querystring. e.g. <a href="http://www.mysite.com/affiliate?id=123">Affiliate Link</a> In my robots.txt file, i have: Disallow: ...
1
vote
2answers
111 views

Does specifying full webpage path in robots.txt would affect my website?

It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some label path of my website using robots.txt file. User-agent: * Disallow: ...
1
vote
3answers
8k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
1
vote
3answers
205 views

Right way to allow all access to website in robots.txt?

user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?
1
vote
1answer
79 views

Best way to prevent Google from indexing a directory [duplicate]

I've researched many methods on how to prevent Google/other search engines from crawling a specific directory. The two most popular ones I've seen are: Adding it into the robots.txt file: Disallow: ...
0
votes
2answers
64 views

Confused for writing robots.txt file

I am getting DNS error from last few 5 days in my Google Webmaster Tools. That's the reason I would like to confirm my robots.txt file. Is the below robots.txt file is correct or not? User-agent: * ...
0
votes
1answer
66 views

Google indexed page a day before also reflecting in search but today everything vanish

We had robots.txt which disallow all robots as we were in development. We are live now. We change robots.txt as per our requirement a day before. Submit indexes using Google Webmaster Tools index ...
1
vote
1answer
108 views

Robots.txt Disallow command

How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error. The URL which been crawled incorrectly is: ...
3
votes
1answer
294 views

Blocking pages from being crawled with specific URL encoded parameters

While seeing the Google Webmaster Tools report there are certain URL encoded parameters like %5c and %22 coming up in URL's of site. We tried to identify the issue and observed that due to incorrect ...
1
vote
2answers
162 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
1
vote
1answer
299 views

Disallowed images in the robots.txt of my Joomla site can't be displayed when shared in Facebook

I have noticed that since I have disallowed images using the robots.txt in my Joomla site, when sharing an article in Facebook, the image will not be displayed. Why is that? Is it indeed related? My ...
4
votes
2answers
485 views

What is the correct way to write my “robots.txt” file?

I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: Sitemap: http://example.com/sitemap.xml Is ...
2
votes
4answers
1k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
0
votes
2answers
142 views

Google Webmaster Tools robots test not working

Within Webmaster Tools I have supplied my test content: User-agent: * Disallow:/admin/ Disallow: /tag/ When I specify the URL to test against, for example: http://www.site.com/tag/ It gives me ...
3
votes
1answer
617 views

Why does my sitemap have a “.gz” extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
2
votes
1answer
65 views

To allow or disallow alphabetical content directory pages?

We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
1
vote
2answers
677 views

Robots.txt disallowing URLs

I need to disallow some URLs on my site but I am not sure how to do that. I have a site that has products and reviews. When someone makes a review, the site generates a URL automatically like this: ...
3
votes
1answer
818 views

How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?

I have this line: Crawl-delay: 50 in the robots.txt file at the root of my website. But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes. I ...
2
votes
2answers
458 views

How much Google takes to deindex pages after disallowed from robots?

By accident I have let Google index lots of junk pages. Now I have added them to disallow in robots.txt. Will Google completely remove the pages after seeing the are disallowed from robots? Please ...
2
votes
1answer
462 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
4
votes
1answer
275 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
3answers
285 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
3
votes
2answers
976 views

How can an exception be created for Facebook on robots.txt?

I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
2
votes
1answer
50 views

what should be my initial address for sitemap and robots.txt if I'm using redirection

What should be my initial adress when using this site: http://www.xml-sitemaps.com? I have a folder inside my root where I have all my files. For example I have domain.com and my start directory is ...
2
votes
3answers
158 views

Should I use both Robots.txt and META NAME=“ROBOTS”?

I am currently using the Robots.txt file (the standard one of joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior. Is there a reason to ...
2
votes
1answer
72 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
2answers
88 views

Robots.txt Blocking Blog Tags?

Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled? User-agent: ...
1
vote
1answer
38 views

Prohibit search engines before going live

I want to prohibit search engines to index my web site before going live with it. If I simple do a robots.txt with: User-agent: * Disallow: / Would that cause any issues later when I decide to ...
0
votes
1answer
76 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
104 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...