Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
2answers
1k views

Correct usage of robots.txt file (“Googlebot-Image” and “Mediapartners-Google”)

This is the current robots.txt file I am using on a site of mine. I have excluded large parts of the IP.Board forum in order to try and cut down on duplicate content. I've also excluded some WordPress ...
0
votes
1answer
285 views

Sitemaps: rel=“alternate” hreflang=“x” naming

I'm creating a Sitemap for rel="alternate" hreflang="x", but no where in Google's guidelines do they mention how it should be named. Also can we reference it separately in the robots.txt file, or does ...
3
votes
3answers
7k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
3
votes
1answer
1k views

Does Google respect a redirect header for robots.txt to a different file name?

With respect to robots.txt requests, Google states "Redirects will generally be followed until a valid result can be found". Does this mean that, if I create a 301 redirect for ...
1
vote
3answers
109 views

robots.txt file Question

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google
5
votes
2answers
8k views

What's the proper way to handle Allow and Disallow in robots.txt?

I run a fairly large-scale Web crawler. We try very hard to operate the crawler within accepted community standards, and that includes respecting robots.txt. We get very few complaints about the ...
2
votes
2answers
1k views

Prevent crawling of mobile site

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users. The mobile content is in /mobile/ The regular site will redirect mobile users to the ...
0
votes
2answers
3k views

How long does it take for Google to remove page from index? Or, why aren't these pages excluded?

A set of pages are marked as noindex and nofollow, both in robots.txt and with X-Robots-Tag: noindex, nofollow When checking with Google Webmaster Tools, the pages are reported as "Denied by ...
0
votes
1answer
256 views

How to prevent webpages translated with Google Translate Plugin from being indexed?

I've been reading the information here https://support.google.com/translate/#2534535 about the Google Translate plugin. At first I thought that if I used the plugin my site wouldn't be indexed, but ...
3
votes
1answer
1k views

Why does my sitemap have a “.gz” extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
0
votes
1answer
51 views

Do I need disallow other websites directory in robots.txt?

I have a hosting which has several websites in www folder. I set up a WordPress website in root directory for main domain. Do I need disallow other websites directory in robots.txt? I think these ...
0
votes
1answer
120 views

How to block certain URL from robots.txt file? [closed]

This is simple question but also I have a confusion. Below is the code which I wrote in my robots.txt: User-Agent: * Disallow: /phpmyadmin/ Disallow: /login/ Disallow: /newsletter/ Disallow: ...
4
votes
2answers
591 views

What is the correct way to write my “robots.txt” file?

I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: Sitemap: http://example.com/sitemap.xml Is ...
4
votes
5answers
396 views

With Robots.txt disallow all, why was my site still getting traffic?

In failing to check my robots.txt on a community site, it had: User-agent: * Disallow: / I only clued into this because of a message in my AdSense account saying that five URL's couldn't be ...
1
vote
2answers
123 views

Handling duplicate content caused by 'new' smart-url structure

I used to have an URL structure like this: mydomain.com/search.php?key=value which I've since changed to: mydomain.com/key/value. I've changed all the internal links in my site, but Google ...
0
votes
1answer
97 views

Searches for “more information about text formats”

I have a site using Drupal as CMS, and got "infected" badly with spammer bots. Moreover, recently I signed up for Google's Webmaster Tools, and I see that one of the top search queries that result in ...
1
vote
2answers
198 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
0
votes
1answer
323 views

Noindex, Nofollow affect “Google Custom Search” results?

I am using Google Custom Search on my site, if I set pages to "noindex" or "nofollow", does it also affect the custom search results on my site?
1
vote
2answers
114 views

Does specifying full webpage path in robots.txt would affect my website?

It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some label path of my website using robots.txt file. User-agent: * Disallow: ...
1
vote
3answers
215 views

Right way to allow all access to website in robots.txt?

user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?
0
votes
1answer
67 views

Google indexed page a day before also reflecting in search but today everything vanish

We had robots.txt which disallow all robots as we were in development. We are live now. We change robots.txt as per our requirement a day before. Submit indexes using Google Webmaster Tools index ...
0
votes
2answers
69 views

Confused for writing robots.txt file

I am getting DNS error from last few 5 days in my Google Webmaster Tools. That's the reason I would like to confirm my robots.txt file. Is the below robots.txt file is correct or not? User-agent: * ...
1
vote
1answer
400 views

Disallowed images in the robots.txt of my Joomla site can't be displayed when shared in Facebook

I have noticed that since I have disallowed images using the robots.txt in my Joomla site, when sharing an article in Facebook, the image will not be displayed. Why is that? Is it indeed related? My ...
3
votes
1answer
363 views

Blocking pages from being crawled with specific URL encoded parameters

While seeing the Google Webmaster Tools report there are certain URL encoded parameters like %5c and %22 coming up in URL's of site. We tried to identify the issue and observed that due to incorrect ...
3
votes
4answers
2k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
7
votes
1answer
1k views

Should I block Wordpress archive pages from search engines?

I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives. Google has indexed the site fully. Since the archives currently display full posts I've thought that I shouldn't ...
1
vote
1answer
563 views

Robots.txt practices with .htaccess redirections (inherits)

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains ...
2
votes
1answer
416 views

HTTP 303 redirection and robots.txt

On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
76 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
3
votes
1answer
927 views

How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?

I have this line: Crawl-delay: 50 in the robots.txt file at the root of my website. But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes. I ...
2
votes
1answer
686 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
5
votes
3answers
1k views

Google indexing page with parameters but page is Disallowed in robots.txt

I have the following in robots.txt: User-agent: * Disallow: /refer.php User-agent: NinjaBot Allow: / Sitemap: http://www.mysite.com/sitemap.xml The refer.php file does various things depending on ...
4
votes
1answer
350 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
3answers
339 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
3
votes
2answers
2k views

How can an exception be created for Facebook on robots.txt?

I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
2
votes
1answer
57 views

what should be my initial address for sitemap and robots.txt if I'm using redirection

What should be my initial adress when using this site: http://www.xml-sitemaps.com? I have a folder inside my root where I have all my files. For example I have domain.com and my start directory is ...
2
votes
2answers
111 views

Robots.txt Blocking Blog Tags?

Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled? User-agent: ...
1
vote
1answer
38 views

Prohibit search engines before going live

I want to prohibit search engines to index my web site before going live with it. If I simple do a robots.txt with: User-agent: * Disallow: / Would that cause any issues later when I decide to ...
5
votes
3answers
92 views

SEO Removal Time?

I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here. I believe it will be the solution but what i still need to ...
0
votes
1answer
77 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
120 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
48 views

Question about duplicate groups (agents) in robots.txt

I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file. I've ...
6
votes
4answers
729 views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
2
votes
2answers
1k views

Will 301 redirection to a Google Indexed URL can remove the URL from Google index?

I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
2
votes
2answers
232 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
2answers
294 views

Would a 403 page, a blank page and a page redirect be indexed by search robots?

I am wondering if a disallow directive needs to be issued to robots on pages that will generate: a 403 forbidden error; a blank page, or; a page redirect (to log in page for example). Would not ...
2
votes
2answers
275 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
5
votes
1answer
112 views

Handling SEO for Infinite pages that cause external slow API calls

I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
1
vote
1answer
508 views

Block access to robots.txt file .htaccess

I read in this forum that this .htaccess code should block access to the robot.txt file: RewriteCond %{HTTP_USER_AGENT} ^Mozilla RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely) RewriteRule ...
4
votes
2answers
559 views

Should I prevent search engines indexing tag/category pages?

On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...