Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

0
votes
2answers
68 views

Confused for writing robots.txt file

I am getting DNS error from last few 5 days in my Google Webmaster Tools. That's the reason I would like to confirm my robots.txt file. Is the below robots.txt file is correct or not? User-agent: * ...
0
votes
1answer
67 views

Google indexed page a day before also reflecting in search but today everything vanish

We had robots.txt which disallow all robots as we were in development. We are live now. We change robots.txt as per our requirement a day before. Submit indexes using Google Webmaster Tools index ...
1
vote
1answer
124 views

Robots.txt Disallow command

How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error. The URL which been crawled incorrectly is: ...
3
votes
1answer
329 views

Blocking pages from being crawled with specific URL encoded parameters

While seeing the Google Webmaster Tools report there are certain URL encoded parameters like %5c and %22 coming up in URL's of site. We tried to identify the issue and observed that due to incorrect ...
1
vote
2answers
179 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
1
vote
1answer
349 views

Disallowed images in the robots.txt of my Joomla site can't be displayed when shared in Facebook

I have noticed that since I have disallowed images using the robots.txt in my Joomla site, when sharing an article in Facebook, the image will not be displayed. Why is that? Is it indeed related? My ...
4
votes
2answers
536 views

What is the correct way to write my “robots.txt” file?

I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: Sitemap: http://example.com/sitemap.xml Is ...
3
votes
4answers
2k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
0
votes
2answers
145 views

Google Webmaster Tools robots test not working

Within Webmaster Tools I have supplied my test content: User-agent: * Disallow:/admin/ Disallow: /tag/ When I specify the URL to test against, for example: http://www.site.com/tag/ It gives me ...
3
votes
1answer
862 views

Why does my sitemap have a “.gz” extension, and how can I edit it?

An XML sitemap generator plugin for WordPress puts the following strings in my robots.txt file: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ But other WordPress blogs have lots of ...
2
votes
1answer
68 views

To allow or disallow alphabetical content directory pages?

We have a content that publishes fresh content 2-3 times a day. The site has category pages for tags and each alphabet. They both list the title with links to all the content within that category or ...
1
vote
2answers
775 views

Robots.txt disallowing URLs

I need to disallow some URLs on my site but I am not sure how to do that. I have a site that has products and reviews. When someone makes a review, the site generates a URL automatically like this: ...
3
votes
1answer
872 views

How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?

I have this line: Crawl-delay: 50 in the robots.txt file at the root of my website. But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes. I ...
2
votes
2answers
555 views

How much Google takes to deindex pages after disallowed from robots?

By accident I have let Google index lots of junk pages. Now I have added them to disallow in robots.txt. Will Google completely remove the pages after seeing the are disallowed from robots? Please ...
2
votes
1answer
589 views

“Google couldn't access your site because of a server connectivity issue”

In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site". Under the Current Status it has: DNS - SUCCESS Server Connectivity ...
4
votes
1answer
308 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
3answers
319 views

Googlebot is blocked from my domain

My robots.txt is: User-agent: * Disallow: / Sitemap: http://example.com/sitemap.xml Sitemap: http://example.com/sitemap.xml.gz It has been the same for well over a week now, but when I go to ...
3
votes
2answers
1k views

How can an exception be created for Facebook on robots.txt?

I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
2
votes
1answer
51 views

what should be my initial address for sitemap and robots.txt if I'm using redirection

What should be my initial adress when using this site: http://www.xml-sitemaps.com? I have a folder inside my root where I have all my files. For example I have domain.com and my start directory is ...
2
votes
3answers
164 views

Should I use both Robots.txt and META NAME=“ROBOTS”?

I am currently using the Robots.txt file (the standard one of joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior. Is there a reason to ...
2
votes
1answer
75 views

GoogleBot can't reach my site

For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
2
votes
2answers
101 views

Robots.txt Blocking Blog Tags?

Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled? User-agent: ...
1
vote
1answer
38 views

Prohibit search engines before going live

I want to prohibit search engines to index my web site before going live with it. If I simple do a robots.txt with: User-agent: * Disallow: / Would that cause any issues later when I decide to ...
0
votes
1answer
77 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
117 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
45 views

Question about duplicate groups (agents) in robots.txt

I'm looking at a corner case with robots.txt, and am curious if you have any guidance. Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file. I've ...
1
vote
1answer
140 views

Block search engines from the main domain while allowing add-on domains to rank well

I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page. Can I tell bots not to index the main domain WITHOUT ...
6
votes
4answers
613 views

Robots.txt vs Sitemap — Who wins in a Conflict

If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
2
votes
2answers
1k views

Will 301 redirection to a Google Indexed URL can remove the URL from Google index?

I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
2
votes
4answers
1k views

Google Indexing Pages with noindex meta tag

I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs. For some of the pages listed, there was nothing to ...
2
votes
2answers
658 views

Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?

As I understand, having repetitive content is a detriment to search engine placement. Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
2
votes
2answers
264 views

Robots.txt on one line

Can I combine robots.txt rules on one line, separated by whitespace? Will this work? User-agent: * Disallow: /
1
vote
4answers
990 views

URL blocked by robots.txt in webmaster for Blogger site

Remove: Disallow: /search From where can I remove this line? I am using Blogger and in webmaster it shows many of my pages are blocked. From the following Q&A, I know that I need to remove this ...
6
votes
2answers
218 views

What kind of redirect (301 or 302) for an email links tracker?

We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application ...
7
votes
1answer
1k views

Should I block Wordpress archive pages from search engines?

I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives. Google has indexed the site fully. Since the archives currently display full posts I've thought that I shouldn't ...
4
votes
2answers
331 views

Impact of blocking site search from googlebot, but listing content pages in sitemap.xml

We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
5
votes
1answer
169 views

How to fix thousands of 404's?

The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
2
votes
3answers
696 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
0
votes
1answer
1k views

SEO not indexing my site: “A description for this result is not available because of this site's robots.txt – learn more.” [duplicate]

When googling my site I get following text under the result for my domain: "A description for this result is not available because of this site's robots.txt – learn more." My robots.txt, ...
3
votes
2answers
315 views

Will duplicate international (i18n) content hinder SEO rankings?

Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
4
votes
2answers
156 views

Preventing search engines to crawl on specific folders

I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow: ...
2
votes
2answers
595 views

Network unreachable: robots.txt unreachable

I am trying to add a valid sitemap to Google Webmaster. Yet, it says: Network unreachable: robots.txt unreachableWe were unable to crawl your Sitemap because we found a robots.txt file at ...
5
votes
3answers
91 views

SEO Removal Time?

I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here. I believe it will be the solution but what i still need to ...
6
votes
4answers
3k views

Hide a Subdomain from Google/Search Engine SEO Result?

I have a subomain which I do not want to be listed in any search engine result. Let say I have: http://www.example.com http://child.example.com How can I hide all URLs of child.example.com domain ...
0
votes
0answers
355 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
2
votes
2answers
219 views

How to fix 404 crawl error of a website

I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below +top_folder -config.php +includes_folder ...
2
votes
2answers
127 views

Prevent crawler that doesn't honoring robots.txt

I have some problem, when I try to write robots.txt for my site ... I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
1
vote
2answers
381 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
1
vote
3answers
1k views

Robots.txt never downloaded but some blocked URLs in GWT

There is something I don't understand in Google Webmaster Tools (GWT) for my WordPress site. In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked ...
1
vote
2answers
299 views

Can search engine robots read file with permission 640?

I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt. As per the following google group ...