Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
1answer
91 views

What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?

I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
1
vote
4answers
381 views

Can I include a robots meta tag outside of the head in HTML snippets indeded to be AJAXed?

I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) ...
1
vote
2answers
553 views

robots.txt, how effective is it and how long does it take?

We recently updated the site to a single page site using jQuery to slide between "pages". So we now have only index.php. When you search the company on engines such as Google, you get the site and a ...
1
vote
1answer
427 views

Block access to robots.txt file .htaccess

I read in this forum that this .htaccess code should block access to the robot.txt file: RewriteCond %{HTTP_USER_AGENT} ^Mozilla RewriteCond %{HTTP_USER_AGENT} !(Slurp|surfsafely) RewriteRule ...
1
vote
1answer
419 views

Complex Disallow pattern in robots.txt

I have a URL like this: www.example.com/freelance-jobs-new-york I had a problem and many duplicated pages have been created like this: www.example.com/freelance-jobs-new-york-php-php ...
1
vote
1answer
258 views

Need to sanity-check my .htaccess, especially Limit GET POST line for Google repellent

I need a sanity check on this .htaccess (from a WordPress site) I inherited from a 5 month+ old site. What's the symptom? Google + Bing crawl, but don't index any of the pages. Let me be clear: I'm ...
1
vote
2answers
493 views

How to submit sitemap when your website has partial https? - Error: “Not in Domain”

My website is an eCommerce that is set up to do HTTP for the item browsing portion, but HTTPS for things like shopping cart, contact us, etc.. (anything that has forms on it). I've submitted my ...
1
vote
1answer
514 views

Robots.txt practices with .htaccess redirections (inherits)

I have a question regarding how to write robots.txt files for many domains and subdomains with redirects in place. We have a hosting account that enacts primary and add-on domains. All of our domains ...
1
vote
1answer
76 views

Search and Tags for robots.txt

After setting up a blog at blogspot.com I added the site to my webmasters tools of google. I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
1
vote
1answer
1k views

How to hide my website from search engines using robots.txt? [duplicate]

Possible Duplicate: Robots denied by domain is still listed in search results I want to hide my website from all search engines for that i write robots.txt in that robots.txt i write code ...
1
vote
1answer
123 views

To disallow indexing the category and tag listings in a blog

Mark Wilson says that category and tag listings in a blog should be disallowed in order to prevent duplicate content. I understand this. However, I want to put internal links on keywords in the blog ...
1
vote
1answer
35 views

Why would I use a robots.txt file? [duplicate]

From what I understand after reading Google's Controlling Crawling and Indexing: The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
1
vote
1answer
90 views

RSS feeds and robots.txt

I have RSS feeds on my site. I've decided to follow Stack Exchange and disallow my RSS feeds in the robots.txt. I don't want search engine to display the RSS feed page to people, that's not really a ...
1
vote
4answers
745 views

URL blocked by robots.txt in webmaster for Blogger site

Remove: Disallow: /search From where can I remove this line? I am using Blogger and in webmaster it shows many of my pages are blocked. From the following Q&A, I know that I need to remove this ...
1
vote
3answers
1k views

Robots.txt never downloaded but some blocked URLs in GWT

There is something I don't understand in Google Webmaster Tools (GWT) for my WordPress site. In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked ...
1
vote
0answers
14 views

Mobile site robots.txt [duplicate]

Possible Duplicate: Should I block indexing of my mobile site? Should I allow webcrawlers to index the mobile versions of my websites? Is that a good practice from a SEO perspective? ...
1
vote
2answers
354 views

Facebook and Crawl-delay in robots.txt?

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
1
vote
2answers
136 views

Is this a correct robots.txt file?

I would like to allow Googlebot and Mediapartners-Google (AdSense useragent) to crawl my website. So I have written below code inside my robots.txt file. User-agent: Googlebot Disallow: ...
1
vote
2answers
596 views

SEO: disallowing Google from indexing forms?

From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not? I mean my forms hardly ever contains keyword/keyphrases. So I'm wondering what's the point of letting ...
0
votes
3answers
1k views

How to fix robots.txt file error in my GWT?

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file. My blog’s robots.txt file ...
0
votes
3answers
6k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
0
votes
1answer
1k views

SEO not indexing my site: “A description for this result is not available because of this site's robots.txt – learn more.” [duplicate]

When googling my site I get following text under the result for my domain: "A description for this result is not available because of this site's robots.txt – learn more." My robots.txt, ...
0
votes
1answer
341 views

How to secure robots.txt file?

I would like for User-agents to index my relative pages only without accessing any directory on my server. As initial thought, i had this version in mind: User-agent: * Disallow: */* Sitemap: ...
0
votes
1answer
42 views

Google say: robots.txt not valid for subdomains?

I'm very confused about Google's robots.txt documentation Examples of valid robots.txt URLs: Not valid for http://other.example.com/ https://example.com/ http://example.com:8181/ ...
0
votes
1answer
132 views

How to prevent webpages translated with Google Translate Plugin from being indexed?

I've been reading the information here https://support.google.com/translate/#2534535 about the Google Translate plugin. At first I thought that if I used the plugin my site wouldn't be indexed, but ...
0
votes
3answers
128 views

Robots are still disallowed even without a robots.txt file

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has ...
0
votes
2answers
127 views

Will robots.txt block crawling things like menu and footer links?

If I am blocking a page called pay.php, google won't crawl this page. But what about the links that are on this page as default as menu and footer links? Will these links also be blocked from ...
0
votes
1answer
75 views

How to block certain URL from robots.txt file? [closed]

This is simple question but also I have a confusion. Below is the code which I wrote in my robots.txt: User-Agent: * Disallow: /phpmyadmin/ Disallow: /login/ Disallow: /newsletter/ Disallow: ...
0
votes
2answers
63 views

Confused for writing robots.txt file

I am getting DNS error from last few 5 days in my Google Webmaster Tools. That's the reason I would like to confirm my robots.txt file. Is the below robots.txt file is correct or not? User-agent: * ...
0
votes
1answer
227 views

Noindex, Nofollow affect “Google Custom Search” results?

I am using Google Custom Search on my site, if I set pages to "noindex" or "nofollow", does it also affect the custom search results on my site?
0
votes
1answer
339 views

Is “robots.txt” necessarily needed? [duplicate]

Possible Duplicate: Do I need robots.txt? From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
0
votes
1answer
460 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
0
votes
1answer
24 views

Remove page from Search Result Bunch [duplicate]

When I do a branded search for a website the structure shows like ( see image ). I would love to remove Privacy Policy page or Manipulate the structure I see now ( showing pages I want to ). How I can ...
0
votes
3answers
83 views

Remove subdomains from Google index and stop indexing them

I am running static content through a CDN, cdn1-cdn5 I am using such subdomains for that. I am loading just images, CSS and JS files this way, but apparently Google has indexed some pages on ...
0
votes
1answer
29 views

How to use wildcards to remove print pages from search engines?

Our site has pages that are print-friendly, which I would like to remove from search engines. Here's an example: ...
0
votes
1answer
102 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
0
votes
1answer
49 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
0
votes
2answers
24 views

robots.txt dissalow url containing string with a '/' at the end

I have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
0
votes
1answer
78 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...
0
votes
1answer
32 views

My first robots.txt [closed]

I'm creating my first robots.txt and wanted to get a second opinion on it. Basically I have a FTP setup on my board for some special users to transfer files between each other and I do NOT want that ...
0
votes
1answer
114 views

Sitemaps: rel=“alternate” hreflang=“x” naming

I'm creating a Sitemap for rel="alternate" hreflang="x", but no where in Google's guidelines do they mention how it should be named. Also can we reference it separately in the robots.txt file, or does ...
0
votes
1answer
46 views

Do I need disallow other websites directory in robots.txt?

I have a hosting which has several websites in www folder. I set up a WordPress website in root directory for main domain. Do I need disallow other websites directory in robots.txt? I think these ...
0
votes
1answer
35 views

Searches for “more information about text formats”

I have a site using Drupal as CMS, and got "infected" badly with spammer bots. Moreover, recently I signed up for Google's Webmaster Tools, and I see that one of the top search queries that result in ...
0
votes
1answer
65 views

Google indexed page a day before also reflecting in search but today everything vanish

We had robots.txt which disallow all robots as we were in development. We are live now. We change robots.txt as per our requirement a day before. Submit indexes using Google Webmaster Tools index ...
0
votes
2answers
134 views

Google Webmaster Tools robots test not working

Within Webmaster Tools I have supplied my test content: User-agent: * Disallow:/admin/ Disallow: /tag/ When I specify the URL to test against, for example: http://www.site.com/tag/ It gives me ...
0
votes
1answer
74 views

How to overcome duplicate URLS

I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls. From: /psd-layouts.html to /psd-web-layouts.html Now my question is that, I had ...
0
votes
2answers
91 views

Googlebot ignore my new site [duplicate]

I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google. For some technical problem I have delete it and I have newly created using WordPress ...
0
votes
1answer
586 views

Google Webmaster Tools is showing incorrect warnings - blocked by robots.txt

I'm getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt when they are not. Has any one come across this or a way to resolve this before ? ...
0
votes
1answer
54 views

If I blocked a directory, is a 301 error expected?

I blocked a directory in my site that is not secret but meant for internal discussions that are not relevant for search engine results. After running "Search as Google" at Google Webmaster Tools, the ...
0
votes
1answer
279 views

wordpress robots.txt is this reasonable?

this is my first time to write a robots.txt, I am not sure this is reasonable or not. sorry for open a question. I use wordpress cms. all the page urls like http://mydomain.com/it/page1, ...