Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
1answer
40 views

My site's robots.txt was blocking everyone for a year. What to do now that I fixed that?

My site's robots.txt was blocking everyone for a year (The "Discourage search engines from indexing this site" option was checked in WordPress). I guess Google will re-try to crawl it in the upcoming ...
3
votes
1answer
103 views

Robots.txt destroyed my ranking?

I had to do a massive URL change stuff (Categories and Products) on my e-commerce site while keeping 301 redirect of old URLs to new ones. I did change category URLs (appr: 800 URLs) to new and ...
0
votes
1answer
328 views

Clicks counting and crawler bots

I am currently running a small affiliate-program for Facebook users. We use an auto-poster to publish links to fan pages. Every hit is stored in our database and we have included a 24 hour reload ...
1
vote
3answers
148 views

Google is not crawling and indexing my site after updating my robots.txt file

I have an issue with Google not being able to properly crawl my site. I have read other questions where people have had the same issue. I've tried to follow their solution of using this in my ...
1
vote
3answers
103 views

Making a dynamic robots.txt but system is appending forward slash

I am making robots.txt dynamically instead of creating a physical robots.txt file. I am making this from the CMS and the system is appending a forward slash to every page. In the case of robots.txt, I ...
2
votes
1answer
124 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
11
votes
2answers
883 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
5
votes
1answer
553 views

Robots.txt got changed to disallow crawling by accident

Our robots.txt mistakenly got changed to: User-agent: * Disallow: / This occurred about 2 weeks ago and was changed back yesterday to allow crawling. Our listed pages in the Googles have ...
2
votes
1answer
334 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
1
vote
1answer
153 views

Google Webmaster Tools: Robots disallow does not seem to be working for staging site

I've added robots.txt on my staging site at staging.mydomain.com as: User-agent: * Disallow: / I then added and verified the staging site in Google Webmasters Tools. In Crawl > Blocked URLs, I can ...
2
votes
1answer
2k views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
6
votes
4answers
2k views

Should I set NOINDEX header for my JS, CSS and image files?

Are there any harms if my site send NOINDEX headers for all my static assets? For image files, I refer to those valueless, e.g. background images, button images, etc. Update: more background ...
1
vote
3answers
111 views

How do I remove a LOT of indexed pages from Google?

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
1
vote
1answer
182 views

CDN virtual subdomain causes duplicated content

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act ...
1
vote
1answer
327 views

Sudden Increase in “Blocked from robots”

In the Index Status section of Google Webmaster Tools a site I manage has seen a sudden increase in pages "Blocked from robots". There's been no spike in Crawl Errors since this happened a week and ...
3
votes
3answers
2k views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
6
votes
2answers
293 views

What kind of redirect (301 or 302) for an email links tracker?

We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application ...
1
vote
1answer
71 views

Confused about redirect and robots.txt

I am updating a site by creating new pages and deleting old ones. To avoid crawling errors should I use both a robots.txt file and 301 redirect on .htaccess? robots.txt: Disallow: /products.php ...
2
votes
2answers
201 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
1
vote
1answer
198 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
1
vote
2answers
312 views

Need to encourage google to refresh robots.txt [duplicate]

My old robots.txt is User-agent: * Disallow: / Is blocking me for uploading a new sitemap and also blocking me for fetching robots.txt manually. I don't know what to do.
5
votes
2answers
154 views

Is there any reason for putting humans.txt except of acknowledgement?

Are there any valid reasons of putting humans.txt? The only reason I see so far is to give credit to the team who created the site, and open source libraries it is using.
2
votes
1answer
65 views

Robots and Sitemaps

I have some problems regarding indexing. Google indexed so much in the list. Now I wanted to remove everything from my site. If I have disallow all in robots.txt and send my sitemap.xml. Would ...
0
votes
1answer
153 views

Hiding page content from Google when it is duplicated across several sites

I would like to hide some content from Google. Because a client has multiple websites with the same content then he gets a warning about duplicate content on some of them (multiple countries, same ...
0
votes
1answer
60 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
0
votes
1answer
516 views

How my robots.txt should look like for a single page app

I can understand how to disallow bots to crawl some pages/folder in normal application. For example for google-bot it is nicely described here. But what should I do if I have a single page ...
0
votes
1answer
660 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
4
votes
2answers
1k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
3
votes
4answers
2k views

Google Indexing Pages with noindex meta tag

I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs. For some of the pages listed, there was nothing to ...
1
vote
2answers
270 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
2
votes
5answers
1k views

Optimizing and securing robots.txt

I have a couple of doubts / questions / ideas related to robots.txt: Can we deny website for all bots except for chosen ones in order to tell other bots not to crawl site: User-agent: * Disallow: / ...
2
votes
1answer
310 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
8
votes
1answer
182 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
0
votes
1answer
70 views

Remove page from Search Result Bunch [duplicate]

When I do a branded search for a website the structure shows like ( see image ). I would love to remove Privacy Policy page or Manipulate the structure I see now ( showing pages I want to ). How I can ...
1
vote
1answer
57 views

Why would I use a robots.txt file? [duplicate]

From what I understand after reading Google's Controlling Crawling and Indexing: The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
1
vote
3answers
792 views

Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
0
votes
1answer
211 views

Google say: robots.txt not valid for subdomains?

I'm very confused about Google's robots.txt documentation Examples of valid robots.txt URLs: Not valid for http://other.example.com/ https://example.com/ http://example.com:8181/ ...
1
vote
3answers
3k views

How to fix robots.txt file error in my GWT?

In my blog’s Webmaster Tools, there is a notification in Crawl Errors section, that is Google couldn't crawl your site because we were unable to access the robots.txt file. My blog’s robots.txt file ...
1
vote
1answer
832 views

Robots.txt error “Crawl postponed because robots.txt was inaccessible” from Google Webmaster Tools

On the robots.txt fetch tab there is a yellow triangle when hovered over. I am getting the following message. "Crawl postponed because robots.txt was inaccessible" However I am showing 0/151 ...
1
vote
1answer
209 views

How can I set my robots.txt file to let bots not crawl certain folders in root directory?

I have a small website with about 20 pages with .html extensions, containing unique articles I wrote consisting of about 1000 words or more. These 20 pages are the main feature of my site. Root ...
1
vote
1answer
68 views

Will there be a conflict between my Sitemap.xml and robots.txt?

I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: ...
2
votes
3answers
6k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
0
votes
0answers
15 views

How to prevent Google (and others) from indexing the site under construction? [duplicate]

Recently I faced with the situation that the wrong site pages (with wrong URLs, wrong content) was indexed by Google during the site developing. Should I place robots.txt with User-agent: * Disallow: ...
3
votes
2answers
507 views

How should “AdSense and DoubleClick” phrase in Google Webmaster Guidelines be interpreted?

I was reading Google Webmaster Guidelines and came across a phrase I do not understand. Make reasonable efforts to ensure that advertisements do not affect search engine rankings. For example, ...
1
vote
2answers
493 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
4
votes
2answers
135 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
4
votes
2answers
352 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
3answers
455 views

Would a self-signed certificate impact my site for search engines?

Would a self-signed certificate impact my site for search engines? By that I mean, do robots refuse to index sites with self-signed certificates?
4
votes
2answers
179 views

How to temporarily remove a website from search engines?

We are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain. Now we are already being crawled and indexed by bots (Google, Yahoo, and others) ...
2
votes
2answers
251 views

Can a website require search engine robots to support cookies?

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other ...