Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

3
votes
3answers
600 views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
1
vote
1answer
54 views

Confused about redirect and robots.txt

I am updating a site by creating new pages and deleting old ones. To avoid crawling errors should I use both a robots.txt file and 301 redirect on .htaccess? robots.txt: Disallow: /products.php ...
2
votes
2answers
113 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
1
vote
1answer
129 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
1
vote
2answers
142 views

Need to encourage google to refresh robots.txt [duplicate]

My old robots.txt is User-agent: * Disallow: / Is blocking me for uploading a new sitemap and also blocking me for fetching robots.txt manually. I don't know what to do.
2
votes
1answer
60 views

Robots and Sitemaps

I have some problems regarding indexing. Google indexed so much in the list. Now I wanted to remove everything from my site. If I have disallow all in robots.txt and send my sitemap.xml. Would ...
0
votes
1answer
66 views

Hiding page content from Google when it is duplicated across several sites

I would like to hide some content from Google. Because a client has multiple websites with the same content then he gets a warning about duplicate content on some of them (multiple countries, same ...
1
vote
1answer
645 views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
1
vote
1answer
111 views

CDN virtual subdomain causes duplicated content

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act ...
0
votes
1answer
52 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
1
vote
1answer
171 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
0
votes
1answer
149 views

How my robots.txt should look like for a single page app

I can understand how to disallow bots to crawl some pages/folder in normal application. For example for google-bot it is nicely described here. But what should I do if I have a single page ...
6
votes
1answer
380 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
0
votes
1answer
205 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
4
votes
2answers
605 views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
1
vote
3answers
64 views

Making a dynamic robots.txt but system is appending forward slash

I am making robots.txt dynamically instead of creating a physical robots.txt file. I am making this from the CMS and the system is appending a forward slash to every page. In the case of robots.txt, I ...
6
votes
2answers
326 views

80% of Website Traffic coming from Just one Town in USA - Strange?

For the last month, 80% of my total traffic was coming from just one Town in the USA. Here are the attributes: It is one of the richest towns in California in USA which defies logic since I run a ...
5
votes
2answers
144 views

Is there any reason for putting humans.txt except of acknowledgement?

Are there any valid reasons of putting humans.txt? The only reason I see so far is to give credit to the team who created the site, and open source libraries it is using.
2
votes
2answers
95 views

Does the Instant Preview in Google Webmaster Tools take robots.txt in account?

Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as googlebot, it fetches ok, ...
1
vote
2answers
134 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
2
votes
5answers
989 views

Optimizing and securing robots.txt

I have a couple of doubts / questions / ideas related to robots.txt: Can we deny website for all bots except for chosen ones in order to tell other bots not to crawl site: User-agent: * Disallow: / ...
2
votes
3answers
276 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
2
votes
1answer
97 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
1
vote
2answers
284 views

CPanel Visitors filled with bandwidth thieves

Issue: Strange and seemingly random entries on the CPanel Latest Visitors console, showing IPs accessing image files without any referring URL and not identified as Bots. Background: I have never had ...
8
votes
1answer
130 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
0
votes
1answer
47 views

Remove page from Search Result Bunch [duplicate]

When I do a branded search for a website the structure shows like ( see image ). I would love to remove Privacy Policy page or Manipulate the structure I see now ( showing pages I want to ). How I can ...
1
vote
1answer
25 views

robots.txt could be overridden by another robots.txt in sub folders?

robots.txt is supposed to go in the root directory, I know. But, could it be overridden by another robots.txt inside a sub directory?
1
vote
1answer
50 views

Why would I use a robots.txt file? [duplicate]

From what I understand after reading Google's Controlling Crawling and Indexing: The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
1
vote
3answers
3k views

Remove subdomains from Google index and stop indexing them

I am running static content through a CDN, cdn1-cdn5 I am using such subdomains for that. I am loading just images, CSS and JS files this way, but apparently Google has indexed some pages on ...
1
vote
3answers
370 views

Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
0
votes
1answer
121 views

Google say: robots.txt not valid for subdomains?

I'm very confused about Google's robots.txt documentation Examples of valid robots.txt URLs: Not valid for http://other.example.com/ https://example.com/ http://example.com:8181/ ...
0
votes
1answer
187 views

Clicks counting and crawler bots

I am currently running a small affiliate-program for Facebook users. We use an auto-poster to publish links to fan pages. Every hit is stored in our database and we have included a 24 hour reload ...
1
vote
1answer
477 views

Robots.txt error “Crawl postponed because robots.txt was inaccessible” from Google Webmaster Tools

On the robots.txt fetch tab there is a yellow triangle when hovered over. I am getting the following message. "Crawl postponed because robots.txt was inaccessible" However I am showing 0/151 ...
1
vote
1answer
126 views

How can I set my robots.txt file to let bots not crawl certain folders in root directory?

I have a small website with about 20 pages with .html extensions, containing unique articles I wrote consisting of about 1000 words or more. These 20 pages are the main feature of my site. Root ...
1
vote
1answer
55 views

Will there be a conflict between my Sitemap.xml and robots.txt?

I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: ...
2
votes
1answer
178 views

Blogspot noindex search label

The default robots.txt of Blogspot is: User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: ...
0
votes
0answers
15 views

How to prevent Google (and others) from indexing the site under construction? [duplicate]

Recently I faced with the situation that the wrong site pages (with wrong URLs, wrong content) was indexed by Google during the site developing. Should I place robots.txt with User-agent: * Disallow: ...
3
votes
2answers
200 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
1answer
603 views

Restrict all robots except Googlebot and Yandexbot

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages. All other bots will not be accessed my pages. Is this robots.txt ...
2
votes
3answers
171 views

Would a self-signed certificate impact my site for search engines?

Would a self-signed certificate impact my site for search engines? By that I mean, do robots refuse to index sites with self-signed certificates?
1
vote
1answer
340 views

Preventing Twitterbot to access website

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And ...
4
votes
2answers
98 views

Do I need a robots.txt file if most of my site's pages require authorization?

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization. Does that mean I need to have a robots.txt file to Disallow these pages with ...
1
vote
2answers
766 views

Should I use “Disallow”, “NoFollow”, or “NoIndex” for paginated listing pages?

I own a web site which has mainly two types of pages: Content pages: they are pretty much static, and link to the previous and next content page (chronologically). The structure is that of a double ...
2
votes
2answers
92 views

Sitemap specified in robots.txt not found by Google

I would like Google to find my sitemap.xml via robots.txt, rather than having to manually upload it. My robots.txt looks like this: User-agent: * Disallow: Sitemap: ...
1
vote
1answer
273 views

“Syntax not understood” error in Google Webmaster Tools robots.txt analysis when I use RewriteRule

In order to redirect my www.site.com to non-www, I employed this bit of code found somewhere on the web. RewriteCond %{HTTP_HOST} ^www.website.com [NC] RewriteRule ^(.*)$ http://website.com/$1 ...
2
votes
2answers
160 views

Can a website require search engine robots to support cookies?

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other ...
4
votes
3answers
251 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
5
votes
1answer
99 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
5
votes
1answer
150 views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...
4
votes
2answers
114 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...