Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

0
votes
2answers
22 views

Sitemap XML title in search results

I have a website that uses an XML sitemap and a robots.txt file. If I do a Google search of: site: mywebsite.co.uk "sitemap" I see the usual Google response, similar to the following image, ...
1
vote
1answer
74 views

Problem with Google Mobile Test - Blocking images

I have a situation when I check my Drupal 7 website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 56 resources ...
8
votes
2answers
762 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
2
votes
2answers
42 views

Should I block an .htm resource in robots.txt

We use scripts to create static .htm files to show content on our site. Mainly in an effort to reduce database calls. For example, a script that builds a file every 5 minutes that contains a list of ...
0
votes
0answers
19 views

Error messages when checking my Blog Page

I ran a test and this is the message. I am not sure why it is not able to access robot txt files. The important stuff: Your web server [23.229.212.161] is working fine. Your RSS feed returned ...
1
vote
2answers
99 views

Should i noindex low/quality duplicate content tag pages?

Google have indexed 300k tag pages of my site, these are automatically generated based on the words on the title of each entry so it's likely to see pages which doesn't make much sense like: ...
3
votes
3answers
1k views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
1
vote
1answer
133 views

Google mobile friendly resources which are blocked by robots.txt

I have a situation when I check my website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 5 resources which are ...
6
votes
5answers
3k views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
2
votes
2answers
34 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
3
votes
1answer
40 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
1
vote
1answer
198 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion). I am building other sites, ...
5
votes
3answers
48 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
7
votes
4answers
397 views

How do you configure robots.txt to allow crawling of the site except for a few directories?

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders? Is there a general setup that should always be used?
7
votes
2answers
183 views

Is this “robots.txt” file really preventing all crawling of our website? I'm trying to find out why our SEO is so poor

So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following: # go away ...
1
vote
1answer
31 views

Remove website pages from search results, but keep them crawled

I would like to remove all pages of my website from the search results, except the home page. Is there a way to do this and keep the pages crawlable? Or is there no way except robots.txt?
3
votes
1answer
45 views

How do I disallow variable parent directories and subdirectory with robots.txt?

I want to disallow robots to index some pages of my site. For example, I want to disallow the following pages: /category/name1/page/1 /category/name1/page/2 /category/etc/page/3 But I want to allow ...
3
votes
6answers
171 views

How do you prevent web pages from appearing in the Google search index?

I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a noindex tag? If so, then how do I do so?
1
vote
1answer
58 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
4
votes
4answers
100 views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it ...
0
votes
0answers
17 views

Robots.txt Query String - Couldn't Disallow [duplicate]

I have url : http://www.website.com/shopping/books/?b=9 I disallowed /?b=9 from robots.txt as User-agent: * Disallow: /?b=9 But when i test this from Google Webmaster, on Robots.txt Test Tools, ...
2
votes
2answers
123 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
3
votes
1answer
59 views

Point Domain Names With keyword to main website?

This is Question has been asked several time on Stack Exchange. I have read all those answer, still there are certain some technical specification answered. After lot of research on Internet , I ...
2
votes
1answer
35 views

Using canonical links to eliminate “similar” search results in google

In our site, we have links as such: /{{category-name}}.html - will display all products /category-name/filters/{{some-filter}}.html - will display filtered products The problem we are facing is ...
0
votes
1answer
405 views

How to prevent webpages translated with Google Translate Plugin from being indexed?

I've been reading the information here https://support.google.com/translate/#2534535 about the Google Translate plugin. At first I thought that if I used the plugin my site wouldn't be indexed, but ...
14
votes
7answers
5k views

Does Google cache robots.txt?

I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see Googlebot loading those exact ...
0
votes
0answers
55 views

SEO: Deindex member profile pages?

I have a forum where users can register and automatically a profile page is created, like here in Stackexchange. The problem with these pages is that sometimes users creates accounts at different ...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
41 views

Sitemap in different domain and robots.txt crawling

I have the following subdomains: www.example.com api.example.com The subdomain api.example.com is supposed to be invisible to Google. Its robots.txt had a disallow / directive. But I don't want ...
0
votes
1answer
163 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
0
votes
1answer
66 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
2
votes
1answer
47 views

Remove several URL of website from Google

I want to remove multiple URL in my website (developped in ASP MVC) All these URL have the format www.mysite.com/planning/xxx I have already adapted the code to make 301 redirect and indicate to ...
0
votes
0answers
15 views

Why does my staging site show up in Google even though my robots.txt file shouldn't allow it? [duplicate]

I'm having a lot of trouble with something that should be simple. I don't want my development site indexed by Google. According to Google's instructions, and just about everything else I can find says ...
1
vote
3answers
9k views

Remove subdomains from Google index and stop indexing them

I am running static content through a CDN, cdn1-cdn5 I am using such subdomains for that. I am loading just images, CSS and JS files this way, but apparently Google has indexed some pages on ...
4
votes
3answers
60 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
1
vote
2answers
49 views

Proper content for robots.txt on server maintenance page?

I know that the proper server response header should be 503 Service Unavailable, to let the search bots know that they shouldn't index the pages like Update in progress and Server maintenance. But ...
2
votes
1answer
46 views

How much time does robots.txt remain in Google cache?

The old robots.txt alerts me with warning about locked URL. With the new one, the URL passes the tester in Google Webmaster. But when I resend the sitemap I get the same error!
0
votes
2answers
115 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
1
vote
1answer
200 views

What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?

I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
0
votes
0answers
44 views

Google reports that robots.txt is blocking site - but I don't think it is

I have a site map at http://www.example.com/sitemap_index.xml that has been sumbitted to Google Webmaster Tools, however... Google is reporting that my sitemap is being blocked by robots.txt. My ...
5
votes
1answer
97 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
3
votes
2answers
134 views

Serving images on CDN; allow crawl of origin or CDN or both?

I have searched and read through a lot of questions regarding this subject, but cannot seem to find anyone that directly addresses my problem. I have a CDN for all my static images img.domain.net, ...
1
vote
1answer
51 views

Configuring the document root on a cookieless domain

I am configuring a cookieless domain for all my static content, I see a lot of articles and how-to's use the same document root for as their main domain. Why? This seems like a disadvantage to me, ...
3
votes
1answer
10k views

How do I disallow an entire directory with robots.txt?

I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example http://someurl.com/first_url ...
1
vote
0answers
99 views

Magento Robots.txt and blocking cached images

We have a Magento based store. Images are resized and served cached dynamically. When the image cache in Magento is cleared, there are generated new paths and files. And I get 404 errors for them ...
1
vote
5answers
906 views

Can I include a robots meta tag outside of the head in HTML snippets indeded to be AJAXed?

I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) ...
2
votes
1answer
2k views

Strange robots.txt - how and why did it get there?

I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
1
vote
1answer
32 views

making friendly urls without making unfriendly url accessible to anyone

As many know, one can make a friendly URL with mod_rewrite in their .htaccess file like so: RewriteRule ^category/(.*)$ /somescript.php?category=$1 [L] This works as long as somescript.php is ...
2
votes
2answers
78 views

How do robots respond to no user-agent in robots.txt?

I'm just curious because I can't seem to find any mention of it anywhere. What is the standard (or at the very least typical) response to robots.txt without a user-agent parameter? Does it assume the ...
0
votes
0answers
65 views

can't remove blocks on my resources in google webmaster tools

I received an alert from Google Webmaster Tools a couple weeks ago that stated: "Google systems have recently detected an issue with your homepage that affects how well our algorithms render and ...