Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

0
votes
0answers
2 views

Magento Robots.txt and blocking cached images

We have a Magento based store. Images are resized and served cached dynamically. When the image cache in Magento is cleared, there are generated new paths and files. And I get 404 errors for them ...
1
vote
5answers
788 views

Can I include a robots meta tag outside of the head in HTML snippets indeded to be AJAXed?

I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) ...
2
votes
1answer
2k views

Strange robots.txt - how and why did it get there?

I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
1
vote
1answer
23 views

making friendly urls without making unfriendly url accessible to anyone

As many know, one can make a friendly URL with mod_rewrite in their .htaccess file like so: RewriteRule ^category/(.*)$ /somescript.php?category=$1 [L] This works as long as somescript.php is ...
2
votes
2answers
35 views

How do robots respond to no user-agent in robots.txt?

I'm just curious because I can't seem to find any mention of it anywhere. What is the standard (or at the very least typical) response to robots.txt without a user-agent parameter? Does it assume the ...
0
votes
0answers
26 views

can't remove blocks on my resources in google webmaster tools

I received an alert from Google Webmaster Tools a couple weeks ago that stated: "Google systems have recently detected an issue with your homepage that affects how well our algorithms render and ...
0
votes
1answer
26 views

URLs with parameters & canonical tags. Block them via robots.txt vs GSC URL parameters?

I have some URLs with parameters, the parameters aren't of the filtering kind, they don't create millions of duplicates. The parameters are simple being used for tracking purpose, content doesn't ...
6
votes
3answers
477 views

Robots.txt for multiple domains with same website

I have the following three domains: example1.com (Disallow) example2.com (Disallow) example3.com (allow) All domain points to same folder which is public_html. How to disallow search ...
4
votes
3answers
8k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
0
votes
0answers
46 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
0
votes
1answer
47 views

Problem with Google Mobile Test - Blocking images

I have a situation when I check my Drupal 7 website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 56 resources ...
6
votes
2answers
393 views

80% of Website Traffic coming from Just one Town in USA - Strange?

For the last month, 80% of my total traffic was coming from just one Town in the USA. Here are the attributes: It is one of the richest towns in California in USA which defies logic since I run a ...
0
votes
1answer
83 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
0
votes
0answers
69 views

Google webmaster tool Severe health issues are found in your property

I'm getting this message in GWT : Severe health issues are found in your property , Some important page is blocked by robots.txt. My robots.txt User-agent: * Disallow: /wp-admin/ Sitemap: ...
0
votes
1answer
20 views

Robots.txt problem in subdirectory website (locale) and “/” character

We have two sites mainsite.com and mainsite.com/de where /de shows the german site. We rely on 1 robots.txt file. It contains all the disallow links for mainsite.com that we set-up and copied it to ...
1
vote
2answers
39 views

Crawl Error on Disallowed Content

I've got a rule in my robots.txt file: # Crawlers Setup User-agent: * # Directories Disallow: /my_directory/ Yet I am getting entries in my Crawl Errors section of Google Webmaster Tools for this ...
0
votes
1answer
40 views

Why is Google saying that the User-Agent line is invalid?

I have the following robots.txt file: User-agent: * Sitemap: https://m.mysite.com.au/sitemap.xml and when I use the Webmaster Tools to validate it I get it showing that the user agent line is ...
1
vote
0answers
43 views

Google mobile friendly resources which are blocked by robots.txt

I have a situation when I check my website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 5 resources which are ...
5
votes
4answers
383 views

What is the best configuration for robots.txt

What is the best initial or general setup for the robots.txt to allow search engines to go through the site, but maybe restrict a few folders? Is there a general setup that should always be used?
1
vote
1answer
117 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
0
votes
0answers
27 views

SEO: Deindex member profile pages?

I have a forum where users can register and automatically a profile page is created, like here in Stackexchange. The problem with these pages is that sometimes users creates accounts at different ...
2
votes
2answers
833 views

Restrict all robots except Googlebot and Yandexbot

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages. All other bots will not be accessed my pages. Is this robots.txt ...
1
vote
1answer
48 views

How to prevent search engine bots from crawling specific pages?

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it. Is it ...
1
vote
1answer
57 views

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?

I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Questions: If I "hide" these links via ...
0
votes
2answers
31 views

User-agent line needed in robots.txt?

I'm about to deploy a website for testing and I want to let all the indexing bots to not index my site during this phase. According to Google Webmasters Help section apparently you only need the ...
0
votes
1answer
29 views

“Verify live version” not available in Google Webmaster Tools

I was searching how to submit a robots.txt file using Google Webmaster Tools and I see no option of Verify live version, which is the forth step in submitting a robots.txt file to Google. I went ...
2
votes
2answers
79 views

robots meta not blocking indexing

We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting ...
1
vote
1answer
77 views

robots.txt being ignored - wordpress install

I just took over maintaining a website and noticed that Google gives "A description for this result is not available because of this site's robots.txt" So I went to look at the site's robots.txt file ...
3
votes
1answer
99 views

Serving images on CDN; allow crawl of origin or CDN or both?

I have searched and read through a lot of questions regarding this subject, but cannot seem to find anyone that directly addresses my problem. I have a CDN for all my static images img.domain.net, ...
0
votes
3answers
68 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
1
vote
2answers
69 views

How should a robots.txt be for a wordpress site in order to be mobile-friendly ?

I run a site built on wordpress and my question kind of sums up what I am looking for. There has been an update on google according to which "mobile-friendliness" of a site is now considered as a ...
2
votes
1answer
56 views

What is the difference between Disallow: /? and Disallow: /*?* in robots.txt?

We have many parameters in our URL and I wonder what will do the parameters pasted below. Will they disallow all the parameters after ?? Please tell me what does this mean: Disallow: /? Disallow: ...
0
votes
1answer
70 views

Am I doing .htaccess right to block google, bing and alexa bots?

I do not need bots at all. I bought my traffic and I do not want my sites to show up on search engine or anything. So I added this: <IfModule mod_setenvif.c> # remove most popular bot ...
0
votes
1answer
37 views

Does the following robots.txt block all pages

I want to knew if the following robots.txt file block the whole pages from being indexed or not. User-agent: * Disallow: Disallow: /cgi-bin/
1
vote
1answer
34 views

Could the robots.txt file be overridden by another robots.txt in sub folder?

I know robots.txt is supposed to go in the root directory. But, could it be overridden by another robots.txt inside a sub directory?
1
vote
2answers
296 views

Google is indexing pages with a “noindex” robots meta tag

I have come across a very annoying problem. I have several pages that shouldn't be indexed, as they are basically pop-up pages and "thanks for signing up" pages. I have set them up with the robots ...
3
votes
2answers
182 views

I don't want my site to be analyzed on WooRank or builtwith.com

I don't want my site to be analyzed on WooRank or builtwith.com. Is there any way I can do that by editing the robots.txt file or any other possible way?
1
vote
1answer
45 views

Should I submit XML sitmaps directly or list them in my robots.txt?

Are there advantages to listing XML sitemap files in robots.txt vs submitting them directly to Google via Webmaster Tools?
5
votes
3answers
2k views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
1
vote
4answers
1k views

URL blocked by robots.txt in webmaster for Blogger site

Remove: Disallow: /search From where can I remove this line? I am using Blogger and in webmaster it shows many of my pages are blocked. From the following Q&A, I know that I need to remove this ...
1
vote
0answers
48 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
2
votes
1answer
477 views

Is it possible to exclude Data-URIs (data:image/xxx) through robots.txt?

At the moment Google (I can not find similar informations for Bing, Yahoo!, etc.) does not index Data-URI. But until it is possible I like to know if it is possible to exclude Data-URIs of being ...
1
vote
2answers
807 views

Complex Disallow pattern in robots.txt

I have a URL like this: www.example.com/freelance-jobs-new-york I had a problem and many duplicated pages have been created like this: www.example.com/freelance-jobs-new-york-php-php ...
2
votes
5answers
11k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
8
votes
2answers
5k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
9
votes
1answer
517 views

What is a minimum valid robots.txt file?

I don't like that I see a lot of 404 errors in the access.log of my web server. I'm getting those errors because crawlers try to open a robots.txt file, but couldn't find any. So I want to place a ...
1
vote
2answers
165 views

Googlebot robots.txt access error on HTTPS when redirected from HTTP

This might be a stupid question, but I never came across this issue before and were not able to find any definite answers to this on the web: Our client migrated to HTTPS a few months ago, using ...
0
votes
0answers
30 views

How to design different web application environments with respect to SEO and server load?

Let's say I have a nice and valuable domain (somedomain.com) and I would like to develop a web application for this domain. I will host this web application on the cloud which I use currently for ...
2
votes
3answers
192 views

Should I use both Robots.txt and META NAME=“ROBOTS”?

I am currently using the Robots.txt file (the standard one of Joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior. Is there a reason to ...