Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

0
votes
0answers
26 views

Google webmaster tool Severe health issues are found in your property

I'm getting this message in GWT : Severe health issues are found in your property , Some important page is blocked by robots.txt. My robots.txt User-agent: * Disallow: /wp-admin/ Sitemap: ...
0
votes
1answer
18 views

Robots.txt problem in subdirectory website (locale) and “/” character

We have two sites mainsite.com and mainsite.com/de where /de shows the german site. We rely on 1 robots.txt file. It contains all the disallow links for mainsite.com that we set-up and copied it to ...
0
votes
1answer
31 views

Why is Google saying that the User-Agent line is invalid?

I have the following robots.txt file: User-agent: * Sitemap: https://m.mysite.com.au/sitemap.xml and when I use the Webmaster Tools to validate it I get it showing that the user agent line is ...
0
votes
0answers
35 views

Google mobile friendly resources which are blocked by robots.txt

I have a situation when I check my website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 5 resources which are ...
0
votes
1answer
36 views

Problem with Google Mobile Test - Blocking images

I have a situation when I check my Drupal 7 website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 56 resources ...
1
vote
1answer
115 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
0
votes
0answers
23 views

SEO: Deindex member profile pages?

I have a forum where users can register and automatically a profile page is created, like here in Stackexchange. The problem with these pages is that sometimes users creates accounts at different ...
1
vote
2answers
34 views

Crawl Error on Disallowed Content

I've got a rule in my robots.txt file: # Crawlers Setup User-agent: * # Directories Disallow: /my_directory/ Yet I am getting entries in my Crawl Errors section of Google Webmaster Tools for this ...
1
vote
1answer
41 views

How to prevent search engine bots from crawling specific pages?

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it. Is it ...
1
vote
1answer
55 views

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?

I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Questions: If I "hide" these links via ...
0
votes
1answer
72 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
0
votes
1answer
26 views

“Verify live version” not available in Google Webmaster Tools

I was searching how to submit a robots.txt file using Google Webmaster Tools and I see no option of Verify live version, which is the forth step in submitting a robots.txt file to Google. I went ...
0
votes
2answers
27 views

User-agent line needed in robots.txt?

I'm about to deploy a website for testing and I want to let all the indexing bots to not index my site during this phase. According to Google Webmasters Help section apparently you only need the ...
2
votes
2answers
70 views

robots meta not blocking indexing

We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting ...
1
vote
1answer
73 views

robots.txt being ignored - wordpress install

I just took over maintaining a website and noticed that Google gives "A description for this result is not available because of this site's robots.txt" So I went to look at the site's robots.txt file ...
0
votes
3answers
60 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
1
vote
2answers
69 views

How should a robots.txt be for a wordpress site in order to be mobile-friendly ?

I run a site built on wordpress and my question kind of sums up what I am looking for. There has been an update on google according to which "mobile-friendliness" of a site is now considered as a ...
0
votes
1answer
58 views

Am I doing .htaccess right to block google, bing and alexa bots?

I do not need bots at all. I bought my traffic and I do not want my sites to show up on search engine or anything. So I added this: <IfModule mod_setenvif.c> # remove most popular bot ...
0
votes
1answer
37 views

Does the following robots.txt block all pages

I want to knew if the following robots.txt file block the whole pages from being indexed or not. User-agent: * Disallow: Disallow: /cgi-bin/
1
vote
2answers
277 views

Google is indexing pages with a “noindex” robots meta tag

I have come across a very annoying problem. I have several pages that shouldn't be indexed, as they are basically pop-up pages and "thanks for signing up" pages. I have set them up with the robots ...
1
vote
1answer
44 views

Should I submit XML sitmaps directly or list them in my robots.txt?

Are there advantages to listing XML sitemap files in robots.txt vs submitting them directly to Google via Webmaster Tools?
2
votes
1answer
53 views

What is the difference between Disallow: /? and Disallow: /*?* in robots.txt?

We have many parameters in our URL and I wonder what will do the parameters pasted below. Will they disallow all the parameters after ?? Please tell me what does this mean: Disallow: /? Disallow: ...
1
vote
0answers
47 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
2
votes
1answer
458 views

Is it possible to exclude Data-URIs (data:image/xxx) through robots.txt?

At the moment Google (I can not find similar informations for Bing, Yahoo!, etc.) does not index Data-URI. But until it is possible I like to know if it is possible to exclude Data-URIs of being ...
3
votes
1answer
90 views

Serving images on CDN; allow crawl of origin or CDN or both?

I have searched and read through a lot of questions regarding this subject, but cannot seem to find anyone that directly addresses my problem. I have a CDN for all my static images img.domain.net, ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
1
vote
2answers
141 views

Googlebot robots.txt access error on HTTPS when redirected from HTTP

This might be a stupid question, but I never came across this issue before and were not able to find any definite answers to this on the web: Our client migrated to HTTPS a few months ago, using ...
0
votes
0answers
30 views

How to design different web application environments with respect to SEO and server load?

Let's say I have a nice and valuable domain (somedomain.com) and I would like to develop a web application for this domain. I will host this web application on the cloud which I use currently for ...
2
votes
1answer
304 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
1
vote
0answers
41 views

Configuring the document root on a cookieless domain

I am configuring a cookieless domain for all my static content, I see a lot of articles and how-to's use the same document root for as their main domain. Why? This seems like a disadvantage to me, ...
1
vote
0answers
30 views

Canonical link or block in robots.txt for duplicate content across domains?

I've built a WordPress multisite for a dealership which has different franchises, ie Honda and BMW. Each franchise has its own website which have a used section, with identical content on them all for ...
1
vote
0answers
10 views

Rules for robots.txt file to exclude a whole subdomain from google's search results [duplicate]

In my website's root folder i already have a robots.txt file with some rules for the domain name www.example.com. My website is multilingual so i have setup with drupal de.example.com, fr.example.com ...
1
vote
1answer
78 views

Should I have a separate sitemap for mobile pages and include that in my robots.txt?

I am working on site that has 20 mobile-friendly pages which are located under a /mobile directory. Should I have a separate sitemap for these mobile pages and include both this and the desktop ...
9
votes
1answer
3k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...
1
vote
1answer
36 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
0
votes
2answers
702 views

Baiduspider is crawling my site even when forbidden by robots.txt, how do I prevent it? [duplicate]

My site has heavy traffic because some bot. I checked access_log, some bot Baiduspider go to my site 10-20 times per minute. I do not need Chinese traffic. I have searched and read ...
1
vote
2answers
76 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
1
vote
1answer
35 views

Does repetition of same rules for each major crawler bot makes sense in robots.txt?

SEO is a bit of arcane topic and sometimes I encounter strange practics which seemingly aren't backed up by any advice but still persists. Now I mean something like this: User-agent: Googlebot ...
1
vote
0answers
50 views

Bing is not crawling and indexing my mobile subdomain

BACKGROUND I have a mobile site located at a sub-domain of m.hydro-cleansing.com. The site is working fine and can be accessed. I am using a search API provided by Bing that should perform a site ...
1
vote
1answer
155 views

How can Iframes be optimized for search engines?

If my site has an Iframe. What is the best way to optimize this for search engines? We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
0
votes
1answer
31 views

surprising visitors on my wordpress site? [closed]

Hi I have a blog in wordpress where I plan to put some c programming tutorials never got time to finish it actually. but still site is live. Yesterday I received an email from my hosting that I might ...
10
votes
1answer
1k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
1
vote
1answer
35 views

Robots.txt exclude certain urls and include others

I have a robots text file which needs to mass exclude certain urls, currently setup as so Disallow: /somestring Disallow: /*/somestring However, some of my URLs generated by users are coming back ...
3
votes
1answer
279 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
1
vote
3answers
102 views

How much time for robots.txt to remove my website from google

I created a robots.txt file in my root but the website is still indexed by Google. How much time for robots.txt to remove my website from Google?
3
votes
2answers
168 views

I don't want my site to be analyzed on WooRank or builtwith.com

I don't want my site to be analyzed on WooRank or builtwith.com. Is there any way I can do that by editing the robots.txt file or any other possible way?
2
votes
3answers
730 views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
5
votes
3answers
2k views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
11
votes
3answers
3k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
1
vote
2answers
127 views

Allow users to access my beta subdomain but pass link juice to the main site

My site can be accessed by either http://example.com or http://beta.example.com. The sub domain beta is the same site but a different version and I want users to be able to use the beta but any links ...