Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
0answers
15 views

robots meta not blocking indexing

We have a staging version of our website to test changes on at trailheadpaddleshack.ca/staging1. This never appeared in search before. Recently the staging site has appeared on Google and is affecting ...
-4
votes
0answers
16 views

What is the effective way to block my web pages by using robots.txt file? [on hold]

I am posting this question, because i want to know about the effective use of robots.txt file to block few pages of my website from search engines. Here are the pages that i want to block from search ...
1
vote
1answer
53 views

robots.txt being ignored - wordpress install

I just took over maintaining a website and noticed that Google gives "A description for this result is not available because of this site's robots.txt" So I went to look at the site's robots.txt file ...
0
votes
3answers
40 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
1
vote
2answers
43 views

How should a robots.txt be for a wordpress site in order to be mobile-friendly ?

I run a site built on wordpress and my question kind of sums up what I am looking for. There has been an update on google according to which "mobile-friendliness" of a site is now considered as a ...
0
votes
1answer
20 views

Am I doing .htaccess right to block google, bing and alexa bots?

I do not need bots at all. I bought my traffic and I do not want my sites to show up on search engine or anything. So I added this: <IfModule mod_setenvif.c> # remove most popular bot ...
0
votes
1answer
33 views

Does the following robots.txt block all pages

I want to knew if the following robots.txt file block the whole pages from being indexed or not. User-agent: * Disallow: Disallow: /cgi-bin/
1
vote
2answers
235 views

Google is indexing pages with a “noindex” robots meta tag

I have come across a very annoying problem. I have several pages that shouldn't be indexed, as they are basically pop-up pages and "thanks for signing up" pages. I have set them up with the robots ...
1
vote
1answer
32 views

Should I submit XML sitmaps directly or list them in my robots.txt?

Are there advantages to listing XML sitemap files in robots.txt vs submitting them directly to Google via Webmaster Tools?
2
votes
1answer
44 views

What is the difference between Disallow: /? and Disallow: /*?* in robots.txt?

We have many parameters in our URL and I wonder what will do the parameters pasted below. Will they disallow all the parameters after ?? Please tell me what does this mean: Disallow: /? Disallow: ...
1
vote
0answers
43 views

Google ate all my bandwidth [closed]

Alright, I am new here and need some good help. I am running an e commerce build on magento called zumastic.com. The issue, I am facing is that google is eating a huge amount of my bandwidth. Last ...
2
votes
1answer
374 views

Is it possible to exclude Data-URIs (data:image/xxx) through robots.txt?

At the moment Google (I can not find similar informations for Bing, Yahoo!, etc.) does not index Data-URI. But until it is possible I like to know if it is possible to exclude Data-URIs of being ...
3
votes
1answer
74 views

Serving images on CDN; allow crawl of origin or CDN or both?

I have searched and read through a lot of questions regarding this subject, but cannot seem to find anyone that directly addresses my problem. I have a CDN for all my static images img.domain.net, ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
1
vote
2answers
92 views

Googlebot robots.txt access error on HTTPS when redirected from HTTP

This might be a stupid question, but I never came across this issue before and were not able to find any definite answers to this on the web: Our client migrated to HTTPS a few months ago, using ...
0
votes
0answers
26 views

How to design different web application environments with respect to SEO and server load?

Let's say I have a nice and valuable domain (somedomain.com) and I would like to develop a web application for this domain. I will host this web application on the cloud which I use currently for ...
2
votes
1answer
172 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
1
vote
0answers
38 views

Configuring the document root on a cookieless domain

I am configuring a cookieless domain for all my static content, I see a lot of articles and how-to's use the same document root for as their main domain. Why? This seems like a disadvantage to me, ...
1
vote
0answers
29 views

Canonical link or block in robots.txt for duplicate content across domains?

I've built a WordPress multisite for a dealership which has different franchises, ie Honda and BMW. Each franchise has its own website which have a used section, with identical content on them all for ...
1
vote
0answers
10 views

Rules for robots.txt file to exclude a whole subdomain from google's search results [duplicate]

In my website's root folder i already have a robots.txt file with some rules for the domain name www.example.com. My website is multilingual so i have setup with drupal de.example.com, fr.example.com ...
1
vote
1answer
61 views

Should I have a separate sitemap for mobile pages and include that in my robots.txt?

I am working on site that has 20 mobile-friendly pages which are located under a /mobile directory. Should I have a separate sitemap for these mobile pages and include both this and the desktop ...
9
votes
1answer
2k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...
1
vote
1answer
35 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
0
votes
2answers
302 views

Baiduspider is crawling my site even when forbidden by robots.txt, how do I prevent it? [duplicate]

My site has heavy traffic because some bot. I checked access_log, some bot Baiduspider go to my site 10-20 times per minute. I do not need Chinese traffic. I have searched and read ...
1
vote
2answers
71 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
1
vote
1answer
33 views

Does repetition of same rules for each major crawler bot makes sense in robots.txt?

SEO is a bit of arcane topic and sometimes I encounter strange practics which seemingly aren't backed up by any advice but still persists. Now I mean something like this: User-agent: Googlebot ...
1
vote
0answers
41 views

Bing is not crawling and indexing my mobile subdomain

BACKGROUND I have a mobile site located at a sub-domain of m.hydro-cleansing.com. The site is working fine and can be accessed. I am using a search API provided by Bing that should perform a site ...
1
vote
1answer
106 views

How can Iframes be optimized for search engines?

If my site has an Iframe. What is the best way to optimize this for search engines? We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
0
votes
1answer
29 views

surprising visitors on my wordpress site? [closed]

Hi I have a blog in wordpress where I plan to put some c programming tutorials never got time to finish it actually. but still site is live. Yesterday I received an email from my hosting that I might ...
11
votes
1answer
801 views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
1
vote
1answer
35 views

Robots.txt exclude certain urls and include others

I have a robots text file which needs to mass exclude certain urls, currently setup as so Disallow: /somestring Disallow: /*/somestring However, some of my URLs generated by users are coming back ...
3
votes
1answer
235 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
1
vote
3answers
92 views

How much time for robots.txt to remove my website from google

I created a robots.txt file in my root but the website is still indexed by Google. How much time for robots.txt to remove my website from Google?
3
votes
2answers
146 views

I don't want my site to be analyzed on WooRank or builtwith.com

I don't want my site to be analyzed on WooRank or builtwith.com. Is there any way I can do that by editing the robots.txt file or any other possible way?
2
votes
3answers
572 views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
4
votes
3answers
1k views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
11
votes
3answers
2k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
1
vote
2answers
116 views

Allow users to access my beta subdomain but pass link juice to the main site

My site can be accessed by either http://example.com or http://beta.example.com. The sub domain beta is the same site but a different version and I want users to be able to use the beta but any links ...
7
votes
7answers
3k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
1
vote
1answer
228 views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
2
votes
1answer
69 views

Robots.txt vs. Noindex Tags

According to matt cutts, even though we block the page using Robots.txt , it is better using noindex tag. ( Source : https://www.youtube.com/watch?v=KBdEwpRQRD0) If it is blocked by Robots.txt, how ...
0
votes
0answers
14 views

Site's pages hasn't been indexed for more than 10 days, why? [duplicate]

I checked out and robots.txt doesn't block the pages that I've been reviewing. The cached versions of the pages I reviewed in Google are made 10 days ago or more. What could be the cause that a site ...
3
votes
1answer
114 views

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain?

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain? For example: I have example.com and mysubdomain.example.com. mysite.com already has a ...
2
votes
1answer
264 views

How to Keep “Good” Bots That Ignore Robots & Nofollow From Getting Into Honeypots?

We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
0
votes
1answer
87 views

explicitly whitelist every page using robots.txt

I need to allow only specific pages at some site. Let's say: / /page1 /page2 ... Now I don't get how to whitelist the root page at robots.txt Disallow: / # block everything Allow: /page1 ...
0
votes
1answer
23 views

Comment Out Link Shows in Webmaster

I had "Privacy Policy" link on my old website. When I launched my new website, I put the link in footer but disallowed that to robots.txt by adding this code: <?php /*?><div ...
0
votes
3answers
100 views

Google is ignoring my robots.txt

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
0
votes
1answer
84 views

Google Bot spending way too much following old 301 redirects

We changed our URL structure a couple of years ago but we still have a lot of backlinks pointing to old URLs. This causes Google bot to keep crawling old paths again and again and spending a lot of ...
3
votes
4answers
460 views

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically generated subdirectory when user uploads a video) /index.html (dynamically generated file ...
2
votes
1answer
46 views

Is wildcard in Robots.txt in middle of string recognized?

I need some string for robots.txt like: disallow: /article/*/ but I don't know if this is a proper way to do this or not?! I need, for example, the following URLs to be followed: ...