Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

5
votes
1answer
128 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
0
votes
2answers
563 views

Will google remove the old caches if noarchive is added to robots.txt

If you want google's cache of your site to remain unchanged is it a good idea to add noarchive to robots.txt? Or will this wipe out your caches?
2
votes
1answer
366 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
6
votes
3answers
2k views

Can a sitemap index contain other sitemap indexes?

I have a bi-lingual website with a sitemap index for each language that links to several different sitemaps (one for videos, one for static content, and one for articles). I would like to introduce ...
1
vote
0answers
43 views

Configuring the document root on a cookieless domain

I am configuring a cookieless domain for all my static content, I see a lot of articles and how-to's use the same document root for as their main domain. Why? This seems like a disadvantage to me, ...
1
vote
0answers
31 views

Canonical link or block in robots.txt for duplicate content across domains?

I've built a WordPress multisite for a dealership which has different franchises, ie Honda and BMW. Each franchise has its own website which have a used section, with identical content on them all for ...
1
vote
0answers
10 views

Rules for robots.txt file to exclude a whole subdomain from google's search results [duplicate]

In my website's root folder i already have a robots.txt file with some rules for the domain name www.example.com. My website is multilingual so i have setup with drupal de.example.com, fr.example.com ...
1
vote
1answer
85 views

Should I have a separate sitemap for mobile pages and include that in my robots.txt?

I am working on site that has 20 mobile-friendly pages which are located under a /mobile directory. Should I have a separate sitemap for these mobile pages and include both this and the desktop ...
1
vote
2answers
883 views

CPanel Visitors filled with bandwidth thieves

Issue: Strange and seemingly random entries on the CPanel Latest Visitors console, showing IPs accessing image files without any referring URL and not identified as Bots. Background: I have never had ...
9
votes
1answer
3k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...
1
vote
1answer
37 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
0
votes
2answers
936 views

Baiduspider is crawling my site even when forbidden by robots.txt, how do I prevent it? [duplicate]

My site has heavy traffic because some bot. I checked access_log, some bot Baiduspider go to my site 10-20 times per minute. I do not need Chinese traffic. I have searched and read ...
1
vote
2answers
236 views

How does robots.txt handle links to disallowed pages?

We have affiliates that link to our site with a querystring. e.g. <a href="http://www.mysite.com/affiliate?id=123">Affiliate Link</a> In my robots.txt file, i have: Disallow: ...
3
votes
2answers
408 views

Will duplicate international (i18n) content hinder SEO rankings?

Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
1
vote
2answers
77 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
1
vote
1answer
36 views

Does repetition of same rules for each major crawler bot makes sense in robots.txt?

SEO is a bit of arcane topic and sometimes I encounter strange practics which seemingly aren't backed up by any advice but still persists. Now I mean something like this: User-agent: Googlebot ...
1
vote
0answers
53 views

Bing is not crawling and indexing my mobile subdomain

BACKGROUND I have a mobile site located at a sub-domain of m.hydro-cleansing.com. The site is working fine and can be accessed. I am using a search API provided by Bing that should perform a site ...
1
vote
2answers
1k views

Should I use “Disallow”, “NoFollow”, or “NoIndex” for paginated listing pages?

I own a web site which has mainly two types of pages: Content pages: they are pretty much static, and link to the previous and next content page (chronologically). The structure is that of a double ...
2
votes
3answers
879 views

Google Webmaster Tools complains about missing robots.txt

Nine days ago, I got a message Google Webmaster Tools: Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. Well, but I don't have a robots.txt on ...
1
vote
1answer
180 views

How can Iframes be optimized for search engines?

If my site has an Iframe. What is the best way to optimize this for search engines? We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
0
votes
1answer
31 views

surprising visitors on my wordpress site? [closed]

Hi I have a blog in wordpress where I plan to put some c programming tutorials never got time to finish it actually. but still site is live. Yesterday I received an email from my hosting that I might ...
10
votes
1answer
1k views

Can we use regex in robots.txt file to block URLs?

I have a few dynamic generated URLs. Can I use regex to block these URLs in a robots.txt file?
2
votes
2answers
115 views

Does the Instant Preview in Google Webmaster Tools take robots.txt in account?

Is that the way to go if I want to visually see what the googlebot sees? I'm trying to check a folder which I have just blocked in my robots.txt. If I fetch the folder as googlebot, it fetches ok, ...
1
vote
1answer
37 views

Robots.txt exclude certain urls and include others

I have a robots text file which needs to mass exclude certain urls, currently setup as so Disallow: /somestring Disallow: /*/somestring However, some of my URLs generated by users are coming back ...
2
votes
3answers
127 views

How to add favicon to robots.txt

How can I disallow this URL: http://www.example.com/assets/ico/favicon.png? Would these lines in my robots.txt work? User-agent: * Disallow: /blah/assets* Disallow: /blah/assets/ico/
3
votes
9answers
14k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
1
vote
1answer
145 views

Block search engines from the main domain while allowing add-on domains to rank well

I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page. Can I tell bots not to index the main domain WITHOUT ...
3
votes
1answer
293 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
2
votes
1answer
366 views

How to Keep “Good” Bots That Ignore Robots & Nofollow From Getting Into Honeypots?

We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
1
vote
1answer
303 views

Is it a problem for Google if robots.txt blocks URLs in the sitemap?

I had submitted my site's URL to Google Webmaster Tools. I am using OpenCart. I also have robots.txt file which is accessible to Googlebot. Now issue I am facing: Googlebot is blocked from accessing ...
1
vote
3answers
107 views

How much time for robots.txt to remove my website from google

I created a robots.txt file in my root but the website is still indexed by Google. How much time for robots.txt to remove my website from Google?
2
votes
3answers
762 views

Will the already indexed URLs be removed if I use a robots.txt for my site?

Many URLs from my site have been added to Google's index, but a lot of them are outdated and they will never result in a 404 error and take the user to my site's homepage. I have submitted a new ...
2
votes
3answers
806 views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
1
vote
4answers
2k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
11
votes
3answers
3k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
7
votes
7answers
3k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
1
vote
2answers
134 views

Allow users to access my beta subdomain but pass link juice to the main site

My site can be accessed by either http://example.com or http://beta.example.com. The sub domain beta is the same site but a different version and I want users to be able to use the beta but any links ...
12
votes
2answers
3k views

Why do Google search results include pages disallowed in robots.txt?

I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this: User-Agent: * Disallow: /email Yet I recently noticed that Google ...
1
vote
1answer
434 views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
2
votes
1answer
85 views

Robots.txt vs. Noindex Tags

According to matt cutts, even though we block the page using Robots.txt , it is better using noindex tag. ( Source : https://www.youtube.com/watch?v=KBdEwpRQRD0) If it is blocked by Robots.txt, how ...
6
votes
1answer
453 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
0
votes
0answers
16 views

Site's pages hasn't been indexed for more than 10 days, why? [duplicate]

I checked out and robots.txt doesn't block the pages that I've been reviewing. The cached versions of the pages I reviewed in Google are made 10 days ago or more. What could be the cause that a site ...
3
votes
1answer
155 views

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain?

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain? For example: I have example.com and mysubdomain.example.com. mysite.com already has a ...
4
votes
3answers
1k views

nofollow doesn't stop crawling?

In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" ...
31
votes
4answers
3k views

Files I should have on every website

I would like to have a list of files that should exist on every website. The idea is to please most browsers and bots and get down with the number of 404 messages in the logs. So far I have the ...
0
votes
1answer
173 views

explicitly whitelist every page using robots.txt

I need to allow only specific pages at some site. Let's say: / /page1 /page2 ... Now I don't get how to whitelist the root page at robots.txt Disallow: / # block everything Allow: /page1 ...
0
votes
1answer
25 views

Comment Out Link Shows in Webmaster

I had "Privacy Policy" link on my old website. When I launched my new website, I put the link in footer but disallowed that to robots.txt by adding this code: <?php /*?><div ...
2
votes
3answers
402 views

Is it OK to block my javascript files in robots.txt?

Is it OK to block my javascript files in robots.txt, or can it hurt my website SEO-wise?
0
votes
3answers
103 views

Google is ignoring my robots.txt

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
3
votes
4answers
510 views

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically generated subdirectory when user uploads a video) /index.html (dynamically generated file ...