Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
2answers
115 views

Where should the robots.txt file be?

If I have a subdomain: subdir.a.com that redirects to a.com/myfolder/ should the robots.txt file be in: subdir.a.com/robots.txt or in a.com/myfolder/robots.txt
1
vote
2answers
141 views

Googlebot robots.txt access error on HTTPS when redirected from HTTP

This might be a stupid question, but I never came across this issue before and were not able to find any definite answers to this on the web: Our client migrated to HTTPS a few months ago, using ...
1
vote
3answers
125 views

Google is not crawling and indexing my site after updating my robots.txt file

I have an issue with Google not being able to properly crawl my site. I have read other questions where people have had the same issue. I've tried to follow their solution of using this in my ...
1
vote
3answers
560 views

Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
1
vote
2answers
123 views

Handling duplicate content caused by 'new' smart-url structure

I used to have an URL structure like this: mydomain.com/search.php?key=value which I've since changed to: mydomain.com/key/value. I've changed all the internal links in my site, but Google ...
1
vote
1answer
149 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
1
vote
2answers
186 views

Robots.txt and specific file inclusion

Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
1
vote
3answers
102 views

How to handle URL parameter for mobile site

I have a phpBB forum with a skin for mobile users. Some javascript detects your device/resolution and sends you to the mobile skin or not. The URL's remain the same, it's based on a cookie. People ...
1
vote
3answers
87 views

Making a dynamic robots.txt but system is appending forward slash

I am making robots.txt dynamically instead of creating a physical robots.txt file. I am making this from the CMS and the system is appending a forward slash to every page. In the case of robots.txt, I ...
1
vote
1answer
34 views

Could the robots.txt file be overridden by another robots.txt in sub folder?

I know robots.txt is supposed to go in the root directory. But, could it be overridden by another robots.txt inside a sub directory?
1
vote
3answers
6k views

Remove subdomains from Google index and stop indexing them

I am running static content through a CDN, cdn1-cdn5 I am using such subdomains for that. I am loading just images, CSS and JS files this way, but apparently Google has indexed some pages on ...
1
vote
2answers
133 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
1
vote
1answer
154 views

Google lists redirected URL instead of target URL

We've recently moved much of our site to a Web CMS which published to www1 rather than the old www. To clean up the search results, we've added an exclusion in robots.txt for the old hostname and ...
1
vote
3answers
109 views

robots.txt file Question

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google
1
vote
1answer
44 views

If I block a page in robots.txt would I still be able to see the page in my browser? [closed]

What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
1
vote
1answer
186 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
1
vote
1answer
160 views

Should a single page fetcher like TinEye obey robots.txt?

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
1
vote
1answer
109 views

What is USH.HTML and why do some robot try to access it?

I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is. Is this a type of robots.txt file or something specific ...
1
vote
1answer
115 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
1
vote
1answer
41 views

How to prevent search engine bots from crawling specific pages?

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it. Is it ...
1
vote
1answer
55 views

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?

I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Questions: If I "hide" these links via ...
1
vote
1answer
73 views

robots.txt being ignored - wordpress install

I just took over maintaining a website and noticed that Google gives "A description for this result is not available because of this site's robots.txt" So I went to look at the site's robots.txt file ...
1
vote
2answers
277 views

Google is indexing pages with a “noindex” robots meta tag

I have come across a very annoying problem. I have several pages that shouldn't be indexed, as they are basically pop-up pages and "thanks for signing up" pages. I have set them up with the robots ...
1
vote
1answer
44 views

Should I submit XML sitmaps directly or list them in my robots.txt?

Are there advantages to listing XML sitemap files in robots.txt vs submitting them directly to Google via Webmaster Tools?
1
vote
1answer
78 views

Should I have a separate sitemap for mobile pages and include that in my robots.txt?

I am working on site that has 20 mobile-friendly pages which are located under a /mobile directory. Should I have a separate sitemap for these mobile pages and include both this and the desktop ...
1
vote
1answer
36 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
1
vote
2answers
76 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
1
vote
1answer
35 views

Robots.txt exclude certain urls and include others

I have a robots text file which needs to mass exclude certain urls, currently setup as so Disallow: /somestring Disallow: /*/somestring However, some of my URLs generated by users are coming back ...
1
vote
1answer
356 views

What should I do when the maximum sitemap size limit is reached according to Google Webmaster Tools?

We have an existing sitemap with the URL: http://example.com/sitemap/sitemap.xml It was submitted to Google Webmaster Tools a long time ago and almost all the pages were indexed, but recently we ...
1
vote
1answer
40 views

My site's robots.txt was blocking everyone for a year. What to do now that I fixed that?

My site's robots.txt was blocking everyone for a year (The "Discourage search engines from indexing this site" option was checked in WordPress). I guess Google will re-try to crawl it in the upcoming ...
1
vote
1answer
267 views

Sudden Increase in “Blocked from robots”

In the Index Status section of Google Webmaster Tools a site I manage has seen a sudden increase in pages "Blocked from robots". There's been no spike in Crawl Errors since this happened a week and ...
1
vote
1answer
63 views

Confused about redirect and robots.txt

I am updating a site by creating new pages and deleting old ones. To avoid crawling errors should I use both a robots.txt file and 301 redirect on .htaccess? robots.txt: Disallow: /products.php ...
1
vote
2answers
244 views

Need to encourage google to refresh robots.txt [duplicate]

My old robots.txt is User-agent: * Disallow: / Is blocking me for uploading a new sitemap and also blocking me for fetching robots.txt manually. I don't know what to do.
1
vote
2answers
216 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
1
vote
2answers
796 views

CPanel Visitors filled with bandwidth thieves

Issue: Strange and seemingly random entries on the CPanel Latest Visitors console, showing IPs accessing image files without any referring URL and not identified as Bots. Background: I have never had ...
1
vote
1answer
411 views

Preventing Twitterbot to access website

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And ...
1
vote
1answer
386 views

“Syntax not understood” error in Google Webmaster Tools robots.txt analysis when I use RewriteRule

In order to redirect my www.site.com to non-www, I employed this bit of code found somewhere on the web. RewriteCond %{HTTP_HOST} ^www.website.com [NC] RewriteRule ^(.*)$ http://website.com/$1 ...
1
vote
2answers
189 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
1
vote
2answers
115 views

Does specifying full webpage path in robots.txt would affect my website?

It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some label path of my website using robots.txt file. User-agent: * Disallow: ...
1
vote
1answer
38 views

Prohibit search engines before going live

I want to prohibit search engines to index my web site before going live with it. If I simple do a robots.txt with: User-agent: * Disallow: / Would that cause any issues later when I decide to ...
1
vote
2answers
458 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
1
vote
3answers
4k views

How to resolve “Google can't find your site's robots.txt” error?

I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
1
vote
2answers
222 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
1
vote
2answers
789 views

Complex Disallow pattern in robots.txt

I have a URL like this: www.example.com/freelance-jobs-new-york I had a problem and many duplicated pages have been created like this: www.example.com/freelance-jobs-new-york-php-php ...
1
vote
2answers
484 views

How do I have Google not index any of my site, except my domain?

Disallow: / I want Google to index the index page, but nothing else. Will this achieve it?
1
vote
2answers
550 views

Should a mobile-only site use robots.txt?

I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
1
vote
1answer
838 views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
1
vote
1answer
59 views

Blocking URLs and canonical questions

We are running a Magento store set up and was looking to block all pages with the exception of a select few. It seems the only way we can do this is with blocking direct paths to files that are in ...
1
vote
2answers
34 views

Crawl Error on Disallowed Content

I've got a rule in my robots.txt file: # Crawlers Setup User-agent: * # Directories Disallow: /my_directory/ Yet I am getting entries in my Crawl Errors section of Google Webmaster Tools for this ...