Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

1
vote
1answer
166 views

Google lists redirected URL instead of target URL

We've recently moved much of our site to a Web CMS which published to www1 rather than the old www. To clean up the search results, we've added an exclusion in robots.txt for the old hostname and ...
1
vote
3answers
114 views

robots.txt file Question

User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /account.php Disallow: /cart.php Is the above actually blocking crawlers? i just saw the extra "disallow" after the Google
1
vote
1answer
45 views

If I block a page in robots.txt would I still be able to see the page in my browser? [closed]

What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
1
vote
1answer
211 views

Do or can robots cause considerable performance issues?

At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website. Our setup: Lets say I have site ...
1
vote
2answers
1k views

Complex Disallow pattern in robots.txt

I have a URL like this: www.example.com/freelance-jobs-new-york I had a problem and many duplicated pages have been created like this: www.example.com/freelance-jobs-new-york-php-php ...
1
vote
1answer
1k views

Why google isn't updating my site title in search results? [duplicate]

Possible Duplicate: Google doesn't seem to update the description or title of my homepage I had my domain for few days before I uploaded site to it, and it had one title, and then when ...
1
vote
1answer
175 views

Should a single page fetcher like TinEye obey robots.txt?

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" ...
1
vote
1answer
114 views

What is USH.HTML and why do some robot try to access it?

I have been getting quite a few hits on my webservers for the resource /USH.HTML. I have never hosted this file, and do not know what it is. Is this a type of robots.txt file or something specific ...
1
vote
2answers
41 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
1answer
93 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
1
vote
2answers
65 views

Proper content for robots.txt on server maintenance page?

I know that the proper server response header should be 503 Service Unavailable, to let the search bots know that they shouldn't index the pages like Update in progress and Server maintenance. But ...
1
vote
1answer
122 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
1
vote
1answer
63 views

How to prevent search engine bots from crawling specific pages?

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it. Is it ...
1
vote
1answer
89 views

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?

I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Questions: If I "hide" these links via ...
1
vote
1answer
96 views

robots.txt being ignored - wordpress install

I just took over maintaining a website and noticed that Google gives "A description for this result is not available because of this site's robots.txt" So I went to look at the site's robots.txt file ...
1
vote
2answers
532 views

Google is indexing pages with a “noindex” robots meta tag

I have come across a very annoying problem. I have several pages that shouldn't be indexed, as they are basically pop-up pages and "thanks for signing up" pages. I have set them up with the robots ...
1
vote
1answer
78 views

Should I submit XML sitmaps directly or list them in my robots.txt?

Are there advantages to listing XML sitemap files in robots.txt vs submitting them directly to Google via Webmaster Tools?
1
vote
1answer
132 views

Should I have a separate sitemap for mobile pages and include that in my robots.txt?

I am working on site that has 20 mobile-friendly pages which are located under a /mobile directory. Should I have a separate sitemap for these mobile pages and include both this and the desktop ...
1
vote
1answer
42 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
1
vote
2answers
85 views

Stop Google from crawling my site (already blocked with robots.txt)

Annoyed of YouTube's music deletion in Germany, I started my own "YouTube clone" just for private use. It automatically downloads my subscriptions and some videos with special keywords or from special ...
1
vote
1answer
41 views

Robots.txt exclude certain urls and include others

I have a robots text file which needs to mass exclude certain urls, currently setup as so Disallow: /somestring Disallow: /*/somestring However, some of my URLs generated by users are coming back ...
1
vote
2answers
120 views

What is the empirical way to tell that Google has removed a URL from the index?

After merging two sites, one with ~40 URLs and one with ~700 URLs (due to a forum) the result was that overall traffic decreased by ~50%. I want to get the traffic back, as it was organic and a ...
1
vote
1answer
610 views

What should I do when the maximum sitemap size limit is reached according to Google Webmaster Tools?

We have an existing sitemap with the URL: http://example.com/sitemap/sitemap.xml It was submitted to Google Webmaster Tools a long time ago and almost all the pages were indexed, but recently we ...
1
vote
1answer
40 views

My site's robots.txt was blocking everyone for a year. What to do now that I fixed that?

My site's robots.txt was blocking everyone for a year (The "Discourage search engines from indexing this site" option was checked in WordPress). I guess Google will re-try to crawl it in the upcoming ...
1
vote
1answer
160 views

Google Webmaster Tools: Robots disallow does not seem to be working for staging site

I've added robots.txt on my staging site at staging.mydomain.com as: User-agent: * Disallow: / I then added and verified the staging site in Google Webmasters Tools. In Crawl > Blocked URLs, I can ...
1
vote
1answer
366 views

Sudden Increase in “Blocked from robots”

In the Index Status section of Google Webmaster Tools a site I manage has seen a sudden increase in pages "Blocked from robots". There's been no spike in Crawl Errors since this happened a week and ...
1
vote
1answer
75 views

Confused about redirect and robots.txt

I am updating a site by creating new pages and deleting old ones. To avoid crawling errors should I use both a robots.txt file and 301 redirect on .htaccess? robots.txt: Disallow: /products.php ...
1
vote
2answers
347 views

Need to encourage google to refresh robots.txt [duplicate]

My old robots.txt is User-agent: * Disallow: / Is blocking me for uploading a new sitemap and also blocking me for fetching robots.txt manually. I don't know what to do.
1
vote
2answers
300 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
1
vote
2answers
1k views

CPanel Visitors filled with bandwidth thieves

Issue: Strange and seemingly random entries on the CPanel Latest Visitors console, showing IPs accessing image files without any referring URL and not identified as Bots. Background: I have never had ...
1
vote
1answer
555 views

Preventing Twitterbot to access website

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And ...
1
vote
1answer
526 views

“Syntax not understood” error in Google Webmaster Tools robots.txt analysis when I use RewriteRule

In order to redirect my www.site.com to non-www, I employed this bit of code found somewhere on the web. RewriteCond %{HTTP_HOST} ^www.website.com [NC] RewriteRule ^(.*)$ http://website.com/$1 ...
1
vote
1answer
168 views

Remove Lots of index pages from google bot from same directory.

My question is about Google bot, seo and webmaster. may be this is not right place for ask, but if person not know so where he place his question he doesn't know!!! If you know my question's answer ...
1
vote
2answers
248 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
1
vote
2answers
122 views

Does specifying full webpage path in robots.txt would affect my website?

It may be a silly question but I need to clarify my doubt because it's related to robots.txt. I need to prevent some label path of my website using robots.txt file. User-agent: * Disallow: ...
1
vote
1answer
39 views

Prohibit search engines before going live

I want to prohibit search engines to index my web site before going live with it. If I simple do a robots.txt with: User-agent: * Disallow: / Would that cause any issues later when I decide to ...
1
vote
2answers
509 views

Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow

Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent. I decreased crawl rate from Google webmaster tools. But I'm supposing to ...
1
vote
2answers
253 views

Robots.txt and “Bad” Robots

I understand robots.txt and its purpose. I have read some people saying that using a Robots.txt gives "bad" robots or robots who do not obey a robots.txt a way to access pages on your site that you do ...
1
vote
2answers
525 views

How do I have Google not index any of my site, except my domain?

Disallow: / I want Google to index the index page, but nothing else. Will this achieve it?
1
vote
1answer
201 views

Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?

I'm not sure if this is a problem specific to how my hosting is structured or not. Let's say the main domain for my account is maindomain.com (for purpose of discussion). I am building other sites, ...
1
vote
2answers
559 views

Should a mobile-only site use robots.txt?

I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
1
vote
1answer
1k views

“Restricted by Robots.txt” in Google Webmaster Tools when no robots.txt exists

In Google Webmaster Tools, I have 832 errors in the sitemap with the error being "URL restricted by robots.txt". I don't have a robots.txt file, and the robots meta tag equals "index, follow". I did ...
1
vote
1answer
61 views

Blocking URLs and canonical questions

We are running a Magento store set up and was looking to block all pages with the exception of a select few. It seems the only way we can do this is with blocking direct paths to files that are in ...
1
vote
1answer
21 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: ...
1
vote
1answer
37 views

The Yandex index robot is redirected when requesting this site so the robots.txt file will not be used. What's the fix?

We have a website: www.example.com which redirects to https://www.example.com. Therefore when you type www.example.com/robots.txt you are redirected to https://www.example.com/robots.txt. Another ...
1
vote
1answer
32 views

Robots.txt disallow all except one directory and access to sitemap

robots.txt containts following instructions: User-agent: * Allow: /public/ Disallow: / Sitemap: http://sitedomain.com/sitemapindex.xml it seems that in this case sitemapindex won't be accessible ...
1
vote
1answer
57 views

Sitemap in different domain and robots.txt crawling

I have the following subdomains: www.example.com api.example.com The subdomain api.example.com is supposed to be invisible to Google. Its robots.txt had a disallow / directive. But I don't want ...
1
vote
2answers
103 views

Crawl Error on Disallowed Content

I've got a rule in my robots.txt file: # Crawlers Setup User-agent: * # Directories Disallow: /my_directory/ Yet I am getting entries in my Crawl Errors section of Google Webmaster Tools for this ...
1
vote
1answer
53 views

Configuring the document root on a cookieless domain

I am configuring a cookieless domain for all my static content, I see a lot of articles and how-to's use the same document root for as their main domain. Why? This seems like a disadvantage to me, ...
1
vote
1answer
564 views

How can Iframes be optimized for search engines?

If my site has an Iframe. What is the best way to optimize this for search engines? We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...