Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

5
votes
1answer
270 views

How to fix thousands of 404's?

The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
5
votes
1answer
687 views

wget not respecting my robots.txt. Is there an interceptor?

I have a website where I post csv files as a free service. Recently I have noticed that wget and libwww have been scraping pretty hard and I was wondering how to circumvent that even if only a little. ...
5
votes
1answer
117 views

Handling SEO for Infinite pages that cause external slow API calls

I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
4
votes
3answers
647 views

600 visitors per day, 20 backlinks but still not referenced by Google

i've launch a website on wednesday (9 August) There's already in 4 days 12,000 viewed pages 1,400 visits 20 to 25 backlincks a sitemap.xml (130 URL) english language / french language - url like ...
4
votes
3answers
8k views

How to stop certain urls from being indexed

When I type site:example.com(using my domain obviously), I get several link errors showing up in the listing. Typically, they are of the form: /some/fixed/path/admin/unblockUser/11 I am thinking of ...
4
votes
5answers
412 views

With Robots.txt disallow all, why was my site still getting traffic?

In failing to check my robots.txt on a community site, it had: User-agent: * Disallow: / I only clued into this because of a message in my AdSense account saying that five URL's couldn't be ...
4
votes
9answers
16k views

Need to Stop Bots from Killing my Webserver

I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
4
votes
2answers
1k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
4
votes
3answers
627 views

How to disallow robots from the first 185 pages?

I have a website that whereby the first 185 pages are sample profiles for demonstration purpose: http://example.com/profile/1 ... http://example.com/profile/185 I want to block these pages from ...
4
votes
2answers
136 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
4
votes
4answers
3k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
4
votes
3answers
451 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
4
votes
2answers
5k views

Is my whole site is blocked by this Disallow: /

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site. User-agent: Baiduspider Disallow: / User-agent: * Disallow: ...
4
votes
1answer
995 views

Does the line-ending format of robots.txt matter?

Simple question: Should I make sure to use Unix line endings for my robots.txt, or does it not matter?
4
votes
2answers
750 views

What is the correct way to write my “robots.txt” file?

I have written the following code inside my robots.txt file: User-Agent: Googlebot Disallow: User-agent: Mediapartners-Google Disallow: Sitemap: http://example.com/sitemap.xml Is ...
4
votes
3answers
358 views

Bad Bot blocking Revisited

I've read a lot about bad bot blocking, php scripts, .htaccess techniques, etc... Is this a valid method? Since .htacces can rewrite and send a bad bot a 403 deny or forward to something like spam ...
4
votes
2answers
579 views

Should I prevent search engines indexing tag/category pages?

On my site, I currently have no special rules for search engines. It is a blog, statically generated using a Python program. When I search for some of my articles on Google, there is usually a tag or ...
4
votes
2answers
353 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
4
votes
2answers
125 views

Do I need a robots.txt file if most of my site's pages require authorization?

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization. Does that mean I need to have a robots.txt file to Disallow these pages with ...
4
votes
3answers
1k views

nofollow doesn't stop crawling?

In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" ...
4
votes
2answers
179 views

How to temporarily remove a website from search engines?

We are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain. Now we are already being crawled and indexed by bots (Google, Yahoo, and others) ...
4
votes
5answers
110 views

removing robots.txt

Will removing robots.txt make the website 'searchable' again. Curious if you had a robots.txt to avoid indexing, but changed your mind and wanted to get indexed.
4
votes
2answers
133 views

Recovering from an incorrectly deployed robots.txt?

We accidentally deployed a robots.txt from our development site that disallowed all crawling. This has caused traffic to dip dramatically, and google results to report: A description for this ...
4
votes
3answers
353 views

Is it safe to block redirected (but still linked) URLs with robots.txt?

I have a website that has all URLs optimized and 301 redirected from nasty URLs to clean ones. However, everywhere throughout the site the unclean URLs are linked in menus, content, products, etc. ...
4
votes
2answers
363 views

Impact of blocking site search from googlebot, but listing content pages in sitemap.xml

We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
4
votes
2answers
85 views

robots.txt - just a guess about wild-card

so if I disallow tempPage, does it mean tempPage_1, temp_Page_2, tempPage_x are also disallowed? I tried to google this up, but I don't know...
4
votes
3answers
60 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
4
votes
1answer
684 views

How do I deal with content scrapers? [duplicate]

Possible Duplicate: How to protect SHTML pages from crawlers/spiders/scrapers? My Heroku (Bamboo) app has been getting a bunch of hits from a scraper identifying itself as GSLFBot. Googling ...
4
votes
1answer
434 views

Should I block search engines with robots.txt on my Facebook app?

There's a few subdomains on my homepage where I tak advantage of robots.txt to block search engines for indexing their contents, like my images and downloads subdomains as I don't want the direct URLs ...
4
votes
4answers
101 views

How do I disallow a specific query string in robots.txt?

I have the URL http://www.example.com/shopping/books/?b=9 and the following robots.txt file: User-agent: * Disallow: /?b=9 But when I test this in Google Webmaster Tool's robots.txt tester it ...
4
votes
1answer
453 views

Prevent Duplicate content when using CloudFlare

I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
3
votes
3answers
2k views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
3
votes
4answers
599 views

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically generated subdirectory when user uploads a video) /index.html (dynamically generated file ...
3
votes
5answers
145 views

Are indexed Wordpress admin pages a security threat?

Because it was blocking Googlebot from accessing important files, I recently updated our /robots.txt file and removed most of it, including wp-admin. Now, the people on charge of hosting and SEO ...
3
votes
2answers
15k views

Allow all robots robots.txt

In my robots.txt file I have a list of robots that are not allowed to be indexed on my site and for the rest I have to allow all other robots, but I would like to know the real difference between ...
3
votes
2answers
281 views

How can I allow search engines to index my invite only website in ruby on rails?

I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication: before_filter ...
3
votes
3answers
129 views

Do crawlers require a robots.txt file before indexing content?

Do you need a robots.txt file for the crawlers to index your site, or it is just to disallow any content that you don't want crawlers to index?
3
votes
3answers
135 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
3
votes
6answers
171 views

How do you prevent web pages from appearing in the Google search index?

I have some web pages that Google has indexed. I have now added a robots.txt file to exclude them, but should I also add a noindex tag? If so, then how do I do so?
3
votes
3answers
1k views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
3
votes
4answers
2k views

Google Indexing Pages with noindex meta tag

I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs. For some of the pages listed, there was nothing to ...
3
votes
1answer
11k views

How do I disallow an entire directory with robots.txt?

I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example http://someurl.com/first_url ...
3
votes
3answers
776 views

Google is indexing unintended content that is either unpublished or a secret part of site

Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M. How is that even possible? Only thing I can ...
3
votes
3answers
506 views

PageRank: will links pointing to pages protected by robots.txt still count?

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e. ...
3
votes
1answer
357 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
3
votes
1answer
213 views

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain?

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain? For example: I have example.com and mysubdomain.example.com. mysite.com already has a ...
3
votes
2answers
153 views

Sitemap specified in robots.txt not found by Google

I would like Google to find my sitemap.xml via robots.txt, rather than having to manually upload it. My robots.txt looks like this: User-agent: * Disallow: Sitemap: ...
3
votes
2answers
592 views

Grapeshot crawler ignoring robots.txt

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
3
votes
2answers
3k views

How can an exception be created for Facebook on robots.txt?

I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
3
votes
3answers
139 views

SEO consideration for duplicate sites

I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...