Tagged Questions

Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.

learn more… | top users | synonyms

2
votes
1answer
33 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
1
vote
3answers
39 views

How much time for robots.txt to remove my website from google

I created a robots.txt file in my root but the website is still indexed by Google. How much time for robots.txt to remove my website from Google?
3
votes
2answers
57 views

I don't want my site to be analyzed on WooRank or builtwith.com

I don't want my site to be analyzed on WooRank or builtwith.com. Is there any way I can do that by editing the robots.txt file or any other possible way?
2
votes
3answers
157 views

Why is Disallow: /search in Blogger's robots.txt?

Can anyone tell me what does this mean in Blogger's "robots.txt" file? Do I need to edit anything in it? Should I remove /search from the Disallow: line? User-agent: Mediapartners-Google Disallow: ...
4
votes
3answers
201 views

Should we modify our Joomla robots.txt after Google's announcement on crawling of CSS and JavaScript?

Have come across an announcement from Google: http://googlewebmastercentral.blogspot.in/2014/10/updating-our-technical-webmaster.html It states: For optimal rendering and indexing, our new ...
11
votes
3answers
1k views

Robots.txt - CSS allow or disallow

Are CSS and templates allowed in robots.txt? Should it cause any problems? In Joomla, CSS and templates are disallowed in the robots.txt. Please help me find a solution to whether or not put ...
1
vote
2answers
84 views

Allow users to access my beta subdomain but pass link juice to the main site

My site can be accessed by either http://example.com or http://beta.example.com. The sub domain beta is the same site but a different version and I want users to be able to use the beta but any links ...
7
votes
7answers
3k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
1
vote
1answer
40 views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
2
votes
1answer
30 views

Robots.txt vs. Noindex Tags

According to matt cutts, even though we block the page using Robots.txt , it is better using noindex tag. ( Source : https://www.youtube.com/watch?v=KBdEwpRQRD0) If it is blocked by Robots.txt, how ...
0
votes
0answers
13 views

Site's pages hasn't been indexed for more than 10 days, why? [duplicate]

I checked out and robots.txt doesn't block the pages that I've been reviewing. The cached versions of the pages I reviewed in Google are made 10 days ago or more. What could be the cause that a site ...
3
votes
1answer
61 views

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain?

Can I place a robots.txt file inside a subdomain and it only affect that subdomain and not the root domain? For example: I have example.com and mysubdomain.example.com. mysite.com already has a ...
2
votes
1answer
69 views

How to Keep “Good” Bots That Ignore Robots & Nofollow From Getting Into Honeypots?

We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
0
votes
1answer
27 views

explicitly whitelist every page using robots.txt

I need to allow only specific pages at some site. Let's say: / /page1 /page2 ... Now I don't get how to whitelist the root page at robots.txt Disallow: / # block everything Allow: /page1 ...
0
votes
1answer
21 views

Comment Out Link Shows in Webmaster

I had "Privacy Policy" link on my old website. When I launched my new website, I put the link in footer but disallowed that to robots.txt by adding this code: <?php /*?><div ...
0
votes
3answers
85 views

Google is ignoring my robots.txt

Here is content of my robots.txt file: User-agent: * Disallow: /images/ Disallow: /upload/ Disallow: /admin/ As you can see, I explicitly disallowed all robots to index the folders images, upload ...
0
votes
1answer
36 views

Google Bot spending way too much following old 301 redirects

We changed our URL structure a couple of years ago but we still have a lot of backlinks pointing to old URLs. This causes Google bot to keep crawling old paths again and again and spending a lot of ...
3
votes
4answers
255 views

Should I use 'robots.txt' or 'sitemap.xml' to get dynamically created files indexed in search engines?

I've structure my video site like this: / (root) index.html -/videos -/video-id1 (dynamically generated subdirectory when user uploads a video) /index.html (dynamically generated file ...
2
votes
1answer
38 views

Is wildcard in Robots.txt in middle of string recognized?

I need some string for robots.txt like: disallow: /article/*/ but I don't know if this is a proper way to do this or not?! I need, for example, the following URLs to be followed: ...
2
votes
1answer
36 views

should you let search result page to be indexed

I have a pages of a catalogs, which contains a list of keywords of all products inside that catalogs, and I want to create a link from those keywords to a search page. For example, if the page ...
3
votes
1answer
64 views

Should I disallow(robots.txt) archive/author pages with links already available on the front page?

I am working on a simple Wordpress blog where when an article is published, it appears on ALL these pages: Homepage - Headline(clickable) + 3-line summary Parent category page - Headline(clickable) ...
4
votes
2answers
96 views

Recovering from an incorrectly deployed robots.txt?

We accidentally deployed a robots.txt from our development site that disallowed all crawling. This has caused traffic to dip dramatically, and google results to report: A description for this ...
0
votes
2answers
45 views

What is the empirical way to tell that Google has removed a URL from the index?

After merging two sites, one with ~40 URLs and one with ~700 URLs (due to a forum) the result was that overall traffic decreased by ~50%. I want to get the traffic back, as it was organic and a ...
0
votes
1answer
43 views

Will adding X-Robots-Tag:noindex remove an already indexed page from searchresults

I have a site that has already been indexed by google and shows in the search results. After it had already been indexed I added the X-Robots-Tag:noindex header to all responses. Is that enough for ...
4
votes
5answers
149 views

How do I get google to prefer one branch/directory of my site over another?

I write a software library that maintains "latest" and "beta" branches: gojs.net/latest gojs.net/beta Ideally all of this content would be available via Google search, so I'm reluctant to disallow ...
1
vote
3answers
56 views

How to handle URL parameter for mobile site

I have a phpBB forum with a skin for mobile users. Some javascript detects your device/resolution and sends you to the mobile skin or not. The URL's remain the same, it's based on a cookie. People ...
2
votes
2answers
67 views

How to add favicon to robots.txt

How can I disallow this URL: http://www.example.com/assets/ico/favicon.png? Would these lines in my robots.txt work? User-agent: * Disallow: /blah/assets* Disallow: /blah/assets/ico/
1
vote
1answer
93 views

What should I do when the maximum sitemap size limit is reached according to Google Webmaster Tools?

We have an existing sitemap with the URL: http://example.com/sitemap/sitemap.xml It was submitted to Google Webmaster Tools a long time ago and almost all the pages were indexed, but recently we ...
4
votes
3answers
603 views

How to disallow robots from the first 185 pages?

I have a website that whereby the first 185 pages are sample profiles for demonstration purpose: http://example.com/profile/1 ... http://example.com/profile/185 I want to block these pages from ...
1
vote
1answer
39 views

My site's robots.txt was blocking everyone for a year. What to do now that I fixed that?

My site's robots.txt was blocking everyone for a year (The "Discourage search engines from indexing this site" option was checked in WordPress). I guess Google will re-try to crawl it in the upcoming ...
3
votes
1answer
53 views

Robots.txt destroyed my ranking?

I had to do a massive URL change stuff (Categories and Products) on my e-commerce site while keeping 301 redirect of old URLs to new ones. I did change category URLs (appr: 800 URLs) to new and ...
1
vote
3answers
70 views

Google is not crawling and indexing my site after updating my robots.txt file

I have an issue with Google not being able to properly crawl my site. I have read other questions where people have had the same issue. I've tried to follow their solution of using this in my ...
2
votes
3answers
108 views

Will the already indexed URLs be removed if I use a robots.txt for my site?

Many URLs from my site have been added to Google's index, but a lot of them are outdated and they will never result in a 404 error and take the user to my site's homepage. I have submitted a new ...
2
votes
1answer
81 views

Hiding duplicate content - should I use meta tags or robots.txt?

Last week my site was utterly pummeled in Google's rankings - losing 95% of impressions overnight according to Google Webmaster Tools. It now only shows up if you search for the URL/site name itself. ...
5
votes
1answer
531 views

Robots.txt got changed to disallow crawling by accident

Our robots.txt mistakenly got changed to: User-agent: * Disallow: / This occurred about 2 weeks ago and was changed back yesterday to allow crawling. Our listed pages in the Googles have ...
1
vote
1answer
81 views

Google Webmaster Tools: Robots disallow does not seem to be working for staging site

I've added robots.txt on my staging site at staging.mydomain.com as: User-agent: * Disallow: / I then added and verified the staging site in Google Webmasters Tools. In Crawl > Blocked URLs, I can ...
11
votes
2answers
824 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
3answers
68 views

How do I remove a LOT of indexed pages from Google?

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
5
votes
4answers
518 views

Should I set NOINDEX header for my JS, CSS and image files?

Are there any harms if my site send NOINDEX headers for all my static assets? For image files, I refer to those valueless, e.g. background images, button images, etc. Update: more background ...
1
vote
1answer
120 views

Sudden Increase in “Blocked from robots”

In the Index Status section of Google Webmaster Tools a site I manage has seen a sudden increase in pages "Blocked from robots". There's been no spike in Crawl Errors since this happened a week and ...
3
votes
3answers
494 views

Is there a way to disallow crawling of only HTTPS in robots.txt?

I just realized that Bingbot is crawling my company's website's pages over https. Bing already crawls the site over http, so this seems frivolous. Is there a way to specify Disallow: / for https ...
1
vote
1answer
51 views

Confused about redirect and robots.txt

I am updating a site by creating new pages and deleting old ones. To avoid crawling errors should I use both a robots.txt file and 301 redirect on .htaccess? robots.txt: Disallow: /products.php ...
2
votes
2answers
93 views

No description for any page on the website is available in Google despite robots.txt allowing crawling

I seem to have the weirdest issue with Search Engine Optimization, and I asked the IT folks at my university, I asked people on Joomla forums and I am trying to sort this issue out using Google ...
1
vote
1answer
111 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
1
vote
2answers
94 views

Need to encourage google to refresh robots.txt [duplicate]

My old robots.txt is User-agent: * Disallow: / Is blocking me for uploading a new sitemap and also blocking me for fetching robots.txt manually. I don't know what to do.
2
votes
1answer
60 views

Robots and Sitemaps

I have some problems regarding indexing. Google indexed so much in the list. Now I wanted to remove everything from my site. If I have disallow all in robots.txt and send my sitemap.xml. Would ...
0
votes
1answer
54 views

Hiding page content from Google when it is duplicated across several sites

I would like to hide some content from Google. Because a client has multiple websites with the same content then he gets a warning about duplicate content on some of them (multiple countries, same ...
1
vote
1answer
398 views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
1
vote
1answer
86 views

CDN virtual subdomain causes duplicated content

I have created a subdomain and a CNAME record which points to the domain root. The subdomain www.static.example.com is actually a copy of the entire website www.example.com and it is supposed to act ...
0
votes
1answer
50 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...