Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
3
votes
1answer
70 views
Need help with 200,000 + 404's
The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
3
votes
1answer
87 views
Handling SEO for Infinite pages that cause external slow API calls
I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
2
votes
2answers
1k views
Robots.txt and Regular Expressions
If I place a regular expression in robots.txt will it be readable for the web crawlers?
2
votes
9answers
4k views
Need to Stop Bots from Killing my Webserver
I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
2
votes
2answers
207 views
Network unreachable: robots.txt unreachable
I am trying to add a valid sitemap to Google Webmaster. Yet, it says:
Network unreachable: robots.txt unreachableWe were unable to crawl
your Sitemap because we found a robots.txt file at ...
2
votes
2answers
217 views
robots.txt and disallow
Which one is correct?
User-agent: *
Disallow: index.php
or
User-agent: *
Disallow: /index.php
inorder to block all pages like index.php?day=7 index.php?day=15 index.php?day=30
2
votes
1answer
205 views
Stop bots from crawling old links with extensions
I've recently switched to MVC3 which is extension-less for the URL's, but Google and Bing have a wealth of links that they are crawling which no longer exist.
So I'm trying to find out if there is a ...
2
votes
2answers
241 views
Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?
Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
2
votes
3answers
293 views
Google is indexing unintended content that is either unpublished or a secret part of site
Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M.
How is that even possible? Only thing I can ...
2
votes
3answers
165 views
Hiding php includes from search spiders?
Quick and simple question.
I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are ...
2
votes
1answer
177 views
How do I disallow robots to access links on a certain port?
I have a private web server set up on a non-conventional port (say 6677). All people can have access to the whole site's urls from 6677. The links by mysite:6677 are also indexed by Google, which is ...
2
votes
2answers
364 views
Image Hot-links & Google's Instant Previews
I'm trying to adjust my site:
1. stop Image Hot-links
2. block Googlebot-Image (because I do not want my images to be searchable)
3. allow Google's Instant Previews
I've tried this but the Instant ...
2
votes
2answers
121 views
Does Yahoo Slurp crawl hammer a site?
A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
2
votes
2answers
120 views
Pointing to Google that a specific website is for mobiles
I have 2 websites: let's call them A and B.
A is a Desktop optimized website.
B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A
How ...
2
votes
1answer
93 views
Adsense and blocking robots from web pages
If I use a meta tag to make a page 'noindex' does this mean that my adsense ads will not be targeted?
2
votes
1answer
235 views
How do I disallow an entire directory with robots.txt?
I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example
http://someurl.com/first_url
...
2
votes
3answers
250 views
After adding *.xls to robots.txt how long will the results continue to show up in google
Is there an estimated amount of time before the document will not appear in google results?
We have a public records document. One person raised a concern when they saw their "PUBLIC INFORMATION" on ...
2
votes
1answer
34 views
How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.
I ...
2
votes
1answer
40 views
“Google couldn't access your site because of a server connectivity issue”
In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site".
Under the Current Status it has:
DNS - SUCCESS
Server Connectivity ...
2
votes
1answer
30 views
what should be my initial address for sitemap and robots.txt if I'm using redirection
What should be my initial adress when using this site: http://www.xml-sitemaps.com?
I have a folder inside my root where I have all my files.
For example I have domain.com and my start directory is ...
2
votes
2answers
87 views
Prevent crawler that doesn't honoring robots.txt
I have some problem, when I try to write robots.txt for my site ...
I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
2
votes
2answers
104 views
How to get search engines to properly index an ajax driven search page
I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial ...
2
votes
1answer
148 views
Edit robots.txt on google sites
Getting many 404 errors on my website for the following URLs:
/system/app/pages/admin/revisions*
I was going to put Disallow in robots.txt, but it seems there is no way to edit robots.txt in google ...
2
votes
1answer
393 views
Why am I getting this message “Some important page is blocked by robots.txt”?
My site's URL is: www.hackinguniverse.org
From some day in Google Webmaster tool a message is showing that says "Some important page is blocked by robots.txt".
My robots.txt is:
User-agent: ...
2
votes
2answers
2k views
robots.txt for a CodeIgniter based website?
I am working on a php site that uses Code Igniter framework and has the following folder structure on the server.
public_html
application
public
sql
style
system
design
images
...
2
votes
1answer
1k views
Search engines that do _not_ respect robots.txt?
Are there any search engines that do not care about robots.txt?
2
votes
2answers
40 views
How can an exception be created for Facebook on robots.txt?
I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
2
votes
1answer
107 views
Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?
As I understand, having repetitive content is a detriment to search engine placement.
Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
2
votes
2answers
156 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
2
votes
1answer
274 views
Googlebot can't access my site when crawling from rootdomain
I can't explain why I get this message for my rootdomain result in Google :
trocmalin.com/
A description for this result is not available because of this site's robots.txt – learn more.
...
2
votes
3answers
4k views
How to remove old robots.txt from google as old file block the whole site
I have a website which still shows old robots.txt in the google webmaster tools.
User-agent: *
Disallow: /
Which is blocking Googlebot. I have removed old file updated new robots.txt file with ...
2
votes
1answer
304 views
HTTP 303 redirection and robots.txt
On a site I'm working on, we're using the HTTP 303 redirect pattern (see this article for background) to distinguish between information and non-information resources. So: some URL's under /id get ...
2
votes
1answer
173 views
What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]
Possible Duplicate:
How to Remove URLs from Google Search Engine
In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
2
votes
1answer
438 views
Prevent crawling of mobile site
I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users.
The mobile content is in /mobile/
The regular site will redirect mobile users to the ...
2
votes
2answers
129 views
Robots.txt syntax -> Can this be done?
When creating a robots.txt, are the following the same?
# Slurp
User-agent: Slurp
Disallow: /
and
# Slurp User-agent: Slurp Disallow: /
It seems like for someone disallowing a large number of ...
2
votes
1answer
131 views
How can I let robots access my site?
I have a webpage on my university's server, which means I cannot access the root directory and change files there.
Is there some other way to give access to robots to my page?
Thanks.
2
votes
3answers
544 views
What is a good robots.txt for WP?
What is the "best" setup for robots.txt?
I'm using the following permalink structure in Wordpress: /%category%/%postname%/.
My robots.txt currently looks like this (copied from somewhere a long time ...
2
votes
2answers
4k views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
2
votes
1answer
49 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
2
votes
3answers
62 views
Should I use both Robots.txt and META NAME=“ROBOTS”?
I am currently using the Robots.txt file (the standard one of joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.
Is there a reason to ...
2
votes
2answers
26 views
Question about duplicate groups (agents) in robots.txt
I'm looking at a corner case with robots.txt, and am curious if you have any guidance.
Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file.
I've ...
2
votes
2answers
88 views
Will duplicate international (i18n) content hinder SEO rankings?
Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
2
votes
2answers
162 views
Would a 403 page, a blank page and a page redirect be indexed by search robots?
I am wondering if a disallow directive needs to be issued to robots on pages that will generate:
a 403 forbidden error;
a blank page, or;
a page redirect (to log in page for example).
Would not ...
1
vote
2answers
275 views
robots.txt ignored
I'm working on a joomla website, and robots.txt is messing with me... i have it in domain.com/website/robots.txt and I just wanna prevent users from browsing directories.
By default it had Disallow: ...
1
vote
2answers
62 views
Serious Googlebot spike
One of the sites I maintain has seen googlebot use 23gb of bandwidth this month.
This seems utterly mental to me, what can I do about it?
I have reduced the crawl rate on webmaster tools, and ...
1
vote
6answers
77 views
Uploading a non-finished website
I have a pretty basic question. I developed a neet little website wich I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
1
vote
3answers
357 views
Preventing Spiders on One Subdomain?
We currently host multiple sites on the same domain that all live in the same physical directory on the server. Each is database driven and lives at a unique subdomain. For example, site1.example.com, ...
1
vote
3answers
108 views
robots.txt file Help needed
What if I put same robots.txt file in multiple directories and folders?
User-agent: *
Disallow: /otherdirname/dirname2/
Can it'll affect anyhow ??
And how much time it generally takes to refine ...
1
vote
2answers
118 views
Problem with my robots.txt
I want to allow a certain html file and my site's index file to be indexed by search engines. Everything else should be disallowed. My home directory does not actually contain an index file, I am ...
1
vote
1answer
1k views
Strange robots.txt - how and why did it get there?
I recently created a very simple, pure HTML website which I have hosted with "hostmonster". Hostmonster had very good reviews on some comparison website and in general so far they appear to be ...
