Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
2
votes
1answer
30 views
How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?
I have this line:
Crawl-delay: 50
in the robots.txt file at the root of my website.
But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes.
I ...
2
votes
1answer
39 views
“Google couldn't access your site because of a server connectivity issue”
In Google Webmaster Tools on my dashboard I have a notice that appeared 2-3 days ago that says "Googlebot can't access your site".
Under the Current Status it has:
DNS - SUCCESS
Server Connectivity ...
2
votes
1answer
48 views
Prevent Duplicate content when using CloudFlare
I have a website which uses CloudFlare CDN. At the CDN setup, CloudFlare created a sub-domain like direct.example.com to my website, which can be used to override CloudFlare and access the site ...
1
vote
3answers
40 views
Googlebot is blocked from my domain
My robots.txt is:
User-agent: *
Disallow: /
Sitemap: http://example.com/sitemap.xml
Sitemap: http://example.com/sitemap.xml.gz
It has been the same for well over a week now, but when I go to ...
2
votes
2answers
40 views
How can an exception be created for Facebook on robots.txt?
I have a directory that I don't want Google to index because it could confuse people who don't know the site. However, I want everyone else to get updates when I post them on our Facebook page. The ...
2
votes
1answer
30 views
what should be my initial address for sitemap and robots.txt if I'm using redirection
What should be my initial adress when using this site: http://www.xml-sitemaps.com?
I have a folder inside my root where I have all my files.
For example I have domain.com and my start directory is ...
2
votes
3answers
62 views
Should I use both Robots.txt and META NAME=“ROBOTS”?
I am currently using the Robots.txt file (the standard one of joomla, to say). Now I have found that there are also specific HTML meta tags that specify desired robot behavior.
Is there a reason to ...
1
vote
1answer
44 views
GoogleBot can't reach my site
For some reason, the other day, out of nowhere, Google Webmaster Tools started showing all these errors while crawling my site, saying that it couldn't reach my pages. I checked my robots.txt and ...
1
vote
2answers
31 views
Robots.txt Blocking Blog Tags?
Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled?
User-agent: ...
1
vote
1answer
33 views
Prohibit search engines before going live
I want to prohibit search engines to index my web site before going live with it.
If I simple do a robots.txt with:
User-agent: *
Disallow: /
Would that cause any issues later when I decide to ...
0
votes
1answer
38 views
How to overcome duplicate URLS
I am having a website which is 3 years old, now I wanted to do some SEO chages into it. so, changed the some urls.
From: /psd-layouts.html to /psd-web-layouts.html
Now my question is that, I had ...
0
votes
2answers
53 views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress ...
2
votes
2answers
26 views
Question about duplicate groups (agents) in robots.txt
I'm looking at a corner case with robots.txt, and am curious if you have any guidance.
Basically, I'm in a situation where multiple people will be making edits to the same robots.txt file.
I've ...
0
votes
1answer
39 views
Addon domains and SEO
I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page.
Can I tell bots not to index the main domain WITHOUT ...
5
votes
4answers
191 views
Robots.txt vs Sitemap — Who wins in a Conflict
If I block off the directory /foo in robots.txt, but my xml sitemap contains URLs with /foo, will the URLs in the sitemap get picked up by Google and other search engines? In other words, does the ...
1
vote
2answers
105 views
Will 301 redirection to a Google Indexed URL can remove the URL from Google index?
I need to remove some indexed URLs from Google SERP. The problem is I cannot use the 3 types of removal patterns Google suggests like robots.txt, meta name "noindex" and remove the page and make it as ...
1
vote
3answers
98 views
Google Indexing Pages with noindex meta tag
I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs.
For some of the pages listed, there was nothing to ...
2
votes
1answer
105 views
Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?
As I understand, having repetitive content is a detriment to search engine placement.
Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
3
votes
2answers
185 views
Robots.txt on one line
Can I combine robots.txt rules on one line, separated by whitespace?
Will this work?
User-agent: * Disallow: /
1
vote
3answers
131 views
URL blocked by robots.txt in webmaster for blogger site
Remove: Disallow: /search
From where can i remove this line. I am using blogger and in webmaster it shows my many pages are blocked. From one of your answers i know that i need to remove this text ...
6
votes
2answers
74 views
What kind of redirect (301 or 302) for an email links tracker?
We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application ...
5
votes
1answer
219 views
should I block wordpress archive pages from search engines?
I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives.
Google has indexed the site fully.
Since the archives currently display full posts I've thought that I shouldn't ...
3
votes
2answers
126 views
Impact of blocking site search from googlebot, but listing content pages in sitemap.xml
We have a fairly large site 600k+ unique pages with various unique URLs along with a custom, local search interface for the site. The search interface has sorting and faceting capabilities to allow ...
3
votes
1answer
70 views
Need help with 200,000 + 404's
The business I work for recently migrated our e-commerce site from Celerant to Magento. Despite using a robots.txt file and a 301 redirect extension, webmaster tools is reporting that our "not found" ...
1
vote
2answers
125 views
Google Webmaster Tools complains about missing robots.txt
Nine days ago, I got a message Google Webmaster Tools:
Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt.
Well, but I don't have a robots.txt on ...
0
votes
1answer
501 views
SEO not indexing my site: “A description for this result is not available because of this site's robots.txt – learn more.” [duplicate]
When googling my site I get following text under the result for my domain:
"A description for this result is not available because of this site's
robots.txt – learn more."
My robots.txt, ...
2
votes
2answers
86 views
Will duplicate international (i18n) content hinder SEO rankings?
Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
4
votes
2answers
104 views
Preventing search engines to crawl on specific folders
I have set the robot.txt of my website as follow after seeing lots of errors in google's webmaster tools. Mainly because of the Smarty template files. So I changed my robots file as follow:
...
2
votes
2answers
205 views
Network unreachable: robots.txt unreachable
I am trying to add a valid sitemap to Google Webmaster. Yet, it says:
Network unreachable: robots.txt unreachableWe were unable to crawl
your Sitemap because we found a robots.txt file at ...
4
votes
3answers
72 views
SEO Removal Time?
I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here.
I believe it will be the solution but what i still need to ...
3
votes
4answers
512 views
Hide a Subdomain from Google/Search Engine SEO Result?
I have a subomain which I do not want to be listed in any search engine result. Let say I have:
http://www.example.com
http://child.example.com
How can I hide all URLs of child.example.com domain ...
0
votes
0answers
140 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
1
vote
0answers
79 views
Methods for keeping no-good scrapers off the site? [duplicate]
Possible Duplicate:
Tactics for dealing with misbehaving robots
I want to keep no-good scrapers (aka. bad bots) that steal content and consume bandwidth off my site. At the same time, I do ...
2
votes
2answers
156 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
2
votes
2answers
86 views
Prevent crawler that doesn't honoring robots.txt
I have some problem, when I try to write robots.txt for my site ...
I find some issues by search on Google, and tell me about honor and not honoring robots.txt, how I can prevent it, can I perform it ...
1
vote
2answers
134 views
Googlebot fetches my pages very frequent, rel-nofollow, meta-noindex or robots.txt-disallow
Googlebot fetches pages in my site very frequently. And this slowens my website. I don't want Googlebot to crawl too frequent.
I decreased crawl rate from Google webmaster tools.
But I'm supposing to ...
1
vote
3answers
498 views
Robots.txt never downloaded but some blocked URLs in GWT
There is something I don't understand in Google Webmaster Tools (GWT) for my Wordpress site.
In menu "Blocked URLs", it mention that my robots.txt has never been downloaded but there are some blocked ...
1
vote
2answers
86 views
Can search engine robots read file with permission 640?
I am on a shared web hosting linux server. I want search engine robots/spiders to be able to read the robots.txt but not any one typing www.mysite.com/robots.txt.
As per the following google group ...
1
vote
1answer
48 views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
0
votes
1answer
214 views
google webmaster showing warnings - blocked by robots.txt, but its not?
im getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt but its not, has any one come across this or a way to resolve this before ?
the ...
1
vote
1answer
40 views
If I block a page in robots.txt would I still be able to see the page in my browser? [closed]
What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
0
votes
0answers
262 views
blocking bad bots with robots.txt in 2012 [closed]
does it still work good? I have this:
# Generated using http://solidshellsecurity.com services
# Begin block Bad-Robots from robots.txt
User-agent: asterias
Disallow:/
User-agent: BackDoorBot/1.0
...
2
votes
3answers
165 views
Hiding php includes from search spiders?
Quick and simple question.
I have 80+ html files which I want to be crawled. They are individual product pages. Each of these pages calls its content using php includes. These php include files are ...
0
votes
1answer
49 views
If I blocked a directory, is a 301 error expected?
I blocked a directory in my site that is not secret but meant for internal discussions that are not relevant for search engine results. After running "Search as Google" at Google Webmaster Tools, the ...
0
votes
4answers
165 views
Can I include a robots meta tag outside of the head in HTML snippets indeded to be AJAXed?
I have a number of files in my site which are not intended for independent viewing, but rather to be AJAXed into content within the site. They obviously don't meet HTML standards (no body, head, etc.) ...
1
vote
1answer
116 views
How to disallow indexing but allow crawling?
In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles.
I want to disallow the front page to prevent duplicate content. ...
2
votes
2answers
104 views
How to get search engines to properly index an ajax driven search page
I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial ...
2
votes
1answer
274 views
Googlebot can't access my site when crawling from rootdomain
I can't explain why I get this message for my rootdomain result in Google :
trocmalin.com/
A description for this result is not available because of this site's robots.txt – learn more.
...
3
votes
3answers
111 views
SEO consideration for duplicate sites
I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...
1
vote
2answers
600 views
How to resolve “Google can't find your site's robots.txt” error?
I've recently found that "Google can't find your site's robots.txt" in crawl errors. When I tried Fetching as Google, I got result "SUCCESS", then I tried looking at crawl errors and it still shows ...
