Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
5
votes
2answers
1k views
Why do Google search results include pages disallowed in robots.txt?
I have some pages on my site that I want to keep search engines away from, so I disallowed them in my robots.txt file like this:
User-Agent: *
Disallow: /email
Yet I recently noticed that Google ...
9
votes
3answers
269 views
Robots.txt: do I need to disallow a page which is not linked anywhere?
There are some pages on my websites that I want the user to be able to visit only if I give him/her the url.
If I disallow the single pages in robots.txt, they will be visible by anybody looking into ...
8
votes
4answers
418 views
Robots denied by domain is still listed in search results
So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question).
However, if ...
4
votes
1answer
131 views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
3
votes
3answers
4k views
How to stop certain urls from being indexed
When I type site:example.com (using my domain obviously),
I get several link errors showing up in the listing. Typically, they are of the form:
www.example.com/some/fixed/path/admin/unblockUser/11
...
0
votes
1answer
147 views
Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?
I'm not sure if this is a problem specific to how my hosting is structured or not.
Let's say the main domain for my account is maindomain.com (for purpose of discussion)
I am building other sites, ...
5
votes
1answer
223 views
should I block wordpress archive pages from search engines?
I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives.
Google has indexed the site fully.
Since the archives currently display full posts I've thought that I shouldn't ...
3
votes
4answers
515 views
Hide a Subdomain from Google/Search Engine SEO Result?
I have a subomain which I do not want to be listed in any search engine result. Let say I have:
http://www.example.com
http://child.example.com
How can I hide all URLs of child.example.com domain ...
2
votes
9answers
4k views
Need to Stop Bots from Killing my Webserver
I am having EXTREME bot problems on the some of my websites within my hosting account. The bots utilize over 98% of my CPU resources and 99% of my bandwidth for my entire hosting account. These bots ...
8
votes
3answers
2k views
Does Google cache robots.txt?
I added a robots.txt file to one of my sites a week ago, which should have prevented googlebot from attempting to fetch certain URLs. However, this weekend I can see googlebot loading those exact ...
6
votes
4answers
309 views
robots.txt not updated
I have updated some url's and files in robots.txt file to block url's and files from google search results but, still files displaying in search results. As per a suggestion from a site I tried to ...
5
votes
3answers
274 views
Does loading images from a server that disallows robots affect SEO?
Lets say a page on example.com has all of its images coming from img.example.com:
<img src="http://img.example.com/keyword.jpg" alt="keyword"/>
while robots.txt on img.example.com disallows ...
4
votes
3answers
72 views
SEO Removal Time?
I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here.
I believe it will be the solution but what i still need to ...
4
votes
3answers
548 views
nofollow doesn't stop crawling?
In my website, most of the pages has a linkedin sharing button written in this way:
<li>
<a id="share_linkedin"
...
2
votes
2answers
156 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
2
votes
3answers
4k views
How to remove old robots.txt from google as old file block the whole site
I have a website which still shows old robots.txt in the google webmaster tools.
User-agent: *
Disallow: /
Which is blocking Googlebot. I have removed old file updated new robots.txt file with ...
2
votes
2answers
4k views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
0
votes
0answers
142 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
0
votes
1answer
214 views
google webmaster showing warnings - blocked by robots.txt, but its not?
im getting a warning when submitting my sitemap to webmaster tools it saying that my pages are blocked by robots.txt but its not, has any one come across this or a way to resolve this before ?
the ...
