All Questions
Tagged with web-crawlers robots.txt
201 questions
1
vote
3
answers
373
views
Applebot not crawling sitemap.xml
On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages.
We have no robots.txt restrictions for that bot, on the contrary we ...
1
vote
1
answer
273
views
Unexplainable non-existing page requested for months
While I am quite familiar with bots requesting well-know pages as used for WordPress and phpMyAdmin on any site the can reach, I'm now dealing with a bot requested a page I really can't explain.
For ...
- 111
7
votes
3
answers
38k
views
Do I really have to block MJ12Bot (as the prevailing visitor on my site)?
I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
- 972
3
votes
1
answer
699
views
How to disallow Google crawling out of my website through images?
I'm taking care of a website which has multiple images coming from exterior links.
The thing is, those links are outgoing ones, and I want to avoid that as much as possible.
I tough of one solution ...
- 133
3
votes
2
answers
652
views
Optional query string specification with "Disallow" in robots.txt
I'm creating a robots.txt file for my website which now has a lot of pages.
I'm disallowing all the pages that I do not require to crawl by so:
Disallow: /folder/file.aspx
I've quite a lot of ...
- 133
1
vote
1
answer
44
views
Homepage not being crawled by search engines [duplicate]
My website is using WordPress with the following robots.txt
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/cache
Disallow: /trackback
Disallow: /...
- 113
3
votes
1
answer
6k
views
Can I safely block Google AdsBot from spidering specific links?
My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different.
I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
- 281
2
votes
2
answers
73
views
How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?
After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
- 13.6k
5
votes
2
answers
2k
views
Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?
A lot of websites hide their sitemap.xml file.
What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file.
So my question is ...
- 689
2
votes
1
answer
756
views
cookies and useragent is not enough ? to emulate browser?
I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt
Let's take an example, I need to get this page:
http://www.aliexpress.com/...
- 21
18
votes
3
answers
12k
views
Are web crawlers able to find a secondary robots.txt in a sub-directory?
I have a sub-directory that I would like to hide from web crawlers.
One way to do this is to use a robots.txt in the root directory of the server and add a rule to exclude that sub-directory. However,...
user60182
2
votes
2
answers
6k
views
Is there a way to block wget from crawling and downloading my website?
I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
- 23
4
votes
1
answer
336
views
Using robots.txt to deny access to MediaWiki special pages using substring matching
I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
- 165
4
votes
3
answers
182
views
Google shows the wrong title for my website - robots.txt issue
Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows
this
in translation from Romanian it says:
"pages_rss_title"
I noticed the ...
1
vote
1
answer
2k
views
How to block the most popular spider crawlers via robots.txt?
I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots.
I want to disallow content and media (images, videos) crawlers.
The reason ...
- 160
2
votes
2
answers
146
views
Disallow Crawling of All Search Pages
Using robots.txt I am attempting to stop all crawling of search URLs
Disallow: /rest_of_url/search&tour*
Above is what I am using. Our URL looks like the following for all search results. ...
- 253
0
votes
0
answers
12
views
How to make sub-directories appear in Google Search Engine? [duplicate]
I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
- 101
3
votes
3
answers
1k
views
Using robots.txt to block sessionID URLs
I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
- 531
4
votes
3
answers
85
views
Securing images from browsers and humans
Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
- 544
3
votes
2
answers
5k
views
How to allow Googlebot and Bingbot to crawl mysite?
Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
- 33
10
votes
1
answer
2k
views
How does "Noindex:" in robots.txt work?
I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.
Disallow: /page-one.html
...
- 99.4k
2
votes
2
answers
6k
views
redirecting bots and crawlers to another but not human via htaccess
I would to apply this diagram via htaccess I tried a lots of codes but I failed every time
So I need to redirecting bots and crawlers especially from facebook via .htaccess
- 29
1
vote
1
answer
148
views
What are proper process to disallow a site from google crawl
I have done google and get the two solution for disallow whole a site from
google crawl.
1:
User-agent: *
Disallow:
2:
User-agent: *
Disallow: /
Now can anyone tell me which proper code ...
- 175
1
vote
1
answer
155
views
How to prevent search engine bots from crawling specific pages?
My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it.
Is it ...
- 235
1
vote
2
answers
587
views
Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?
I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages.
Questions:
If I "hide" these links via ...
- 1,619
0
votes
1
answer
668
views
Is it a good practice to block spam bots by IP ranges?
I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall.
Is ...
- 257
1
vote
3
answers
4k
views
Robot.txt disallow *?s=
Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ?
Disallow: *?s=
- 13
34
votes
3
answers
10k
views
Is there a difference between an empty robots.txt and no robots.txt at all?
On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all.
Do crawlers behave differently in ...
- 453
9
votes
1
answer
7k
views
How can I stop Google from indexing embedded iframes?
I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
- 245
12
votes
2
answers
20k
views
Only allow Google and Bing bots to crawl a site
I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site.
User-...
- 223
1
vote
1
answer
74
views
Robots: block /lang/page from the index but keep /page
My URL structure looks a bit like this:
/group1/
/group2/
/group3/
Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL:
/...
- 190
2
votes
1
answer
2k
views
How can Iframes be optimized for search engines?
If my site has an Iframe. What is the best way to optimize this for search engines?
We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
4
votes
1
answer
674
views
Semalt ignores robots.txt, does their own form actually do what they promise?
Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess.
I just found out that they're also have a ...
- 1,600
7
votes
7
answers
4k
views
Will banning bots make a site harder to find on search engines?
I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing.
I want to parse the logs and get accurate statistics as to how many ...
- 175
15
votes
7
answers
26k
views
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
...
- 1,600
2
votes
1
answer
2k
views
How to Keep "Good" Bots That Ignore Robots & Nofollow From Getting Into Honeypots?
We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
- 3,339
11
votes
2
answers
1k
views
What bots are really worth letting onto a site?
Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site?
My first ...
- 2,080
1
vote
3
answers
219
views
How do I remove a LOT of indexed pages from Google?
A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
- 11
1
vote
1
answer
422
views
Is this a good WordPress robots.txt configuration?
I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
- 3,658
5
votes
1
answer
6k
views
Redirect Google crawler to different robots.txt via .htaccess
I have googled for the answer all day and still couldn't find an answer.
I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
- 61
0
votes
1
answer
108
views
Does hosting the pre-release website on domain affect SEO?
Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
- 175
6
votes
1
answer
767
views
robots.txt blocking specific files also block unnecessary URLs
I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as
# Files
User-agent: *
Disallow: /mage
But this ...
- 301
0
votes
1
answer
2k
views
Google Webmaster Tool: “Severe health issues are found on your site” [closed]
I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?"
message on Google Webmaster Tools.
This is what I have in robots.txt:
User-agent: *
Allow: /*?p=...
- 103
6
votes
2
answers
6k
views
Serve a different robots.txt file for every site hosted in the same directory
We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
- 381
1
vote
2
answers
446
views
What is better for SEO robots.txt file or dynamically generated robots.txt URL?
I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
- 669
3
votes
1
answer
948
views
Google bots and iframe
Here is the problem:
Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains:
Disallow: /article/
...
- 53
11
votes
1
answer
2k
views
Combine user-agents in robots.txt
Can user-agents be listed together, followed by their common rules like so in a robots.txt?
User-agent: Googlebot
User-agent: ia_archiver #Alexa
User-agent: BingPreview #Microsoft
User-agent: bingbot ...
- 395
1
vote
1
answer
85
views
Why would I use a robots.txt file? [duplicate]
From what I understand after reading Google's Controlling Crawling and Indexing:
The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
- 111
4
votes
3
answers
8k
views
Hide a site entirely from search engines (Google, Bing etc.)
My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc.
However, the websites still need to be accessible for our customers, and ...
- 161
1
vote
1
answer
131
views
Will there be a conflict between my Sitemap.xml and robots.txt?
I have a robots.txt with the following in it:
User-agent: *
Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=*
Sitemap: http://tests.compkerworld.com/...
- 11