Questions tagged [web-crawlers]
A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)
97
questions
42
votes
9
answers
34k
views
Preventing robots from crawling specific part of a page
As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
26
votes
5
answers
6k
views
Does the Google spider render JavaScript?
Does the Google spider knows how to render JavaScript, or only HTML?
15
votes
4
answers
3k
views
Prevent malicious bots from posting spam
I remember a site closed due to misuse and I wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
15
votes
3
answers
3k
views
How does Google find a domain with no links to it?
I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else.
I just discovered tonight that the page is already indexed in ...
10
votes
1
answer
2k
views
How does "Noindex:" in robots.txt work?
I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt.
Disallow: /page-one.html
...
21
votes
6
answers
5k
views
Good tool to crawl my site and help me find dead link and unlinked files [closed]
I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
16
votes
7
answers
33k
views
How to block baidu spiders
Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
8
votes
1
answer
4k
views
For how long does the google bot wait after document ready before taking a snapshot of the webpage?
As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ?
Suppose I have an AJAX response that takes longer ...
5
votes
2
answers
4k
views
How To Track Down and Stop Rogue Bots?
Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says:
Unknown robot (identified by 'bot*') consumed 164 GB this month.
By comparison, Googlebot ...
3
votes
2
answers
501
views
Google Search Console: 404 errors on existing pages
There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
25
votes
3
answers
13k
views
SEO - Responsive Website and Duplicated Menus
Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
19
votes
3
answers
5k
views
Is it possible to slow the Baiduspider crawl frequency?
Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy."
I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
12
votes
1
answer
19k
views
Why does baidu crawl my site like crazy
When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks.
It's not that i care much about it but I'm really curious about why he's doing ...
12
votes
4
answers
10k
views
Google doesn't crawl CDN files
I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website.
Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
9
votes
3
answers
984
views
Tactics for dealing with misbehaving robots
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.
Obviously we've had ...
7
votes
2
answers
3k
views
Is it possible for web crawlers to see static pages without following a link to them?
If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
5
votes
2
answers
2k
views
Google reports 404 errors on URLs that haven't existed for six months and are no longer linked to on my site
My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found).
But when I look at the source pages that are accused of ...
5
votes
1
answer
263
views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
5
votes
2
answers
85
views
Should we still care about ajax-crawling specification?
It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site.
Are they still valid and necessary to be followed?
5
votes
2
answers
2k
views
Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?
A lot of websites hide their sitemap.xml file.
What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file.
So my question is ...
4
votes
1
answer
657
views
Semalt ignores robots.txt, does their own form actually do what they promise?
Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess.
I just found out that they're also have a ...
3
votes
1
answer
705
views
Googlebot craw rate is too slow for a huge site
I have a new website with 140k pages, according to my sitemaps.
After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady pace,...
3
votes
2
answers
391
views
Does a "nofollow" attribute on a link prevent URL discovery by search engines?
I know that nofollow prevents link juice from being passed across a link. But if search engine robots discover a link with a nofollow on it, will they add that link to their crawl queue?
In other ...
3
votes
2
answers
2k
views
Do all search engines obey nofollow rel attribute on hyperlinks
I have been informed that search engine support is inconsistent for the rel attribute with the 'nofollow' value.
Is this the case.
From my research Google, Yahoo and Bing seem to obey this attribute ...
3
votes
3
answers
3k
views
Does Google index images declared using 'srcset'?
As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
3
votes
2
answers
1k
views
How to prevent hotlinking for specific domains using .htaccess?
I'm being heavily spammed by image scraper sites. There are more than 150+ domains targeting my images & sending spammed links to my site. I just want to prevent hotlinking from them. I just want ...
2
votes
3
answers
209
views
Why isn't Google crawling on my blog? [duplicate]
I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
1
vote
1
answer
2k
views
Sudden decrease in pages crawled per day
During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
1
vote
2
answers
681
views
Fake Links Showing up in Google Webmaster Tools
Google Webmaster tools is reporting a large increase in 'not found errors' on my site, and I see approximately 1700 'Not Found Errors over the past 2 months. All of them are for a url that is ...
34
votes
3
answers
10k
views
Is there a difference between an empty robots.txt and no robots.txt at all?
On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all.
Do crawlers behave differently in ...
26
votes
4
answers
1k
views
Is Yahoo Search the same as Bing Search now?
We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
16
votes
4
answers
5k
views
Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?
I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
14
votes
8
answers
5k
views
Is it possible to keep the existence of a domain secret?
If I bought a domain like a9j47fn83jd8j49.tld, and only a friend and I ever visit it, who would know about it? It sounds like for most TLDs, there is some reporting that has to happen after you buy a ...
13
votes
6
answers
24k
views
How to properly (dis)allow the archive.org bot? Did things change, if so when?
I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this:
User-agent: *
Disallow: /
...
13
votes
4
answers
558
views
Status of Crawlable Ajax?
I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang).
http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html
...
10
votes
8
answers
9k
views
How long did it take for your new website to be indexed by search engines [duplicate]
Possible Duplicate:
Why isn't my website in Google search results?
I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
10
votes
1
answer
747
views
How did Google manage to crawl my 403 pages?
I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
9
votes
2
answers
597
views
Should we drop AJAX crawling scheme?
So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9
votes
2
answers
1k
views
Which meta "robots" tag gets preference?
My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
9
votes
2
answers
4k
views
Ask.com sitemap crawler down (for good)?
I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
9
votes
1
answer
7k
views
How can I stop Google from indexing embedded iframes?
I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
9
votes
11
answers
23k
views
Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?
Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
7
votes
2
answers
19k
views
How to identify if IP address is really google's IP
From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with 74....
7
votes
2
answers
1k
views
Do search engines still crawl a noindex page
Do search engines crawl a page that has a 'noindex' meta attribute?
The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex'...
7
votes
3
answers
36k
views
Do I really have to block MJ12Bot (as the prevailing visitor on my site)?
I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
6
votes
1
answer
745
views
What is difference between robots.txt, sitemap, robots meta tag, robots header tag?
So I am trying to learn SEO and I am honestly confused and have following 8 questions.
Do I tell a bot not to visit a certain link through X-Robots-Tag or through robot meta tag or robots.txt?
Is it ...
6
votes
2
answers
6k
views
Serve a different robots.txt file for every site hosted in the same directory
We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
6
votes
1
answer
409
views
How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache
The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
5
votes
3
answers
2k
views
Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources
I have a robots.txt file:
User-agent:*
Disallow:/path/page
Disallow:/path/
Allow:/
The disallowed path is still getting crawled.
I have searched this problem and what they said, order of precedence ...
5
votes
1
answer
3k
views
Is there any reason to allow Yahoo! Slurp to crawl my site?
I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...