Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
42 votes
9 answers
34k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
WebbyTheWebbor's user avatar
26 votes
5 answers
6k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
user avatar
15 votes
4 answers
3k views

Prevent malicious bots from posting spam

I remember a site closed due to misuse and I wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
user avatar
15 votes
3 answers
3k views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
DisgruntledGoat's user avatar
10 votes
1 answer
2k views

How does "Noindex:" in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
Stephen Ostermiller's user avatar
21 votes
6 answers
5k views

Good tool to crawl my site and help me find dead link and unlinked files [closed]

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
MrChrister's user avatar
16 votes
7 answers
33k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
Zhianc's user avatar
  • 375
8 votes
1 answer
4k views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
Vaibhav Jalan's user avatar
5 votes
2 answers
4k views

How To Track Down and Stop Rogue Bots?

Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says: Unknown robot (identified by 'bot*') consumed 164 GB this month. By comparison, Googlebot ...
Itai's user avatar
  • 6,007
3 votes
2 answers
501 views

Google Search Console: 404 errors on existing pages

There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
neptune's user avatar
  • 173
25 votes
3 answers
13k views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
Howdy_McGee's user avatar
19 votes
3 answers
5k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
samthebrand's user avatar
12 votes
1 answer
19k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
yokoloko's user avatar
  • 405
12 votes
4 answers
10k views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
tonelot's user avatar
  • 191
9 votes
3 answers
984 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
Kris's user avatar
  • 4,487
7 votes
2 answers
3k views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
divided's user avatar
  • 183
5 votes
2 answers
2k views

Google reports 404 errors on URLs that haven't existed for six months and are no longer linked to on my site

My Google Webmaster Tools account, shows that I have crawling errors that are caused by links to pages that do not exist (404 not found). But when I look at the source pages that are accused of ...
webdeskil's user avatar
  • 195
5 votes
1 answer
263 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
BrunoLM's user avatar
  • 355
5 votes
2 answers
85 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
Alireza Mirian's user avatar
5 votes
2 answers
2k views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question is ...
CP3O's user avatar
  • 689
4 votes
1 answer
657 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
the's user avatar
  • 1,570
3 votes
1 answer
705 views

Googlebot craw rate is too slow for a huge site

I have a new website with 140k pages, according to my sitemaps. After submission of my sitemap index to Google, the Googlebot has started crawling the site a couple of hours ago, and at a steady pace,...
cnst's user avatar
  • 972
3 votes
2 answers
391 views

Does a "nofollow" attribute on a link prevent URL discovery by search engines?

I know that nofollow prevents link juice from being passed across a link. But if search engine robots discover a link with a nofollow on it, will they add that link to their crawl queue? In other ...
Stephen Ostermiller's user avatar
3 votes
2 answers
2k views

Do all search engines obey nofollow rel attribute on hyperlinks

I have been informed that search engine support is inconsistent for the rel attribute with the 'nofollow' value. Is this the case. From my research Google, Yahoo and Bing seem to obey this attribute ...
RyanP13's user avatar
  • 145
3 votes
3 answers
3k views

Does Google index images declared using 'srcset'?

As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
Undistraction's user avatar
3 votes
2 answers
1k views

How to prevent hotlinking for specific domains using .htaccess?

I'm being heavily spammed by image scraper sites. There are more than 150+ domains targeting my images & sending spammed links to my site. I just want to prevent hotlinking from them. I just want ...
Ankit Jha's user avatar
2 votes
3 answers
209 views

Why isn't Google crawling on my blog? [duplicate]

I have a blog. I made it in August, 2010. It has been awhile, but I am unable to make Google crawl on my site. I have requested Google to crawl but there is still no response. Are there any reasons of ...
user avatar
1 vote
1 answer
2k views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
Noam's user avatar
  • 2,008
1 vote
2 answers
681 views

Fake Links Showing up in Google Webmaster Tools

Google Webmaster tools is reporting a large increase in 'not found errors' on my site, and I see approximately 1700 'Not Found Errors over the past 2 months. All of them are for a url that is ...
xur17's user avatar
  • 113
34 votes
3 answers
10k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
Foo Bar's user avatar
  • 453
26 votes
4 answers
1k views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
Jeff Atwood's user avatar
  • 13.9k
16 votes
4 answers
5k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
Cesar's user avatar
  • 405
14 votes
8 answers
5k views

Is it possible to keep the existence of a domain secret?

If I bought a domain like a9j47fn83jd8j49.tld, and only a friend and I ever visit it, who would know about it? It sounds like for most TLDs, there is some reporting that has to happen after you buy a ...
dcc310's user avatar
  • 243
13 votes
6 answers
24k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
the's user avatar
  • 1,570
13 votes
4 answers
558 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...
philfreo's user avatar
  • 271
10 votes
8 answers
9k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
10 votes
1 answer
747 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
grautur's user avatar
  • 201
9 votes
2 answers
597 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
Mario Trucco's user avatar
9 votes
2 answers
1k views

Which meta "robots" tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
Tim Pietzcker's user avatar
9 votes
2 answers
4k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
SameOldNick's user avatar
9 votes
1 answer
7k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
Yin Yang's user avatar
  • 245
9 votes
11 answers
23k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
Codebeat's user avatar
  • 211
7 votes
2 answers
19k views

How to identify if IP address is really google's IP

From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with 74....
Mike -- No longer here's user avatar
7 votes
2 answers
1k views

Do search engines still crawl a noindex page

Do search engines crawl a page that has a 'noindex' meta attribute? The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex'...
GrapeSoda's user avatar
  • 1,187
7 votes
3 answers
36k views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
cnst's user avatar
  • 972
6 votes
1 answer
745 views

What is difference between robots.txt, sitemap, robots meta tag, robots header tag?

So I am trying to learn SEO and I am honestly confused and have following 8 questions. Do I tell a bot not to visit a certain link through X-Robots-Tag or through robot meta tag or robots.txt? Is it ...
Quillion's user avatar
  • 185
6 votes
2 answers
6k views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
Edgar Quintero's user avatar
6 votes
1 answer
409 views

How to get rid of crawling errors due to the URL Encoded Slashes (%2F) problem in Apache

The Google web crawler has indexed a whole set of URLs with encoded slashes (%2F) for our site. I assume it has picked up the pages from our XML sitemap file. The problem is that the live pages will ...
user14198's user avatar
5 votes
3 answers
2k views

Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources

I have a robots.txt file: User-agent:* Disallow:/path/page Disallow:/path/ Allow:/ The disallowed path is still getting crawled. I have searched this problem and what they said, order of precedence ...
arthur's user avatar
  • 51
5 votes
1 answer
3k views

Is there any reason to allow Yahoo! Slurp to crawl my site?

I thought a year or more ago Yahoo! would be using another search engine for results, and no longer using their own Slurp bot. However, a couple of the sites I manage Yahoo! Slurp continues to crawl ...
James Skemp's user avatar