A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

27
votes
1answer
5k views

Bingbot request for trafficbasedsspsitemap.xml which does not exist

The logs for a website I manage show a request for a non-existent file by Bingbot. The details of the request are Path: /trafficbasedsspsitemap.xml Useragent: "Mozilla/5.0 (compatible; bingbot/2.0; ...
24
votes
4answers
938 views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...
21
votes
2answers
3k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
20
votes
6answers
3k views

Good tool to crawl my site and help me find dead link and unlinked files [closed]

I have a pretty big legacy site with literally thousands of PDFs that are sometimes accounting for in a database, but often are just links on the page, and are stored in most every directory on the ...
17
votes
4answers
4k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
16
votes
6answers
8k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
13
votes
3answers
3k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
12
votes
4answers
482 views

Status of Crawlable Ajax?

I saw that Google had a nice proposal/standard for making Ajax applications crawlable, via #! (hash bang). http://googlewebmastercentral.blogspot.com/2009/10/proposal-for-making-ajax-crawlable.html ...
11
votes
3answers
5k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
11
votes
2answers
898 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
10
votes
3answers
1k views

How does Google find a domain with no links to it?

I recently registered a new domain, pointed it to my existing server, and set up a minimal page just saying "test" and nothing else. I just discovered tonight that the page is already indexed in ...
10
votes
1answer
681 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
9
votes
1answer
6k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...
9
votes
3answers
1k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
9
votes
2answers
312 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9
votes
2answers
3k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
9
votes
8answers
6k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
9
votes
4answers
2k views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
9
votes
4answers
7k views

Adsense click bot is click bombing my site

I have a site that get's roughly 7,000 - 10,000 page views per day right now. Starting around 1 AM on 7/1/12 I noticed the CTR was rising dramatically. These clicks would be credited then ...
9
votes
2answers
266 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success ...
8
votes
6answers
8k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
8
votes
2answers
816 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
8
votes
3answers
721 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
8
votes
2answers
4k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
8
votes
2answers
6k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
8
votes
2answers
3k views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
8
votes
1answer
201 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
8
votes
0answers
521 views

Is there a search engine that indexes source code of a web-page? [closed]

I need to search the web for sites that are in our industry that use the same Adwords management company, to ensure that the said company is not violating our contract, as they have been accused of ...
7
votes
7answers
3k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
7
votes
2answers
794 views

@font-face, google fonts and SEO

Someone I know claims that search engines like Google have trouble with fonts that aren't the old standard fonts of yesteryear, resulting in pages not getting parsed by search spiders, and that to be ...
7
votes
2answers
570 views

Which meta “robots” tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
7
votes
2answers
462 views

Is it possible for web crawlers to see static pages without following a link to them?

If I create a static page on a domain (http://www.domain.com/page.html), can a crawler still see it if there aren't any links to it anywhere on the site?
7
votes
7answers
3k views

Is there an index of the IP addresses used by indexing bots?

I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list. Is ...
7
votes
3answers
112 views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
7
votes
1answer
13k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
6
votes
1answer
517 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
6
votes
1answer
120 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
6
votes
1answer
307 views

Will Google and other search engines follow IP address only sites (i.e., no URL)

Will Google and other search engines follow links to and index sites that don't have a URL, but instead just use their IP address's (obviously they would need a unique IP) ?
6
votes
1answer
3k views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
6
votes
3answers
155 views

How to handle SEO for brand that has a special character in the name

I am building a website (small online retail store) for a company whose brand name has a special character in it. The name of the brand is actually a legitimate Polish word that has an Ł, which is an ...
6
votes
1answer
248 views

List of all URI elements for a website

Does anyone know a good way of getting a list of all the URI elements for a website? I plan on moving a website to a new CMS and would like to setup some 301 redirects for articles, images, css, and ...
6
votes
2answers
233 views

Does the adsense bot ever get bored?

Each time someone clicks on a page like http://www.example.com/crazy.cgi?input=silly+banana where the "silly banana" bit represents some arbitrary user input which changes each time, I get yet ...
5
votes
5answers
2k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
5
votes
6answers
15k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
5
votes
2answers
200 views

Tell Google the list of URLs to crawl

Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find ...
5
votes
3answers
617 views

Can google crawl an URL (say some static HTML file) that has no pointing link? [duplicate]

Possible Duplicate: Is it possible for web crawlers to see static pages without following a link to them? I have some URLs (some pdfs and static HTML files) in my website that I want only ...
5
votes
1answer
215 views

Do I need robots.txt?

I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need to create it, what should it look ...
5
votes
2answers
140 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
5
votes
2answers
1k views

How often are sitemap.xml checked for updates by crawlers?

How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
5
votes
5answers
235 views

What measures can be taken to make sure Google is aware of the existence of a newly created page?

Consider a website with a large number of pages. New pages are published regularly. When publishing a new page the website operator wants to get the newly created paged crawled/indexed in Google as ...