A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
2answers
32 views

Can I limit content scrapers by counting the number of hits from an IP?

I would like to ban aggressive scrapers that access x number of pages in an hour, say 1000. I plan to implement such a check via fail2ban and count the hits from the same ip via access logs. Should I ...
3
votes
4answers
745 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
0answers
5 views

How does google discover new hosts? [duplicate]

Assume I register a new host, foobar.com, and assume that Google doesn't know about foobar.com at all. Never in the history of internet was foobar.com mentioned in any web page. The two ways I know ...
1
vote
2answers
61 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
1
vote
2answers
281 views

What are spiders trying to achieve on my site by fetching lots of non-existant urls such as various versions of phpMyAdmin?

I've recently set up a build server that compiles Java source on Github webhooks. I created a public page for downloading latest .jars with a top level domain and I'm logging requests to it. What are ...
4
votes
2answers
527 views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
2
votes
1answer
66 views

Adsense crawler login for dynamic page urls

I have a website that largely features dynamic content. Every piece of dynamic content has it's own url, such as "www.example.com/page/*/*" The problem I have is that many of these pages require a ...
1
vote
1answer
113 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
1
vote
2answers
62 views

Private folders which are not indexable

I have a requirement where I need to distribute about 400 copies of a pdf document (about 6 MB) and .pub files, that are personalized for each user. Every user must have access to only his/her set of ...
1
vote
3answers
239 views

500 error returns website content

I'm using CloudFlare on my shared host. Recently all search engines and is-it-up sites have been getting a 500 error. I'm not sure who's to blame, my host or CloudFlare. I don't remember changing ...
8
votes
4answers
238 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
1
vote
5answers
10k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
0
votes
1answer
88 views

Is it OK to use translations obtained via crawling a translation service with advertising?

Google and Bing not only have their API-based translation, but also their websites which allow for one-off translation. While these are intended for use by humans, this also allows for a webcrawler ...
8
votes
2answers
4k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
0
votes
0answers
13 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
6
votes
1answer
8k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
4
votes
1answer
70 views

If I dynamically add a robots meta tag (via JS) will Google and other search engines recognize it?

I want to have a chunk of JS in the middle of my page insert a meta-robots element into the <head> based on some arbitrary condition. Will Google and other search engines recognize this tag the ...
5
votes
1answer
109 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
7
votes
2answers
405 views

Which meta “robots” tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
2
votes
2answers
676 views

Robots meta tag with “noimageindex”

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
2
votes
1answer
130 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
2
votes
2answers
651 views

Fake links cause crawl error in Google Webmaster Tools

Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message: Google detected a significant increase in the number of URLs that return a 404 (Page ...
5
votes
2answers
326 views

Google Webmaster Tools reports fake 404 errors

I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...
2
votes
1answer
596 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
2
votes
1answer
453 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
3
votes
2answers
62 views

How to set crawl rate in Google Webmaster Tools for seven days or a month?

My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...
2
votes
2answers
152 views

Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?

My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
2
votes
3answers
89 views

How can I force Google to re-crawl a link?

Google Webmaster Tools is showing thousands of links from a North Korean travel agency, but when I go to the link, they don't exist! How can I force Google to re-index that link so it drops off my ...
1
vote
2answers
45 views

Do search engine robots crawl dynamic URLs that differ by only the value of a parameter in the query string?

I know there are a lot of similar questions and I searched around but I never completely found an answer, so here is my question: Do search engine robots crawl my dynamically generated URLs? With ...
1
vote
1answer
47 views

Google's cache (text version) does not show hyperlinks

I have list of product links on one of our website's page. For some reason when I view the google's cached version of this page, google strips out all the product links. Links in the header and footer ...
0
votes
2answers
22 views

Allow crawl bots to only see articles and not index page?

I'm currently trying to improve my search engine results. I got a index page that lists all articles with a little teaser and then the article pages themselves. Now when I e.g. google "websiteXXX ...
1
vote
0answers
67 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
9
votes
1answer
2k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...
3
votes
1answer
104 views

Does Google penalize sites for showing a different menu for mobile compared to desktop?

I'm really concerned about a design I have been handed by a designer. It changes the menu for mobile and desktop. On mobile you can only access the main categories, that will lead to all the content ...
1
vote
1answer
34 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
2
votes
1answer
58 views

If a 302 redirect is implemented, would the old site still be indexed?

If a 302 redirect is implemented, would the old site still be indexed? I’m assuming as the redirect is in place, a user would not see the contact on the old page hence search/crawl bots would not be ...
1
vote
2answers
509 views

Google Webmaster Tools is reporting crawl errors for URLs with “joomla” on a WordPress site

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
1answer
23 views

Are “embed” HTML tags significant for crawlers and preview generators?

I'm wondering if <embed> tags get used to crawl and/or affect SEO rankings. Say I'm making a "widget" of HTML (maybe eventually a Polymer web widget) but let's say it's just a plain old HTML5 ...
2
votes
1answer
37 views

Fixing website title on Bing

I have recently created my own portfolio website, and the title meta tags specify the title of my front page as: "Thomas Russell's Portfolio and Blog", but when I search for "Thomas Russell's ...
5
votes
1answer
71 views

Is it expected behaviour that Google also crawls #! urls?

I'm a developer for an AJAX webapp and, following Google's specification for crawlable webapps, we use #! to indicate that it's a AJAX application such that we can serve a static page to Google ...
1
vote
1answer
86 views

How can Iframes be optimized for search engines?

If my site has an Iframe. What is the best way to optimize this for search engines? We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
1
vote
2answers
85 views

Is it okay to block all user agents that start with “PHP/5.”?

I'm getting a lot of spambots from China (and elsewhere, likely botnets) accessing my blog which have a user agent string starting with PHP/5.. These include such nonsense values as PHP/5.3.70 and ...
2
votes
1answer
99 views

Is there a way to opt out of Alexa rankings or prevent them from gathering data about my site?

My competitor is using Alexa to see what I am doing. I need to get stop Alexa from including my sites in their rankings. How can I do so? Is it possible to deny access to any visitor with the Alexa ...
4
votes
1answer
89 views

Does robot traffic visiting your website have any advantages?

What are the advantages of having bot traffic on your website? Will it raise your SEO ranking on Google? Real visitors would raise awareness of your product and site, but the visitors I'm seeing ...
0
votes
2answers
195 views

Pages crawled with Cloudflare

I activated Cloudflare on my blog a few days ago and in their Analytics it shows me that there are 255 pages crawled by Bing, 16 by Yandex and only 1 by Google. Why only one page for Google? Should I ...
2
votes
1answer
88 views

Different blocked URLs stats in Google Webmaster Tools

Just want to know I got different blocked URLs stats at Google Webmaster Tools. I blocked entire site for revamp and latest builds implementation. Here are different stats Under Health tab > Blocked ...
1
vote
1answer
45 views

Notify search engine crawlers about user generated content so that our rankings are not biased by keywords used by users

Is there a meta tag or a way to notify crawlers that a page is generated by the user? To clarify the issue, We have a website already promoted by SEO and we want to add dynamic user generated content ...
4
votes
1answer
182 views

How to make page content crawlable when it is usually requested via AJAX on page load

I hope that the contents which is pulled on page load could be crawled by google robot, how to do that ? I went to Google Webmasters tool to ensure that whether Google crawler could see the content ...