A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
0answers
7 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
0
votes
3answers
11 views

Is there any special tag for web master to crawler page

My page is based on HTML and I need to subscribe for google webmaster and bing so I know how to give short description about my page So that it easy to crawl my page my page is full of html code and ...
0
votes
0answers
4 views

Using GA's Import Data function w/ external data [closed]

I'm working on a series of posts about how webmasters can benefit form external data. Does anyone have a cool use case where they've used/or could use scraped data or data from an API with GA? ...
1
vote
2answers
37 views

Why does Google Analytics report that I have traffic at a calculator.php URL for my blog?

I frequently see this URL having traffic via Google Analytics although I don't have any page and with that strucuter either. URL - ...
0
votes
2answers
27 views

Generating a list of URLs including metadata [closed]

Basically I want to do an editorial sweep on my website, I need a program to crawl my website and generate a list of URLS and their metadata. A program that produces something like a CSV would be ...
2
votes
1answer
69 views

Adsense crawler login for dynamic page urls

I have a website that largely features dynamic content. Every piece of dynamic content has it's own url, such as "www.example.com/page/*/*" The problem I have is that many of these pages require a ...
1
vote
1answer
123 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
0
votes
3answers
41 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
13
votes
5answers
5k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
1
vote
2answers
34 views

Pages with dynamically loaded content and SEO

This question is not bound to a particular web technology. If I load a page that is basically a blank shell and on page load I start getting all the content from the server with ajax requests, won't ...
1
vote
2answers
73 views

Private folders which are not indexable

I have a requirement where I need to distribute about 400 copies of a pdf document (about 6 MB) and .pub files, that are personalized for each user. Every user must have access to only his/her set of ...
1
vote
2answers
37 views

Can I limit content scrapers by counting the number of hits from an IP?

I would like to ban aggressive scrapers that access x number of pages in an hour, say 1000. I plan to implement such a check via fail2ban and count the hits from the same ip via access logs. Should I ...
4
votes
4answers
778 views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
0
votes
0answers
5 views

How does google discover new hosts? [duplicate]

Assume I register a new host, foobar.com, and assume that Google doesn't know about foobar.com at all. Never in the history of internet was foobar.com mentioned in any web page. The two ways I know ...
1
vote
2answers
76 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
1
vote
2answers
294 views

What are spiders trying to achieve on my site by fetching lots of non-existant urls such as various versions of phpMyAdmin?

I've recently set up a build server that compiles Java source on Github webhooks. I created a public page for downloading latest .jars with a top level domain and I'm logging requests to it. What are ...
4
votes
2answers
661 views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
1
vote
3answers
254 views

500 error returns website content

I'm using CloudFlare on my shared host. Recently all search engines and is-it-up sites have been getting a 500 error. I'm not sure who's to blame, my host or CloudFlare. I don't remember changing ...
8
votes
4answers
339 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
2
votes
5answers
10k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
0
votes
1answer
92 views

Is it OK to use translations obtained via crawling a translation service with advertising?

Google and Bing not only have their API-based translation, but also their websites which allow for one-off translation. While these are intended for use by humans, this also allows for a webcrawler ...
8
votes
2answers
4k views

Google Webmaster Tools tells me that robots is blocking access to the sitemap

This is my robots.txt: User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Sitemap: http://www.mydomain.org/sitemap.xml.gz But Google Webmaster Tools tells me that robots is blocking access ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
0
votes
0answers
13 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
6
votes
1answer
9k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
4
votes
1answer
77 views

If I dynamically add a robots meta tag (via JS) will Google and other search engines recognize it?

I want to have a chunk of JS in the middle of my page insert a meta-robots element into the <head> based on some arbitrary condition. Will Google and other search engines recognize this tag the ...
5
votes
1answer
113 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
7
votes
2answers
419 views

Which meta “robots” tag gets preference?

My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
2
votes
2answers
746 views

Robots meta tag with “noimageindex”

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
2
votes
1answer
172 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
2
votes
2answers
692 views

Fake links cause crawl error in Google Webmaster Tools

Google reported Crawl Errors last week on my largest site though Webmaster Tools. Here is the message: Google detected a significant increase in the number of URLs that return a 404 (Page ...
5
votes
2answers
332 views

Google Webmaster Tools reports fake 404 errors

I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...
2
votes
1answer
600 views

How to determine if Google's Crawl Rate for your site is eating up too much Bandwidth?

I was reading about changing Google's crawl rate and how there is a balance in which if the crawl rate is too fast, it will slow your site down; but if the crawl rate is too slow, then page ...
2
votes
1answer
470 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
3
votes
2answers
63 views

How to set crawl rate in Google Webmaster Tools for seven days or a month?

My web host is blocking IPs which make many requests. This includes Googlebot. I found this by checking my webmaster tools account. There it said "network unreachable" for all the URLs which are ...
2
votes
2answers
152 views

Will setting a custom crawl rate in Google Webmaster Tools affects index or have other negative consequences?

My site was running well for the past three years. Recently Googlebot's request my server got increased. and at one time I got network unreachable error. When I contacted my hosting company they asked ...
2
votes
3answers
111 views

How can I force Google to re-crawl a link?

Google Webmaster Tools is showing thousands of links from a North Korean travel agency, but when I go to the link, they don't exist! How can I force Google to re-index that link so it drops off my ...
1
vote
2answers
65 views

Do search engine robots crawl dynamic URLs that differ by only the value of a parameter in the query string?

I know there are a lot of similar questions and I searched around but I never completely found an answer, so here is my question: Do search engine robots crawl my dynamically generated URLs? With ...
1
vote
1answer
53 views

Google's cache (text version) does not show hyperlinks

I have list of product links on one of our website's page. For some reason when I view the google's cached version of this page, google strips out all the product links. Links in the header and footer ...
0
votes
2answers
25 views

Allow crawl bots to only see articles and not index page?

I'm currently trying to improve my search engine results. I got a index page that lists all articles with a little teaser and then the article pages themselves. Now when I e.g. google "websiteXXX ...
1
vote
0answers
86 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
9
votes
1answer
2k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...
3
votes
1answer
117 views

Does Google penalize sites for showing a different menu for mobile compared to desktop?

I'm really concerned about a design I have been handed by a designer. It changes the menu for mobile and desktop. On mobile you can only access the main categories, that will lead to all the content ...
1
vote
1answer
35 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
2
votes
1answer
60 views

If a 302 redirect is implemented, would the old site still be indexed?

If a 302 redirect is implemented, would the old site still be indexed? I’m assuming as the redirect is in place, a user would not see the contact on the old page hence search/crawl bots would not be ...
1
vote
2answers
521 views

Google Webmaster Tools is reporting crawl errors for URLs with “joomla” on a WordPress site

Google Webmaster Tools is reporting crawl errors for links that never existed, and if i check the "Linked from" tab for a given error link, it shows another that never existed. They all mention ...
1
vote
1answer
23 views

Are “embed” HTML tags significant for crawlers and preview generators?

I'm wondering if <embed> tags get used to crawl and/or affect SEO rankings. Say I'm making a "widget" of HTML (maybe eventually a Polymer web widget) but let's say it's just a plain old HTML5 ...
2
votes
1answer
37 views

Fixing website title on Bing

I have recently created my own portfolio website, and the title meta tags specify the title of my front page as: "Thomas Russell's Portfolio and Blog", but when I search for "Thomas Russell's ...
5
votes
1answer
71 views

Is it expected behaviour that Google also crawls #! urls?

I'm a developer for an AJAX webapp and, following Google's specification for crawlable webapps, we use #! to indicate that it's a AJAX application such that we can serve a static page to Google ...