A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
0answers
23 views

My Website is getting a lot of traffic from an unresolved IP address [on hold]

My website is getting lot of traffic from an unresolved IP. I am using AWStats and it is showing that I am getting huge amount of hits within few seconds. My hosting company suspended my website ...
0
votes
0answers
10 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
2
votes
2answers
34 views

Does Google index images declared using 'srcset'?

As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
0
votes
0answers
12 views

robots.txt that allows only Google and Bing [duplicate]

Is it possible to create a robots.txt file that disallows everything except for Google and Bing? Something like: User-agent: * Disallow: / User-agent: Googlebot Allow: / User-agent: Googlebot-News ...
1
vote
1answer
42 views

How to keep friendly URLs consistent?

I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed. The original site ran on a combination of Drupal and a lesser known, ...
1
vote
1answer
19 views

What is this bot in server logs?

Noticed this bot in the top ten bots in the server logs: dwbot Coming from the United States. Does any one have any idea what it is?
0
votes
1answer
20 views

Updated and published website - changes not taking effect in search results?

Question Background: I've recently updated my website and republished it using Azure. This is fine and the changes have taken effect when I open and view the site from a Google search. This is the ...
0
votes
1answer
34 views

Trick to index infinite scroll content on Google?

I have a Question & Answer website, sort of like stackoverflow. When you open it, it displays 20 latest questions. When scroll to bottom of page, and it fetches 20 more, and so on, infinitely. ...
2
votes
1answer
70 views

Does Bing execute Javascript when crawling and indexing web pages (like Google)?

Some content on some web pages is only displayed with Javascript when a button is clicked (for example). We know that Google executes Javascript and is (very likely) to find this content, but what ...
1
vote
1answer
116 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
1
vote
1answer
71 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
0
votes
2answers
40 views

What is the best way to detect if the traffic is legit or (bot) not?

We are planning to start program something similar to youtube where users get paid as per views. But problem we are facing is that people will game the system and send traffic using botnets to gain ...
3
votes
1answer
46 views

What determines the frequency the Wayback Machine crawls one's website?

I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
0
votes
0answers
18 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
2
votes
1answer
48 views

Does e.preventDefault on link prevent the page from being crawled?

I have a couple of links in navigation on homepage that lead to new pages. When clicked on link, i want to prevent going on that page (e.preventDefault()), and instead load that page in popup with ...
0
votes
1answer
31 views

Format of _escaped_fragment_ for complex states

I understand that for a url such as www.example.com/about, Google's crawler will look for www.example.com?escaped_fragment_=about. However, what will google use as the value for the ...
0
votes
0answers
19 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...
1
vote
1answer
41 views

How to prevent search engine bots from crawling specific pages?

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it. Is it ...
0
votes
2answers
44 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
0
votes
1answer
23 views

Crawl errors for pdf download links in my website

We are building a local business classified website which has about 6000 unique listingsunder huge number of categories and sub-categories. For each listing there is a details page in which we have ...
1
vote
1answer
55 views

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?

I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Questions: If I "hide" these links via ...
0
votes
1answer
76 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
0
votes
0answers
14 views

spam referers bypass htaccess rules [duplicate]

Here is a sample code I'm using in .htaccess to block spam referers: RewriteEngine On RewriteCond %{HTTP_REFERER} ^http://.*buttons-for-your-website\.com/ [NC,OR] RewriteCond %{HTTP_REFERER} ...
2
votes
3answers
70 views

How to block a user-agent that has a spacing in its name?

I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess and nginx configuration as I'm running nginx as a reverse proxy in ...
1
vote
1answer
32 views

Is there anything unusual or nefarious about private IP addresses in server logs?

A site I help take care of has just started getting crawled by somebody from 172.31.13.241 and 172.31.42.227 . According to WHOIS info, these are part of a block reserved by IANA for special purposes, ...
0
votes
1answer
36 views

How to find out which links of website posted on Twitter

I recenlty noticed many requests from Twitterbot user agent in my website's logs which I believe it doesn't crawl by itself unless someone is posting links of my website on Twitter. These posts might ...
1
vote
0answers
9 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
0
votes
2answers
51 views

Generating a list of URLs including metadata [closed]

Basically I want to do an editorial sweep on my website, I need a program to crawl my website and generate a list of URLS and their metadata. A program that produces something like a CSV would be ...
1
vote
2answers
49 views

Why does Google Analytics report that I have traffic at a calculator.php URL for my blog?

I frequently see this URL having traffic via Google Analytics although I don't have any page and with that strucuter either. URL - ...
0
votes
3answers
61 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
1
vote
2answers
48 views

Pages with dynamically loaded content and SEO

This question is not bound to a particular web technology. If I load a page that is basically a blank shell and on page load I start getting all the content from the server with ajax requests, won't ...
0
votes
0answers
6 views

How does google discover new hosts? [duplicate]

Assume I register a new host, foobar.com, and assume that Google doesn't know about foobar.com at all. Never in the history of internet was foobar.com mentioned in any web page. The two ways I know ...
-1
votes
3answers
18 views

Is there any special tag for web master to crawler page

My page is based on HTML and I need to subscribe for google webmaster and bing so I know how to give short description about my page So that it easy to crawl my page my page is full of html code and ...
1
vote
2answers
41 views

Can I limit content scrapers by counting the number of hits from an IP?

I would like to ban aggressive scrapers that access x number of pages in an hour, say 1000. I plan to implement such a check via fail2ban and count the hits from the same ip via access logs. Should I ...
1
vote
2answers
102 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
8
votes
4answers
625 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
1
vote
3answers
276 views

500 error returns website content

I'm using CloudFlare on my shared host. Recently all search engines and is-it-up sites have been getting a 500 error. I'm not sure who's to blame, my host or CloudFlare. I don't remember changing ...
0
votes
0answers
14 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
4
votes
1answer
118 views

If I dynamically add a robots meta tag (via JS) will Google and other search engines recognize it?

I want to have a chunk of JS in the middle of my page insert a meta-robots element into the <head> based on some arbitrary condition. Will Google and other search engines recognize this tag the ...
2
votes
1answer
307 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
1
vote
2answers
99 views

Do search engine robots crawl dynamic URLs that differ by only the value of a parameter in the query string?

I know there are a lot of similar questions and I searched around but I never completely found an answer, so here is my question: Do search engine robots crawl my dynamically generated URLs? With ...
2
votes
1answer
81 views

Adsense crawler login for dynamic page urls

I have a website that largely features dynamic content. Every piece of dynamic content has it's own url, such as "www.example.com/page/*/*" The problem I have is that many of these pages require a ...
2
votes
3answers
279 views

How can I force Google to re-crawl a link?

Google Webmaster Tools is showing thousands of links from a North Korean travel agency, but when I go to the link, they don't exist! How can I force Google to re-index that link so it drops off my ...
1
vote
1answer
72 views

Google's cache (text version) does not show hyperlinks

I have list of product links on one of our website's page. For some reason when I view the google's cached version of this page, google strips out all the product links. Links in the header and footer ...
0
votes
2answers
28 views

Allow crawl bots to only see articles and not index page?

I'm currently trying to improve my search engine results. I got a index page that lists all articles with a little teaser and then the article pages themselves. Now when I e.g. google "websiteXXX ...
1
vote
0answers
111 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
4
votes
1answer
162 views

Does Google penalize sites for showing a different menu for mobile compared to desktop?

I'm really concerned about a design I have been handed by a designer. It changes the menu for mobile and desktop. On mobile you can only access the main categories, that will lead to all the content ...
9
votes
1answer
3k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...