A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

2
votes
0answers
24 views

Does Bing execute Javascript when crawling and indexing web pages (like Google)?

Some content on some web pages is only displayed with Javascript when a button is clicked (for example). We know that Google executes Javascript and is (very likely) to find this content, but what ...
1
vote
1answer
106 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
1
vote
1answer
54 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
0
votes
2answers
29 views

What is the best way to detect if the traffic is legit or (bot) not?

We are planning to start program something similar to youtube where users get paid as per views. But problem we are facing is that people will game the system and send traffic using botnets to gain ...
3
votes
1answer
38 views

What determines the frequency the Wayback Machine crawls one's website?

I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
0
votes
0answers
16 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
2
votes
1answer
41 views

Does e.preventDefault on link prevent the page from being crawled?

I have a couple of links in navigation on homepage that lead to new pages. When clicked on link, i want to prevent going on that page (e.preventDefault()), and instead load that page in popup with ...
0
votes
1answer
24 views

Format of _escaped_fragment_ for complex states

I understand that for a url such as www.example.com/about, Google's crawler will look for www.example.com?escaped_fragment_=about. However, what will google use as the value for the ...
0
votes
0answers
18 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...
1
vote
1answer
38 views

How to prevent search engine bots from crawling specific pages?

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it. Is it ...
0
votes
2answers
36 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
0
votes
1answer
19 views

Crawl errors for pdf download links in my website

We are building a local business classified website which has about 6000 unique listingsunder huge number of categories and sub-categories. For each listing there is a details page in which we have ...
1
vote
1answer
51 views

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?

I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Questions: If I "hide" these links via ...
0
votes
1answer
66 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
0
votes
0answers
13 views

spam referers bypass htaccess rules [duplicate]

Here is a sample code I'm using in .htaccess to block spam referers: RewriteEngine On RewriteCond %{HTTP_REFERER} ^http://.*buttons-for-your-website\.com/ [NC,OR] RewriteCond %{HTTP_REFERER} ...
2
votes
3answers
64 views

How to block a user-agent that has a spacing in its name?

I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess and nginx configuration as I'm running nginx as a reverse proxy in ...
1
vote
1answer
32 views

Is there anything unusual or nefarious about private IP addresses in server logs?

A site I help take care of has just started getting crawled by somebody from 172.31.13.241 and 172.31.42.227 . According to WHOIS info, these are part of a block reserved by IANA for special purposes, ...
0
votes
1answer
30 views

How to find out which links of website posted on Twitter

I recenlty noticed many requests from Twitterbot user agent in my website's logs which I believe it doesn't crawl by itself unless someone is posting links of my website on Twitter. These posts might ...
1
vote
0answers
9 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
0
votes
2answers
38 views

Generating a list of URLs including metadata [closed]

Basically I want to do an editorial sweep on my website, I need a program to crawl my website and generate a list of URLS and their metadata. A program that produces something like a CSV would be ...
1
vote
2answers
47 views

Why does Google Analytics report that I have traffic at a calculator.php URL for my blog?

I frequently see this URL having traffic via Google Analytics although I don't have any page and with that strucuter either. URL - ...
0
votes
3answers
52 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
1
vote
2answers
43 views

Pages with dynamically loaded content and SEO

This question is not bound to a particular web technology. If I load a page that is basically a blank shell and on page load I start getting all the content from the server with ajax requests, won't ...
0
votes
0answers
5 views

How does google discover new hosts? [duplicate]

Assume I register a new host, foobar.com, and assume that Google doesn't know about foobar.com at all. Never in the history of internet was foobar.com mentioned in any web page. The two ways I know ...
0
votes
3answers
16 views

Is there any special tag for web master to crawler page

My page is based on HTML and I need to subscribe for google webmaster and bing so I know how to give short description about my page So that it easy to crawl my page my page is full of html code and ...
1
vote
2answers
41 views

Can I limit content scrapers by counting the number of hits from an IP?

I would like to ban aggressive scrapers that access x number of pages in an hour, say 1000. I plan to implement such a check via fail2ban and count the hits from the same ip via access logs. Should I ...
1
vote
2answers
91 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
8
votes
4answers
465 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
1
vote
3answers
263 views

500 error returns website content

I'm using CloudFlare on my shared host. Recently all search engines and is-it-up sites have been getting a 500 error. I'm not sure who's to blame, my host or CloudFlare. I don't remember changing ...
0
votes
0answers
13 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
4
votes
1answer
99 views

If I dynamically add a robots meta tag (via JS) will Google and other search engines recognize it?

I want to have a chunk of JS in the middle of my page insert a meta-robots element into the <head> based on some arbitrary condition. Will Google and other search engines recognize this tag the ...
2
votes
1answer
262 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
1
vote
2answers
82 views

Do search engine robots crawl dynamic URLs that differ by only the value of a parameter in the query string?

I know there are a lot of similar questions and I searched around but I never completely found an answer, so here is my question: Do search engine robots crawl my dynamically generated URLs? With ...
2
votes
1answer
78 views

Adsense crawler login for dynamic page urls

I have a website that largely features dynamic content. Every piece of dynamic content has it's own url, such as "www.example.com/page/*/*" The problem I have is that many of these pages require a ...
2
votes
3answers
187 views

How can I force Google to re-crawl a link?

Google Webmaster Tools is showing thousands of links from a North Korean travel agency, but when I go to the link, they don't exist! How can I force Google to re-index that link so it drops off my ...
1
vote
1answer
64 views

Google's cache (text version) does not show hyperlinks

I have list of product links on one of our website's page. For some reason when I view the google's cached version of this page, google strips out all the product links. Links in the header and footer ...
0
votes
2answers
26 views

Allow crawl bots to only see articles and not index page?

I'm currently trying to improve my search engine results. I got a index page that lists all articles with a little teaser and then the article pages themselves. Now when I e.g. google "websiteXXX ...
1
vote
0answers
102 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
4
votes
1answer
148 views

Does Google penalize sites for showing a different menu for mobile compared to desktop?

I'm really concerned about a design I have been handed by a designer. It changes the menu for mobile and desktop. On mobile you can only access the main categories, that will lead to all the content ...
9
votes
1answer
2k views

Only allow Google and Bing bots to crawl a site

I am using following robots.txt file for a site: Target is to allow googlebot and bingbot to access the site except the page /bedven/bedrijf/* and block all other bots from crawling the site. ...
1
vote
1answer
36 views

Robots: block /lang/page from the index but keep /page

My URL structure looks a bit like this: /group1/ /group2/ /group3/ Group3 (and it's subpages) are also available in another language at the following URLs. A lang prefix was added to this URL: ...
2
votes
1answer
62 views

If a 302 redirect is implemented, would the old site still be indexed?

If a 302 redirect is implemented, would the old site still be indexed? I’m assuming as the redirect is in place, a user would not see the contact on the old page hence search/crawl bots would not be ...
4
votes
2answers
861 views

SEO - Responsive Website and Duplicated Menus

Whenever I create a Responsive Website I usually create 2 menus: 1 hidden and used for mobile and the other displayed as the main menu, then hidden to show the mobile menu. Whenever it comes to SEO ...
2
votes
1answer
40 views

Fixing website title on Bing

I have recently created my own portfolio website, and the title meta tags specify the title of my front page as: "Thomas Russell's Portfolio and Blog", but when I search for "Thomas Russell's ...
1
vote
1answer
23 views

Are “embed” HTML tags significant for crawlers and preview generators?

I'm wondering if <embed> tags get used to crawl and/or affect SEO rankings. Say I'm making a "widget" of HTML (maybe eventually a Polymer web widget) but let's say it's just a plain old HTML5 ...
1
vote
1answer
136 views

How can Iframes be optimized for search engines?

If my site has an Iframe. What is the best way to optimize this for search engines? We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
1
vote
2answers
102 views

Is it okay to block all user agents that start with “PHP/5.”?

I'm getting a lot of spambots from China (and elsewhere, likely botnets) accessing my blog which have a user agent string starting with PHP/5.. These include such nonsense values as PHP/5.3.70 and ...
5
votes
1answer
73 views

Is it expected behaviour that Google also crawls #! urls?

I'm a developer for an AJAX webapp and, following Google's specification for crawlable webapps, we use #! to indicate that it's a AJAX application such that we can serve a static page to Google ...