A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
1answer
8 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's ...
3
votes
1answer
30 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. ...
3
votes
2answers
29 views

Preventing a link on a page from being indexed and followed

I have read the post about how nofollow value impacts on crawlers/indexing. However the information from the post answers end on 2012. In my case, a have a file page that is indexed by Google. On ...
1
vote
1answer
26 views

Webmaster Showing Desktop Pages in Mobile Usability errors

I have same URL for both desktop and mobile content, but changes their format according to User-agent. Google search console is showing error as "Touch elements too close" for some pages which are ...
1
vote
2answers
43 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
0
votes
0answers
20 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
2
votes
1answer
271 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
1
vote
1answer
24 views

Bots and image caching

Once again, I'm trying to optimize my site because one test showed the time to first byte is slightly over 200ms, but I need it under 200ms to make google happy. I was looking at code to my site and ...
8
votes
2answers
776 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
3
votes
1answer
31 views

WordPress: Many IPs issuing wp-login redirect + wp-login register + xmlrpc. Problem or not?

My WordPress instance is hosted at my own domain and is installed in subdirectory xxxx (i.e. not in the root directory). 5 days ago I added a WP blog, and also started commenting in social media about ...
4
votes
5answers
7k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
2
votes
3answers
964 views

How can I force Google to re-crawl a link?

Google Webmaster Tools is showing thousands of links from a North Korean travel agency, but when I go to the link, they don't exist! How can I force Google to re-index that link so it drops off my ...
0
votes
0answers
9 views

Spiders cannot crawl all my website pages [duplicate]

I'm struggling with googlebot webpages indexing; in particular I have a page with many links to other subpages that aren't indexed. The strange thing is that, if you submit in any online crawler the ...
0
votes
4answers
80 views

What is the best way to detect if the traffic is legit or (bot) not?

We are planning to start program something similar to youtube where users get paid as per views. But problem we are facing is that people will game the system and send traffic using botnets to gain ...
16
votes
6answers
7k views

Preventing robots from crawling specific part of a page

As a webmaster in charge of a tiny site that has a forum, I regularly receive complains from users that both the internal search engine and that external searches (like when using Google) are totally ...
1
vote
2answers
81 views

Stop Google from crawling proxied pages but still allow the proxy itself to be found via search engines

I have a proxy at http://rahul2001.com/proxy/ I site can be 'proxied' by like this: http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.example.com/ The problem is that Google seems to be ...
16
votes
4answers
4k views

Does the Google spider render JavaScript?

Does the Google spider knows how to render JavaScript, or only HTML?
5
votes
2answers
259 views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
2
votes
2answers
38 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
3
votes
1answer
40 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
5
votes
3answers
49 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
5
votes
6answers
13k views

How to block baidu spiders

Most of my visits are from baidu spiders. I don't think it helps search engines at all so I'm thinking of how to block them. Could this be done via iptables? I'm using nginx as my webserver.
1
vote
2answers
1k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
2
votes
2answers
109 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
6
votes
3answers
110 views

How to handle SEO for brand that has a special character in the name

I am building a website (small online retail store) for a company whose brand name has a special character in it. The name of the brand is actually a legitimate Polish word that has an Ł, which is an ...
2
votes
1answer
715 views

Why are nonexistant files listed in the TRANSFER log?

This snippit from my apache log reflects some piece of malware casually browsing through my site for some unknown purpose. The most puzzling thing to me is that the GET address doesn't correspond ...
1
vote
3answers
184 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
1
vote
2answers
38 views

Is there SEO value or danger when creating product comparison functionality?

I am building a Product comparison web/application into my website. It will compare products by specs, price etc. I am wondering if there any dangers of not making a user log in before using the web ...
4
votes
2answers
29 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
9
votes
2answers
263 views

Disqus thread migration. Gotchas?

I've been migrating a site to a new domain. The site itself is pretty straightforward (it uses Jekyll), and everything has gone fine -- except migration of Disqus threads. I've had partial success ...
3
votes
1answer
23 views

Prevent bing from crawling thousands of essentially identical pages?

I have a web page with a dozen tables of data on it, each with half a dozen columns. Every table can be sorted by a column by clicking on the relevant header, and these get appended to the ...
1
vote
1answer
60 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
1
vote
1answer
37 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
2
votes
2answers
124 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
0
votes
0answers
27 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or ...
0
votes
0answers
12 views

How to gracefully and temporarily stop search engines from crawling a website [duplicate]

I run into these moments every now and then where I want to make a major update to my website but then I feel I just cant do it because search engines crawl it at that time. Is there a way I can ...
9
votes
3answers
825 views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
1
vote
0answers
53 views

Mysterious visits to privileged URLs from anonymous user/bot

I maintain a Rails app. Every page except login and "forgot password" requires user authentication. Yesterday I noticed entries like this one in the access logs: 54.209.60.63 - - ...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
163 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
5
votes
2answers
1k views

How often are sitemap.xml checked for updates by crawlers?

How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
0
votes
1answer
71 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
1
vote
0answers
34 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is ...
8
votes
2answers
255 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
0
votes
1answer
32 views

Can Searchbots Find Web Pages That Aren't Linked To? [duplicate]

I have read this which claims that the listed search engines are not able to show the unlinked doc&XML in its results. My questions are Have any one else experimented like this? This information ...
0
votes
0answers
23 views

Google INDEED crawls ajax? [duplicate]

I don't understand how this didn't make a bigger BOOM online. It is well known in the industry that google does not index AJAX (just google-search "google crawls ajax") and see dozens of articles ...
0
votes
1answer
24 views

Does disavow solve the problem of removing unnatural links in google search?

I have created the site and it got hacked by spammer. now there many unnatural links in it if I disavow it the whole domain will it solve my problem what. Also I have two version of domain ...
0
votes
1answer
17 views

Issue in webmaster tool

I have created a site with domain name https://www.example.com but the site had many problems so I created a new fresh site and removed the previous one. This time I chose the domain name as ...
4
votes
3answers
60 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
3
votes
1answer
292 views

Do web crawlers use the “revised” meta tag?

When adding a WordPress plugin to add a "Last Modified" date to my posts I noticed the plugin also has the option to add a "revised" date meta tag. The revised tag looks like: <meta ...