A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
2answers
42 views

Is finding and using expired domains for backlinks considered Black-Hat?

http://www.matthewwoodward.co.uk/tutorials/build-expired-web-2-0-networks-increase-rankings/ This guide shows how to identify and generate backlinks for a website. If one was to follow this guide: ...
3
votes
1answer
36 views

How to exclude Bots from Google Analytics (sudden rise of them in the last weeks)

Since a few weeks, one of my Google Analytics Accounts shows a continued rise of Visitors and Pageimpressions what seams to be bots. They have a Average Session Duration of «<00:00:01» and by now ...
1
vote
2answers
50 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?
0
votes
2answers
103 views

SEO effects of H1 carousel slider

I maintain a website for a law firm. Originally their main desired keyword was "zoning attorneys." So I created a small banner that has an h1 "ZONING ATTORNEYS" on every page with the attorneys' names ...
3
votes
1answer
38 views

cookies and useragent is not enough ? to emulate browser?

I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt Let's take an example, I need to get this page: http://www.aliexpress.com/...
2
votes
1answer
42 views

Does site structure/URL structure impact SEO? [duplicate]

We would like to change the following URLs as follows: (old versions to the left, being changed to new versions on the right) www.ex.com/xxx/full-seo-term -> www.ex.com/half-seo-term1/half-seo-term2 ...
3
votes
2answers
42 views

4 URLs lead to the same page content, are 301 redirects better or setting rel canonical better?

All the following pages have the same exact content: http://www.example.com/seo-url (rel canonical URL) http://www.example.com/ID1234 http://www.example.com/content/ID1234 http://www.example.com/...
2
votes
1answer
148 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
1answer
61 views

Is it OK for SEO and user experience to 301 redirect self-generated param URLs?

I am using a CMS which offers a wide range of options which lead to many additional special pages mainly in a way duplicating the same content as the original article. These generated pages are ...
3
votes
1answer
95 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
4
votes
3answers
48 views

Preventing a link on a page from being indexed and followed

I have read the post about how nofollow value impacts on crawlers/indexing. However the information from the post answers end on 2012. In my case, a have a file page that is indexed by Google. On ...
1
vote
0answers
160 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
1
vote
1answer
29 views

Bots and image caching

Once again, I'm trying to optimize my site because one test showed the time to first byte is slightly over 200ms, but I need it under 200ms to make google happy. I was looking at code to my site and ...
8
votes
2answers
824 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
1
vote
1answer
18 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's dashboard....
0
votes
0answers
10 views

Spiders cannot crawl all my website pages [duplicate]

I'm struggling with googlebot webpages indexing; in particular I have a page with many links to other subpages that aren't indexed. The strange thing is that, if you submit in any online crawler the ...
0
votes
2answers
98 views

Stop Google from crawling proxied pages but still allow the proxy itself to be found via search engines

I have a proxy at http://rahul2001.com/proxy/ I site can be 'proxied' by like this: http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.example.com/ The problem is that Google seems to be ...
5
votes
2answers
431 views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
2
votes
2answers
81 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
3
votes
1answer
38 views

WordPress: Many IPs issuing wp-login redirect + wp-login register + xmlrpc. Problem or not?

My WordPress instance is hosted at my own domain and is installed in subdirectory xxxx (i.e. not in the root directory). 5 days ago I added a WP blog, and also started commenting in social media about ...
4
votes
1answer
61 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
5
votes
3answers
70 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
6
votes
3answers
168 views

How to handle SEO for brand that has a special character in the name

I am building a website (small online retail store) for a company whose brand name has a special character in it. The name of the brand is actually a legitimate Polish word that has an Ł, which is an ...
1
vote
2answers
46 views

Is there SEO value or danger when creating product comparison functionality?

I am building a Product comparison web/application into my website. It will compare products by specs, price etc. I am wondering if there any dangers of not making a user log in before using the web ...
4
votes
2answers
40 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
1
vote
1answer
122 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
1
vote
1answer
59 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
2
votes
2answers
130 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
3
votes
1answer
27 views

Prevent bing from crawling thousands of essentially identical pages?

I have a web page with a dozen tables of data on it, each with half a dozen columns. Every table can be sorted by a column by clicking on the relevant header, and these get appended to the ...
0
votes
0answers
48 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or directory)...
0
votes
0answers
12 views

How to gracefully and temporarily stop search engines from crawling a website [duplicate]

I run into these moments every now and then where I want to make a major update to my website but then I feel I just cant do it because search engines crawl it at that time. Is there a way I can ...
3
votes
2answers
275 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
1
vote
0answers
61 views

Mysterious visits to privileged URLs from anonymous user/bot

I maintain a Rails app. Every page except login and "forgot password" requires user authentication. Yesterday I noticed entries like this one in the access logs: 54.209.60.63 - - [03/Nov/2015:19:09:...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
111 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
1
vote
0answers
36 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is ...
9
votes
2answers
325 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9
votes
3answers
1k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
0
votes
1answer
61 views

Can Searchbots Find Web Pages That Aren't Linked To? [duplicate]

I have read this which claims that the listed search engines are not able to show the unlinked doc&XML in its results. My questions are Have any one else experimented like this? This information ...
0
votes
0answers
24 views

Google INDEED crawls ajax? [duplicate]

I don't understand how this didn't make a bigger BOOM online. It is well known in the industry that google does not index AJAX (just google-search "google crawls ajax") and see dozens of articles ...
0
votes
1answer
25 views

Does disavow solve the problem of removing unnatural links in google search?

I have created the site and it got hacked by spammer. now there many unnatural links in it if I disavow it the whole domain will it solve my problem what. Also I have two version of domain https://...
0
votes
1answer
18 views

Issue in webmaster tool

I have created a site with domain name https://www.example.com but the site had many problems so I created a new fresh site and removed the previous one. This time I chose the domain name as https:...
1
vote
0answers
13 views

Google displaying wrong site title and snippet [duplicate]

I am having an issue where a website that displays the incorrect site title and snippet in Google Search. From the content it is clear that some spammer has infected the site, but I cannot figure out ...
1
vote
1answer
29 views

Webmaster Showing Desktop Pages in Mobile Usability errors

I have same URL for both desktop and mobile content, but changes their format according to User-agent. Google search console is showing error as "Touch elements too close" for some pages which are ...
4
votes
3answers
65 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
4
votes
2answers
1k views

How to identify if IP address is really google's IP

From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with 74....
0
votes
0answers
20 views

Seo-wise way to remove a portion of website [duplicate]

I have a website with 5000+ products driven from a MySQL database, so far so good. My links are friendly structured; http://www.example.com/brand/product/ About a year ago, I decide to add reviews ...
0
votes
2answers
163 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
1
vote
1answer
25 views

Crawl-error: Google searches for foo.tld/www.foo.tld

Google Webmaster Tools is reporting strange crawl errors. It reports that www.example.com can't be found, yet the domain is valid and is an exact match of our clients domain. A click on the error ...
6
votes
1answer
131 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...