A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
2answers
25 views

Preventing a link on a page from being indexed and followed

I have read the post about how nofollow value impacts on crawlers/indexing. However the information from the post answers end on 2012. In my case, a have a file page that is indexed by Google. On ...
0
votes
0answers
19 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
1
vote
1answer
23 views

Bots and image caching

Once again, I'm trying to optimize my site because one test showed the time to first byte is slightly over 200ms, but I need it under 200ms to make google happy. I was looking at code to my site and ...
8
votes
2answers
767 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
0
votes
0answers
5 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's ...
0
votes
0answers
9 views

Spiders cannot crawl all my website pages [duplicate]

I'm struggling with googlebot webpages indexing; in particular I have a page with many links to other subpages that aren't indexed. The strange thing is that, if you submit in any online crawler the ...
1
vote
2answers
80 views

Stop Google from crawling proxied pages but still allow the proxy itself to be found via search engines

I have a proxy at http://rahul2001.com/proxy/ I site can be 'proxied' by like this: http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.example.com/ The problem is that Google seems to be ...
5
votes
2answers
255 views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
2
votes
2answers
36 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
3
votes
1answer
31 views

WordPress: Many IPs issuing wp-login redirect + wp-login register + xmlrpc. Problem or not?

My WordPress instance is hosted at my own domain and is installed in subdirectory xxxx (i.e. not in the root directory). 5 days ago I added a WP blog, and also started commenting in social media about ...
3
votes
1answer
40 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
5
votes
3answers
49 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
6
votes
3answers
108 views

How to handle SEO for brand that has a special character in the name

I am building a website (small online retail store) for a company whose brand name has a special character in it. The name of the brand is actually a legitimate Polish word that has an Ł, which is an ...
1
vote
2answers
38 views

Is there SEO value or danger when creating product comparison functionality?

I am building a Product comparison web/application into my website. It will compare products by specs, price etc. I am wondering if there any dangers of not making a user log in before using the web ...
4
votes
2answers
29 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
1
vote
1answer
58 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
1
vote
1answer
37 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
2
votes
2answers
123 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
3
votes
1answer
23 views

Prevent bing from crawling thousands of essentially identical pages?

I have a web page with a dozen tables of data on it, each with half a dozen columns. Every table can be sorted by a column by clicking on the relevant header, and these get appended to the ...
0
votes
0answers
26 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or ...
0
votes
0answers
12 views

How to gracefully and temporarily stop search engines from crawling a website [duplicate]

I run into these moments every now and then where I want to make a major update to my website but then I feel I just cant do it because search engines crawl it at that time. Is there a way I can ...
2
votes
2answers
106 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
1
vote
0answers
53 views

Mysterious visits to privileged URLs from anonymous user/bot

I maintain a Rails app. Every page except login and "forgot password" requires user authentication. Yesterday I noticed entries like this one in the access logs: 54.209.60.63 - - ...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
69 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
1
vote
0answers
34 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is ...
8
votes
2answers
251 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9
votes
3answers
816 views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
0
votes
1answer
30 views

Can Searchbots Find Web Pages That Aren't Linked To? [duplicate]

I have read this which claims that the listed search engines are not able to show the unlinked doc&XML in its results. My questions are Have any one else experimented like this? This information ...
0
votes
0answers
23 views

Google INDEED crawls ajax? [duplicate]

I don't understand how this didn't make a bigger BOOM online. It is well known in the industry that google does not index AJAX (just google-search "google crawls ajax") and see dozens of articles ...
0
votes
1answer
24 views

Does disavow solve the problem of removing unnatural links in google search?

I have created the site and it got hacked by spammer. now there many unnatural links in it if I disavow it the whole domain will it solve my problem what. Also I have two version of domain ...
0
votes
1answer
17 views

Issue in webmaster tool

I have created a site with domain name https://www.example.com but the site had many problems so I created a new fresh site and removed the previous one. This time I chose the domain name as ...
1
vote
0answers
13 views

Google displaying wrong site title and snippet [duplicate]

I am having an issue where a website that displays the incorrect site title and snippet in Google Search. From the content it is clear that some spammer has infected the site, but I cannot figure out ...
1
vote
1answer
26 views

Webmaster Showing Desktop Pages in Mobile Usability errors

I have same URL for both desktop and mobile content, but changes their format according to User-agent. Google search console is showing error as "Touch elements too close" for some pages which are ...
1
vote
0answers
22 views

Page accessible but googlebots return 404 error [closed]

I'm working on a project which has Codeigniter integrated with WordPress. My website works fine, all links are accessible. When google crawls my website pages then only for wordpress pages it gives ...
4
votes
3answers
60 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
3
votes
2answers
376 views

How to identify if IP address is really google's IP

From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with ...
0
votes
0answers
19 views

Seo-wise way to remove a portion of website [duplicate]

I have a website with 5000+ products driven from a MySQL database, so far so good. My links are friendly structured; http://www.example.com/brand/product/ About a year ago, I decide to add reviews ...
0
votes
2answers
115 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
1
vote
1answer
25 views

Crawl-error: Google searches for foo.tld/www.foo.tld

Google Webmaster Tools is reporting strange crawl errors. It reports that www.example.com can't be found, yet the domain is valid and is an exact match of our clients domain. A click on the error ...
5
votes
1answer
97 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
0
votes
1answer
40 views

Screaming Frog displaying deleted subdomain

In Screaming Frog, when I do a basic crawl of one of my website's in the 'Internal' tab it shows a sub-domain at the very top. This sub-domain was one I had previously created, but I removed it quite ...
0
votes
2answers
63 views

How do search engines behave when encountering noindex?

I have a question about the behaviour of search engines when they encounter meta name="robots" content="noindex" If I use the meta tag format of robots, I know that search engines will not index the ...
0
votes
2answers
133 views

Should sitemap contain URL for all categories of Product?

I am having open cart eCommerce website. Here I need to add one product under multiple Categories. By default what opencart produces SITEMAP.xml, it includes URL for all the categroies for that ...
0
votes
0answers
22 views

Fetch as Google AngularJS Duplicate Entries

I have pages built in AngularJS that when I fetch as Google in WMT or when I switch my user-agent to Googlebot using a browser extension, duplicate entries are displayed. For example, if there is an ...
0
votes
0answers
149 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
1
vote
2answers
42 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
1
vote
1answer
37 views

How do canonical tags work when you have multiple versions of a website?

I apologize for how vague the question sounds; I was not able to write the complete question in the title because I ran out of characters. Before the duplicate monsters close the question, please read ...
0
votes
2answers
44 views

SEO: How to stop specific sections of a page from being crawled? [duplicate]

I would like to display customer reviews from another website (with their agreement) own my own site for certain product pages. Using these 3rd party reviews would be helpful for users, though ...
0
votes
0answers
13 views

Would use of IFrame in my ecommerce site affect SEO ranking? [duplicate]

I Use an Iframe to show product summary, which is saved HTML Data from ckEditor. (As the summary have it's own html and styles I can't use it in product page directly). <iframe ...