A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
1answer
178 views

How to block a spam bot .htaccess proper formatting?

I have downloaded my access log file from my cPanel and found I have a bot I would like to stop. In the access log, there User-Agent string reads the following; 88.80.205.219 - - ...
0
votes
1answer
52 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
2
votes
1answer
87 views

Is there a way to opt out of Alexa rankings or prevent them from gathering data about my site?

My competitor is using Alexa to see what I am doing. I need to get stop Alexa from including my sites in their rankings. How can I do so? Is it possible to deny access to any visitor with the Alexa ...
2
votes
2answers
41 views

How do I get Google to index two URLs that differ only in the value of a URL parameter?

I have a site which has variable ID's which bring up different content For example: http://example.com/university.php?id=3 http://example.com/university.php?id=4 Technically these are two ...
0
votes
1answer
33 views

How to force search engines not to crawl my site, is there any special mark on them? [duplicate]

I want to block search engines from crawling my websites. Atleast for Msn, Yahoo, Google, and Yandex. i dont really trust robot.txt. Because they can simply ignore it and continue crawling. I would ...
6
votes
1answer
377 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
0
votes
1answer
201 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
4
votes
2answers
592 views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
0
votes
1answer
109 views

Baidu spider does not obey DNS TTL

I moved my website from one VPS to another on 16th April. The A record of the domain name has a TTL of 86400. I made the IP address change at 19:00 GMT 16th April. I do not want the site to face ...
4
votes
1answer
150 views

How to make page content crawlable when it is usually requested via AJAX on page load

I hope that the contents which is pulled on page load could be crawled by google robot, how to do that ? I went to Google Webmasters tool to ensure that whether Google crawler could see the content ...
0
votes
1answer
80 views

Why does bot traffic have a high (e.g. 100%) bounce rate in GA?

Since bots may request many pages at a time, why do they so often have a 100% bounce rate? It seem unlikely that they would only request one page, then leave for over 30 minutes, then request another ...
1
vote
1answer
92 views

Replacing url parameters tool usage with canonical / robots.txt to avoid crawl time

For a long time I've been getting Googlebot found an extremely high number of URLs warnings, which I know don't necessarily indicate there is a problem, but I am trying to reduce the amount of links ...
5
votes
5answers
1k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
1
vote
2answers
133 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
5
votes
3answers
271 views

Does meta name=robots content=nofollow prevent indexing iframes?

Let's say I have a publicly accessible page that contains an iframe back to a different page on my own domain. By default, google (and other crawlers) will index that page and the iframe's content. It ...
1
vote
1answer
89 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
2
votes
1answer
97 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
6
votes
1answer
570 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
8
votes
1answer
130 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
1
vote
1answer
50 views

Why would I use a robots.txt file? [duplicate]

From what I understand after reading Google's Controlling Crawling and Indexing: The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
0
votes
2answers
198 views

How do you determine when to block an IP address or range of addresses?

When analyzing traffic to some of my sites, I almost always notice a large amount of traffic from obscure sources like Japan, Russia, and other countries that really have no business visiting my ...
0
votes
2answers
68 views

Why URL and linked pages submissions for Google?

Why Google Webmaster Tools have limitation on URL and linked pages submissions? Is there any solution to submit more than 10 URL?
0
votes
2answers
168 views

Pages crawled with Cloudflare

I activated Cloudflare on my blog a few days ago and in their Analytics it shows me that there are 255 pages crawled by Bing, 16 by Yandex and only 1 by Google. Why only one page for Google? Should I ...
1
vote
3answers
368 views

Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
1
vote
1answer
41 views

Custom Sitemap or sitemap generator tool [closed]

I have recently launched a website which is using a free sitemap generator. It's a large travel agency site with predominantly dynamically generated content - users can add their products as and when ...
-1
votes
1answer
111 views

Downloading pdf files [closed]

I have very basic programming skills or other computer skills. I wanted to download whole web content including pdf text files from a website, by going through each pdf and downloading them ...
1
vote
1answer
55 views

Will there be a conflict between my Sitemap.xml and robots.txt?

I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: ...
1
vote
2answers
210 views

Preventing links with query parameters from being indexed by Google

I manage a mobile version of a desktop site. The mobile site has an mdot subdomain and is served from a different server. As part of recommended best practices by Google, I've included a link on the ...
1
vote
1answer
459 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
2
votes
1answer
158 views

Upgrading website from XHTML 1.0 Transitional to HTML 5: how can it affect SERP positions?

We have an enough old but still supported website which is XHTML 1.0 Transitional now. We are going to change it to HTML 5 to add some new features in the future (microdata, ...
1
vote
2answers
944 views

What is bingsandbox.com? [closed]

I toyed around a with "Bing Webmaster Tools": https://www.bing.com/toolbox/webmaster And then checked access.log. One referrer was http://www.bingsandbox.com/ Link: http://www.bingsandbox.com/ ...
3
votes
2answers
157 views

Why is AhrefsBot requesting a page that's been removed from my website?

I was reviewing the logs of my website (WordPress), and I saw a line like this : myWebsite:80 5.10.83.28 - - [17/Jan/2014:09:05:53 +0000] "GET myUrl == HTTP/1.1" 404 5716 "-" "Mozilla/5.0 ...
3
votes
2answers
200 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
1answer
593 views

Restrict all robots except Googlebot and Yandexbot

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages. All other bots will not be accessed my pages. Is this robots.txt ...
1
vote
1answer
340 views

Preventing Twitterbot to access website

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And ...
4
votes
2answers
97 views

Do I need a robots.txt file if most of my site's pages require authorization?

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization. Does that mean I need to have a robots.txt file to Disallow these pages with ...
2
votes
2answers
160 views

Can a website require search engine robots to support cookies?

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other ...
4
votes
3answers
250 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
5
votes
1answer
99 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
2
votes
3answers
200 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
1
vote
1answer
385 views

301 redirect causing Bingbot to crash, redirection limit reached

Inside my .htaccess file, I have the following script: RewriteEngine on RewriteCond %{HTTPS} off RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] <IfModule mod_rewrite.c> ...
4
votes
2answers
114 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
1
vote
3answers
302 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
2answers
119 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
1
vote
0answers
236 views

Bot crawling my site, blocked it, now it's back and can't stop it [closed]

I had a bot of some sort crawling my site like crazy for the last day or so. It says in Google Analytics that it's from San Jose in the US. The IP addresses that appeared in the access logs were ...
0
votes
1answer
257 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
1
vote
1answer
80 views

Has Microsoft introduced a bug in Bingbot such that it crawls non-existent images?

I've started receiving requests for paths such as /scale/jackets/00000086/80x80_P2_G1715325W.JPG several times a day from the Bingbot. This started a couple of days ago. Anybody know why this is?
0
votes
1answer
381 views

Serve different content for web crawlers to fix SEO for SPA?

I have my site built as an SPA using Knockout. The main-page populates a list of items dynamically as the user scrolls and each item has a details-page which is also dynamically loaded. I have ...
2
votes
2answers
89 views

Does changing phone numbers in content just for bots qualify as cloaking?

I'm a small business owner (I run a little computer repair shop), and I'm attempting to track calls to my store from different advertising channels. I use three separate tracking phone numbers: one ...
4
votes
2answers
1k views

Is my whole site is blocked by this Disallow: /

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site. User-agent: Baiduspider Disallow: / User-agent: * Disallow: ...