0
votes
1answer
12 views

Baidu spider does not obey DNS TTL

I moved my website from one VPS to another on 16th April. The A record of the domain name has a TTL of 86400. I made the IP address change at 19:00 GMT 16th April. I do not want the site to face ...
1
vote
0answers
9 views

How to make page content crawlable , which is requested in script from ajax on page load

I hope that the contents which is pulled on page load could be crawled by google robot, how to do that ? The Demo page : http://radiansmile.github.io/web/Test/ajax_at_Initial.html I go to google ...
0
votes
1answer
34 views

Why does bot traffic have a high (e.g. 100%) bounce rate in GA?

Since bots may request many pages at a time, why do they so often have a 100% bounce rate? It seem unlikely that they would only request one page, then leave for over 30 minutes, then request another ...
1
vote
1answer
43 views

Replacing url parameters tool usage with canonical / robots.txt to avoid crawl time

For a long time I've been getting Googlebot found an extremely high number of URLs warnings, which I know don't necessarily indicate there is a problem, but I am trying to reduce the amount of links ...
5
votes
5answers
971 views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
1
vote
2answers
56 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
5
votes
3answers
57 views

Does meta name=robots content=nofollow prevent indexing iframes?

Let's say I have a publicly accessible page that contains an iframe back to a different page on my own domain. By default, google (and other crawlers) will index that page and the iframe's content. It ...
1
vote
1answer
45 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
1
vote
1answer
21 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
6
votes
1answer
93 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
7
votes
1answer
69 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
1
vote
1answer
33 views

Why would I use a robots.txt file? [duplicate]

From what I understand after reading Google's Controlling Crawling and Indexing: The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
0
votes
2answers
32 views

How do you determine when to block an IP address or range of addresses?

When analyzing traffic to some of my sites, I almost always notice a large amount of traffic from obscure sources like Japan, Russia, and other countries that really have no business visiting my ...
1
vote
2answers
49 views

Why URL and linked pages submissions for Google?

Why Google Webmaster Tools have limitation on URL and linked pages submissions? Is there any solution to submit more than 10 URL?
0
votes
2answers
37 views

Pages crawled with Cloudflare

I activated Cloudflare on my blog a few days ago and in their Analytics it shows me that there are 255 pages crawled by Bing, 16 by Yandex and only 1 by Google. Why only one page for Google? Should I ...
1
vote
3answers
100 views

Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
0
votes
0answers
25 views

Stopping JavaScript savvy bots from getting into GA in the first place

I'm aware that bots can be filtered out at the profile level in GA, but I hadn't considered the effect of letting all that bogus data into the web property, and therefore ultimately counting against ...
1
vote
1answer
29 views

Custom Sitemap or sitemap generator tool [closed]

I have recently launched a website which is using a free sitemap generator. It's a large travel agency site with predominantly dynamically generated content - users can add their products as and when ...
-1
votes
1answer
32 views

Downloading pdf files [closed]

I have very basic programming skills or other computer skills. I wanted to download whole web content including pdf text files from a website, by going through each pdf and downloading them ...
1
vote
1answer
45 views

Will there be a conflict between my Sitemap.xml and robots.txt?

I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: ...
0
votes
2answers
54 views

Preventing links with query parameters from being indexed by Google

I manage a mobile version of a desktop site. The mobile site has an mdot subdomain and is served from a different server. As part of recommended best practices by Google, I've included a link on the ...
1
vote
1answer
134 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
2
votes
1answer
59 views

Upgrading website from XHTML 1.0 Transitional to HTML 5: how can it affect SERP positions?

We have an enough old but still supported website which is XHTML 1.0 Transitional now. We are going to change it to HTML 5 to add some new features in the future (microdata, ...
1
vote
2answers
151 views

What is bingsandbox.com? [closed]

I toyed around a with "Bing Webmaster Tools": https://www.bing.com/toolbox/webmaster And then checked access.log. One referrer was http://www.bingsandbox.com/ Link: http://www.bingsandbox.com/ ...
3
votes
2answers
67 views

Why is AhrefsBot requesting a page that's been removed from my website?

I was reviewing the logs of my website (WordPress), and I saw a line like this : myWebsite:80 5.10.83.28 - - [17/Jan/2014:09:05:53 +0000] "GET myUrl == HTTP/1.1" 404 5716 "-" "Mozilla/5.0 ...
3
votes
2answers
100 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
1answer
92 views

Restrict all robots except Googlebot and Yandexbot

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages. All other bots will not be accessed my pages. Is this robots.txt ...
1
vote
1answer
108 views

Preventing Twitterbot to access website

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And ...
4
votes
2answers
75 views

Do I need a robots.txt file if most of my site's pages require authorization?

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization. Does that mean I need to have a robots.txt file to Disallow these pages with ...
2
votes
2answers
102 views

Can a website require search engine robots to support cookies?

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other ...
4
votes
3answers
97 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
5
votes
1answer
75 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
2
votes
3answers
93 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
1
vote
1answer
149 views

301 redirect causing Bingbot to crash, redirection limit reached

Inside my .htaccess file, I have the following script: RewriteEngine on RewriteCond %{HTTPS} off RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] <IfModule mod_rewrite.c> ...
4
votes
2answers
101 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
1
vote
3answers
135 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
2answers
72 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
1
vote
0answers
123 views

Bot crawling my site, blocked it, now it's back and can't stop it [closed]

I had a bot of some sort crawling my site like crazy for the last day or so. It says in Google Analytics that it's from San Jose in the US. The IP addresses that appeared in the access logs were ...
0
votes
1answer
100 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
1
vote
1answer
50 views

Has Microsoft introduced a bug in Bingbot such that it crawls non-existent images?

I've started receiving requests for paths such as /scale/jackets/00000086/80x80_P2_G1715325W.JPG several times a day from the Bingbot. This started a couple of days ago. Anybody know why this is?
0
votes
1answer
130 views

Serve different content for web crawlers to fix SEO for SPA?

I have my site built as an SPA using Knockout. The main-page populates a list of items dynamically as the user scrolls and each item has a details-page which is also dynamically loaded. I have ...
2
votes
2answers
69 views

Does changing phone numbers in content just for bots qualify as cloaking?

I'm a small business owner (I run a little computer repair shop), and I'm attempting to track calls to my store from different advertising channels. I use three separate tracking phone numbers: one ...
4
votes
2answers
367 views

Is my whole site is blocked by this Disallow: /

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site. User-agent: Baiduspider Disallow: / User-agent: * Disallow: ...
1
vote
3answers
111 views

In WordPress, should I “Allow search engines to index this site” during development?

I am going to create a web page, and I will do it online. Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
4
votes
2answers
209 views

Why doesn't this web crawler obey requests to stop crawling my pages?

I have a hidden page that verifies internal payments from PayPal. The page has the following in its HTTP headers: X-Robots-Tag: noindex, nofollow But once in a while I see this in the page logs: ...
1
vote
1answer
58 views

Can AdSense crawler view pages that require cookies?

Details I require users to agree to terms and conditions before they can view several pages on my site. Once they have agreed a cookie is set and they can proceed to the webpage. If a user somehow ...
1
vote
1answer
64 views

Is using !# for AJAX URLs the standard when it comes to SEO?

Will URLs like this site.com/!#users only be crawled by Google's search engine bots, or is this a standard adopted by all search engines (e.g., Yahoo, Bing, DuckDuckGo, etc...)?
5
votes
2answers
170 views

Tell Google the list of URLs to crawl

Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find ...
3
votes
1answer
121 views

Should we allow HEAD requests?

Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from ...
0
votes
1answer
78 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...