A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
1answer
78 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
2
votes
1answer
77 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...
6
votes
1answer
401 views

What happens if I try to set a cookie on a bot?

I'm building my site to include some user-identifying cookies. When a visitor chimes in, I will set a cookie with a unique visitor id (a guid) and the date-time of the visit, and save a visit record ...
7
votes
1answer
113 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
1
vote
1answer
50 views

Why would I use a robots.txt file? [duplicate]

From what I understand after reading Google's Controlling Crawling and Indexing: The purpose of robots.txt file is to disallow crawling of some URLs, but those can still be indexed (and appear in ...
0
votes
2answers
184 views

How do you determine when to block an IP address or range of addresses?

When analyzing traffic to some of my sites, I almost always notice a large amount of traffic from obscure sources like Japan, Russia, and other countries that really have no business visiting my ...
0
votes
2answers
64 views

Why URL and linked pages submissions for Google?

Why Google Webmaster Tools have limitation on URL and linked pages submissions? Is there any solution to submit more than 10 URL?
0
votes
2answers
134 views

Pages crawled with Cloudflare

I activated Cloudflare on my blog a few days ago and in their Analytics it shows me that there are 255 pages crawled by Bing, 16 by Yandex and only 1 by Google. Why only one page for Google? Should I ...
1
vote
3answers
276 views

Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc. However, the websites still need to be accessible for our customers, and ...
0
votes
0answers
28 views

Stopping JavaScript savvy bots from getting into GA in the first place

I'm aware that bots can be filtered out at the profile level in GA, but I hadn't considered the effect of letting all that bogus data into the web property, and therefore ultimately counting against ...
1
vote
1answer
37 views

Custom Sitemap or sitemap generator tool [closed]

I have recently launched a website which is using a free sitemap generator. It's a large travel agency site with predominantly dynamically generated content - users can add their products as and when ...
-1
votes
1answer
83 views

Downloading pdf files [closed]

I have very basic programming skills or other computer skills. I wanted to download whole web content including pdf text files from a website, by going through each pdf and downloading them ...
1
vote
1answer
54 views

Will there be a conflict between my Sitemap.xml and robots.txt?

I have a robots.txt with the following in it: User-agent: * Disallow: http://tests.compkerworld.com/report_question_error.php?question_id=*&url=* Sitemap: ...
0
votes
2answers
161 views

Preventing links with query parameters from being indexed by Google

I manage a mobile version of a desktop site. The mobile site has an mdot subdomain and is served from a different server. As part of recommended best practices by Google, I've included a link on the ...
1
vote
1answer
384 views

Sudden decrease in pages crawled per day

During the last few days, Googlebot has crawled much less page in my site than it normally does. I'm not seeing any new errors, and haven't identified a reason for this. Is such a hugh and sudden ...
2
votes
1answer
137 views

Upgrading website from XHTML 1.0 Transitional to HTML 5: how can it affect SERP positions?

We have an enough old but still supported website which is XHTML 1.0 Transitional now. We are going to change it to HTML 5 to add some new features in the future (microdata, ...
1
vote
2answers
781 views

What is bingsandbox.com? [closed]

I toyed around a with "Bing Webmaster Tools": https://www.bing.com/toolbox/webmaster And then checked access.log. One referrer was http://www.bingsandbox.com/ Link: http://www.bingsandbox.com/ ...
3
votes
2answers
130 views

Why is AhrefsBot requesting a page that's been removed from my website?

I was reviewing the logs of my website (WordPress), and I saw a line like this : myWebsite:80 5.10.83.28 - - [17/Jan/2014:09:05:53 +0000] "GET myUrl == HTTP/1.1" 404 5716 "-" "Mozilla/5.0 ...
3
votes
2answers
175 views

Controlling robot crawling concurrency using robots.txt and the undocumented setting “host-load”

I'm not much of a network guy, and I am troubleshooting an issue where one of our sites intermittently becomes unresponsive to a reverse proxy: it simply starts turning down connections, and then ...
2
votes
1answer
418 views

Restrict all robots except Googlebot and Yandexbot

I want to allow Googlebot access all my pages with crawl delay. Yandexbot access only index.html. Bingbot access /tools pages. All other bots will not be accessed my pages. Is this robots.txt ...
1
vote
1answer
325 views

Preventing Twitterbot to access website

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And ...
4
votes
2answers
95 views

Do I need a robots.txt file if most of my site's pages require authorization?

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization. Does that mean I need to have a robots.txt file to Disallow these pages with ...
2
votes
2answers
142 views

Can a website require search engine robots to support cookies?

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other ...
4
votes
3answers
214 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
5
votes
1answer
90 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
2
votes
3answers
174 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
1
vote
1answer
339 views

301 redirect causing Bingbot to crash, redirection limit reached

Inside my .htaccess file, I have the following script: RewriteEngine on RewriteCond %{HTTPS} off RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] <IfModule mod_rewrite.c> ...
4
votes
2answers
110 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
1
vote
3answers
250 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
2answers
111 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
1
vote
0answers
225 views

Bot crawling my site, blocked it, now it's back and can't stop it [closed]

I had a bot of some sort crawling my site like crazy for the last day or so. It says in Google Analytics that it's from San Jose in the US. The IP addresses that appeared in the access logs were ...
0
votes
1answer
240 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
1
vote
1answer
76 views

Has Microsoft introduced a bug in Bingbot such that it crawls non-existent images?

I've started receiving requests for paths such as /scale/jackets/00000086/80x80_P2_G1715325W.JPG several times a day from the Bingbot. This started a couple of days ago. Anybody know why this is?
0
votes
1answer
342 views

Serve different content for web crawlers to fix SEO for SPA?

I have my site built as an SPA using Knockout. The main-page populates a list of items dynamically as the user scrolls and each item has a details-page which is also dynamically loaded. I have ...
2
votes
2answers
84 views

Does changing phone numbers in content just for bots qualify as cloaking?

I'm a small business owner (I run a little computer repair shop), and I'm attempting to track calls to my store from different advertising channels. I use three separate tracking phone numbers: one ...
4
votes
2answers
693 views

Is my whole site is blocked by this Disallow: /

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site. User-agent: Baiduspider Disallow: / User-agent: * Disallow: ...
1
vote
3answers
231 views

In WordPress, should I “Allow search engines to index this site” during development?

I am going to create a web page, and I will do it online. Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
4
votes
2answers
289 views

Why doesn't this web crawler obey requests to stop crawling my pages?

I have a hidden page that verifies internal payments from PayPal. The page has the following in its HTTP headers: X-Robots-Tag: noindex, nofollow But once in a while I see this in the page logs: ...
1
vote
1answer
78 views

Can AdSense crawler view pages that require cookies?

Details I require users to agree to terms and conditions before they can view several pages on my site. Once they have agreed a cookie is set and they can proceed to the webpage. If a user somehow ...
1
vote
2answers
104 views

Is using !# for AJAX URLs the standard when it comes to SEO?

Will URLs like this site.com/!#users only be crawled by Google's search engine bots, or is this a standard adopted by all search engines (e.g., Yahoo, Bing, DuckDuckGo, etc...)?
5
votes
2answers
186 views

Tell Google the list of URLs to crawl

Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find ...
3
votes
1answer
201 views

Should we allow HEAD requests?

Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from ...
0
votes
1answer
136 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...
3
votes
3answers
114 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
2
votes
2answers
397 views

Robots meta tag with “noimageindex”

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
1
vote
4answers
863 views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
2
votes
1answer
283 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
1
vote
2answers
92 views

Does a “nofollow” attribute on a link prevent URL discovery by search engines?

I know that nofollow prevents link juice from being passed across a link. But if search engine robots discover a link with a nofollow on it, will they add that link to their crawl queue? In other ...
0
votes
1answer
72 views

Search engine bots accessing strange URLs

We have ELMAH enabled on our site and get errors whenever a Page Not Found error is triggered on the website. We have recently redesigned a new website and so we understand that search engine ...
3
votes
2answers
156 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...