A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

4
votes
2answers
88 views

Do I need a robots.txt file if most of my site's pages require authorization?

I have a website with many pages, but except for Home page and a View Profile page, all other pages require authorization. Does that mean I need to have a robots.txt file to Disallow these pages with ...
2
votes
2answers
115 views

Can a website require search engine robots to support cookies?

Sometimes I pretend to be a search engine robot to get access to "subscriber only" web pages, or just to get a plain and simple view of a page that isn't cluttered with moving animations and other ...
4
votes
3answers
154 views

Will the content of a PDF on our website affect SEO?

I know that Google crawls websites including PDFs, but does the content contained within PDFs affect SEO rankings? We want to put some PDFs on our website but don't want to see our results take a ...
5
votes
1answer
82 views

What is the default behavior of robots if there is no robots meta tag?

Assuming I have a page without a robots.txt and without the robots meta tag. What is the default behavior of robots? Would they index the page and follow the links as in: <meta name="robots" ...
2
votes
3answers
126 views

How to decrease the frequency of Google/Search-engine bots crawling my site?

My server handles my visitors ok, but recently I found there are many search engine bots crawling my site and my server gets quite busy serving these bots. My site generates over 1000 posts each day, ...
1
vote
1answer
253 views

301 redirect causing Bingbot to crash, redirection limit reached

Inside my .htaccess file, I have the following script: RewriteEngine on RewriteCond %{HTTPS} off RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] <IfModule mod_rewrite.c> ...
4
votes
2answers
107 views

Prevent Google from listing hotlinked images for other sites

I am hosting several domains on my server and hotlink images accross various domains. However, I have found out that in Google searches, the images are listed as originating on the site where the ...
1
vote
3answers
200 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
1
vote
2answers
90 views

Search engines that index your site despite a disallow in robots.txt

Recently, WordPress changed their options from "Block search engines" to "Discourage search engines" So that leaves me wondering, that rewording must be that some search engines disregard your ...
1
vote
0answers
168 views

Bot crawling my site, blocked it, now it's back and can't stop it [closed]

I had a bot of some sort crawling my site like crazy for the last day or so. It says in Google Analytics that it's from San Jose in the US. The IP addresses that appeared in the access logs were ...
0
votes
1answer
174 views

Need help with mybb forum robots.txt file

I have installed the mybb forum in subdomain of my site. The link to my site is here. but when I search Google even the pages that are disallowed are also index. What's wrong with the robots.txt ...
1
vote
1answer
68 views

Has Microsoft introduced a bug in Bingbot such that it crawls non-existent images?

I've started receiving requests for paths such as /scale/jackets/00000086/80x80_P2_G1715325W.JPG several times a day from the Bingbot. This started a couple of days ago. Anybody know why this is?
0
votes
1answer
278 views

Serve different content for web crawlers to fix SEO for SPA?

I have my site built as an SPA using Knockout. The main-page populates a list of items dynamically as the user scrolls and each item has a details-page which is also dynamically loaded. I have ...
2
votes
2answers
75 views

Does changing phone numbers in content just for bots qualify as cloaking?

I'm a small business owner (I run a little computer repair shop), and I'm attempting to track calls to my store from different advertising channels. I use three separate tracking phone numbers: one ...
4
votes
2answers
390 views

Is my whole site is blocked by this Disallow: /

This is a view of my robots.txt file did this query blocked only Baiduspider Search Engine or all of the search engines to check my site. User-agent: Baiduspider Disallow: / User-agent: * Disallow: ...
1
vote
3answers
190 views

In WordPress, should I “Allow search engines to index this site” during development?

I am going to create a web page, and I will do it online. Should I choose the option: "Allow search engines to index this site." during development, or should this be opened after development ends?
4
votes
2answers
259 views

Why doesn't this web crawler obey requests to stop crawling my pages?

I have a hidden page that verifies internal payments from PayPal. The page has the following in its HTTP headers: X-Robots-Tag: noindex, nofollow But once in a while I see this in the page logs: ...
1
vote
1answer
72 views

Can AdSense crawler view pages that require cookies?

Details I require users to agree to terms and conditions before they can view several pages on my site. Once they have agreed a cookie is set and they can proceed to the webpage. If a user somehow ...
1
vote
2answers
97 views

Is using !# for AJAX URLs the standard when it comes to SEO?

Will URLs like this site.com/!#users only be crawled by Google's search engine bots, or is this a standard adopted by all search engines (e.g., Yahoo, Bing, DuckDuckGo, etc...)?
5
votes
2answers
182 views

Tell Google the list of URLs to crawl

Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find ...
3
votes
1answer
165 views

Should we allow HEAD requests?

Our web application is predominantly read-only. As such we only allow GET requests. However, we regularly get errors in our logs because of user agents making HEAD requests. I've seen these come from ...
0
votes
1answer
110 views

Wlll links into pages that are restricted by Disallow rules in robots.txt help SEO?

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable. If I get incoming ...
3
votes
3answers
110 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
2
votes
2answers
231 views

Robots meta tag with “noimageindex”

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
1
vote
4answers
298 views

Does Google ignore robots.txt

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
2
votes
1answer
198 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
0
votes
2answers
67 views

Does a “nofollow” attribute on a link prevent URL discovery by search engines?

I know that nofollow prevents link juice from being passed across a link. But if search engine robots discover a link with a nofollow on it, will they add that link to their crawl queue? In other ...
0
votes
1answer
65 views

Search engine bots accessing strange URLs

We have ELMAH enabled on our site and get errors whenever a Page Not Found error is triggered on the website. We have recently redesigned a new website and so we understand that search engine ...
3
votes
2answers
141 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
3
votes
2answers
244 views

Grapeshot crawler ignoring robots.txt

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
1
vote
1answer
71 views

Can I make query strings produce separate pages?

I have a profile page with a URL like so: localhost/profile.php/?username=Bob I was wondering, if I had a separate <title> which changed according to the username, would they produce separate ...
0
votes
0answers
12 views

If Googlebot is crawling 25 pages per day, why isn't my website indexed by Google? [duplicate]

I have created a word press site which is 2 months old and has 15 posts. The average crawl rate is 25 pages per day. Is that sufficient for my site to get some organic traffic from Google search? If ...
2
votes
1answer
297 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
1
vote
1answer
372 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
2
votes
1answer
105 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
4
votes
3answers
137 views

How can I make Google crawl and index multiple pages?

There are several pages on the web that are not under my direct control but contain content about a product I'm involved with. I'd like make Google robots crawl and index those pages, is there a ...
0
votes
0answers
7 views

Crawling problem with AJAX search result pages [duplicate]

I have a jobs site and the search results are AJAX pages. Googlebot is not able to detect the links to the job offers listed in the search result pages. So I don't have any of the job offers in ...
1
vote
1answer
240 views

Checking Request's user agent versus handling _escaped_fragment_ urls when using pushState only (no hashbangs)

I'm working on an AJAX-based application and thinking of how to make it crawlable by search engines. I've read the Google's AJAX crawling specification but I couldn't get my head around one thing: if ...
1
vote
2answers
126 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
2
votes
1answer
223 views

How to use NOINDEX,FOLLOW with paginated results except index page

I am running a classified website. At the moment I am using lazy loading with jquery to load content while scrolling. I got to know that using jquery loading is not good for search-engine bots ...
1
vote
2answers
155 views

Fake Links Showing up in Google Webmaster Tools

Google Webmaster tools is reporting a large increase in 'not found errors' on my site, and I see approximately 1700 'Not Found Errors over the past 2 months. All of them are for a url that is ...
2
votes
4answers
416 views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
0
votes
2answers
169 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
0
votes
2answers
116 views

Opening forum for search engine bots and SEO

We develop and sell components for other software developers through our website. The main products are ActiveX and WinFroms.NET grid controls. Some years ago we established a forum dedicated to our ...
0
votes
0answers
68 views

Is it OK to use translations obtained via crawling a translation service with advertising?

Google and Bing not only have their API-based translation, but also their websites which allow for one-off translation. While these are intended for use by humans, this also allows for a webcrawler ...
1
vote
1answer
42 views

How to write robots.txt for one hosting which has several websites in different directories?

I have several websites on one hosting account. The main website is in the root. Other websites are in sub-folders off the root directory. In the robots.txt file for the main website, do I need to ...
2
votes
2answers
1k views

Google keeps crawling pages that don't exist

I made a simple new website which went live about a month and a half ago (replacing the last one). The site contains new pages, like for example the navigation markup: <div class="header"> ...
2
votes
1answer
218 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
3
votes
2answers
147 views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 ...
0
votes
1answer
21 views

Indexing a page in which the content is discovered in a map

We want to improve our site's SEO. The page is a GWT (javascript based) web page and its content is discovered through a map or a GPS. Now we have just a normal landing html page, and the map and ...