A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

3
votes
3answers
116 views

robots.txt dissalow url containing string with a '/' at the end

i have a website with thousands of dynamic pages. I want to use the robots.txt file in order to dissalow certain url patterns corresponding to pages with duplicate content. For example i have a page ...
2
votes
2answers
436 views

Robots meta tag with “noimageindex”

I have some doubt regarding noimageindex value in meta robots tag. If I add this tag on my page say http://www.example.com/somepage/someaction.php and on that page the images are served from another ...
1
vote
4answers
917 views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
2
votes
1answer
305 views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
1
vote
2answers
97 views

Does a “nofollow” attribute on a link prevent URL discovery by search engines?

I know that nofollow prevents link juice from being passed across a link. But if search engine robots discover a link with a nofollow on it, will they add that link to their crawl queue? In other ...
0
votes
1answer
74 views

Search engine bots accessing strange URLs

We have ELMAH enabled on our site and get errors whenever a Page Not Found error is triggered on the website. We have recently redesigned a new website and so we understand that search engine ...
3
votes
2answers
157 views

'Buy the app' landing page implementations: redirect or JavaScript popup?

My site (using Django) has an app that I'm trying to push - I currently have a piece of middleware that redirects the user to a page advertising the app if they're accessing the page on the iPhone, ...
3
votes
2answers
314 views

Grapeshot crawler ignoring robots.txt

Has anyone come across a crawler called Grapeshot? They are hammering the same page repeatedly on our website. I believe they are looking for ad related keywords, based on previous content ad ...
1
vote
1answer
74 views

Can I make query strings produce separate pages?

I have a profile page with a URL like so: localhost/profile.php/?username=Bob I was wondering, if I had a separate <title> which changed according to the username, would they produce separate ...
2
votes
1answer
378 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
1
vote
1answer
420 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
2
votes
1answer
107 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
4
votes
3answers
154 views

How can I make Google crawl and index multiple pages?

There are several pages on the web that are not under my direct control but contain content about a product I'm involved with. I'd like make Google robots crawl and index those pages, is there a ...
1
vote
1answer
303 views

Checking Request's user agent versus handling _escaped_fragment_ urls when using pushState only (no hashbangs)

I'm working on an AJAX-based application and thinking of how to make it crawlable by search engines. I've read the Google's AJAX crawling specification but I couldn't get my head around one thing: if ...
1
vote
2answers
154 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
2
votes
1answer
259 views

How to use NOINDEX,FOLLOW with paginated results except index page

I am running a classified website. At the moment I am using lazy loading with jquery to load content while scrolling. I got to know that using jquery loading is not good for search-engine bots ...
1
vote
2answers
183 views

Fake Links Showing up in Google Webmaster Tools

Google Webmaster tools is reporting a large increase in 'not found errors' on my site, and I see approximately 1700 'Not Found Errors over the past 2 months. All of them are for a url that is ...
2
votes
4answers
540 views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
0
votes
2answers
223 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
0
votes
2answers
128 views

Opening forum for search engine bots and SEO

We develop and sell components for other software developers through our website. The main products are ActiveX and WinFroms.NET grid controls. Some years ago we established a forum dedicated to our ...
0
votes
0answers
79 views

Is it OK to use translations obtained via crawling a translation service with advertising?

Google and Bing not only have their API-based translation, but also their websites which allow for one-off translation. While these are intended for use by humans, this also allows for a webcrawler ...
1
vote
1answer
42 views

How to write robots.txt for one hosting which has several websites in different directories?

I have several websites on one hosting account. The main website is in the root. Other websites are in sub-folders off the root directory. In the robots.txt file for the main website, do I need to ...
2
votes
2answers
2k views

Google keeps crawling pages that don't exist

I made a simple new website which went live about a month and a half ago (replacing the last one). The site contains new pages, like for example the navigation markup: <div class="header"> ...
2
votes
1answer
261 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
3
votes
2answers
177 views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 ...
0
votes
1answer
21 views

Indexing a page in which the content is discovered in a map

We want to improve our site's SEO. The page is a GWT (javascript based) web page and its content is discovered through a map or a GPS. Now we have just a normal landing html page, and the map and ...
0
votes
1answer
101 views

News like pages get index, but not showing up in SERP

Well I have some crawlers to scrape news from different news agency websites and auto-publish it to my own website, the thing is my pages get index in Google, but even if I search the exact same ...
1
vote
0answers
70 views

SEO strategy for multiple pages per entity [closed]

I'm trying to decide a good SEO strategy for a web-site with multiple pages per entity. For example, there are main pages for companies which summarize the main information and each sub-page and ...
0
votes
0answers
16 views

Redirecting all 404 pages in Wordpress is not good SEO, but what is? [duplicate]

Maybe a silly question, but if you have a 404 error, than Google does not want to index your site. Of course you are going to say: 'who said that, that's not true', I know... But Google does not like ...
1
vote
2answers
418 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
1answer
107 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
0
votes
2answers
93 views

Google not crawling my website [duplicate]

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling. Now we had ...
2
votes
2answers
230 views

How to do a JavaScript redirection to the same page without hurting my SEO?

I did something that does that when people land on my page for the first time, they get a JavaScript redirection to the same page, but with PHP $_GET URL queries, then this page redirect back to the ...
0
votes
1answer
51 views

Hidden page, but still crawlable?

Is it possible to hide a page from users, but still have search engines crawl the content? I have RSS feeds which are really messy and don't fit my theme, but they have content that I would benefit ...
0
votes
1answer
118 views

Does operating a web crawler/scraper use up my website bandwidth?

I would like to operate a website crawler/scraper from my web server (doing so ethically of course). What I am wondering is whether this uses my website bandwidth up? If so how badly and is it ...
1
vote
3answers
112 views

How to implement an unspammable email address

Providing info email or contact email in plain text helps spammer (robots) to extract the email using regex or any technique, currently I provided an image like this: But this method has usability ...
1
vote
1answer
65 views

What are the future implications of blocking a search engine now?

At a question asked previously at this site regarding Baidu's overactive crawling tendencies, one user comments: I'd only block a search engine spider if you never want to receive any search ...
8
votes
3answers
1k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
1
vote
3answers
9k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
2
votes
2answers
184 views

Googlebot algorithm for crawling?

Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?
2
votes
1answer
96 views

Google Adsense before/above article in Wordpress: SEO problem?

I have set a banner before my article / content in Wordpress on my single.php. Will this be a problem if we look at it in SEO terms? So that Google can't crawl my article or crawl it less good because ...
2
votes
1answer
67 views

Avoiding crawl time for noindex pages

I have pages with noindex in robots meta tag (specifically inner site search results pages). Because my site has many pages and Google hasn't crawled them all and I'm trying to help his crawler focus ...
1
vote
1answer
104 views

Google AdSense crawler unable to crawl my image-based website

I have recently set up a website and applied for Google AdSense. However my website is being not crawled properly. It is basically a image-based website with text under each image. It contains a lot ...
1
vote
3answers
211 views

Right way to allow all access to website in robots.txt?

user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?
1
vote
1answer
73 views

What is the proper SEO way to add and remove links to our site and sitemap?

As content fluctuates on our site, we will obviously add the titles to the new pages to our sitemap, and link list for search engine indexing. Through time, certain links will become less relevant, ...
2
votes
1answer
545 views

How do I block a user-agent from Apache

How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all bots from Apache on my debian server, that have the ...
1
vote
2answers
108 views

Is it possible for Google to discover a website that is not linked to anywhere? [duplicate]

My colleague claims that he has seen cases where websites were "discovered" by Google without having a link to them anywhere on the web. For him, this means that the only way to not being indexed ...
1
vote
3answers
201 views

How does Googlebot crawl a webpage? [closed]

Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
3
votes
4answers
2k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
6
votes
1answer
237 views

Will Google and other search engines follow IP address only sites (i.e., no URL)

Will Google and other search engines follow links to and index sites that don't have a URL, but instead just use their IP address's (obviously they would need a unique IP) ?