Tagged Questions

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

2
votes
1answer
367 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However ...
1
vote
1answer
409 views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
2
votes
1answer
107 views

What is the difference between Disallow and Allow?

I had 2 questions in my mind: what is the difference between Allow and Disallow? which one should I use to allow Google bot for AdSense? User-agent: Mediapartners-Google Disallow: User-agent: ...
4
votes
3answers
149 views

How can I make Google crawl and index multiple pages?

There are several pages on the web that are not under my direct control but contain content about a product I'm involved with. I'd like make Google robots crawl and index those pages, is there a ...
1
vote
1answer
292 views

Checking Request's user agent versus handling _escaped_fragment_ urls when using pushState only (no hashbangs)

I'm working on an AJAX-based application and thinking of how to make it crawlable by search engines. I've read the Google's AJAX crawling specification but I couldn't get my head around one thing: if ...
1
vote
2answers
152 views

Why is Yahoo bot hitting a page when my robots.txt file is configured to disallow all bots?

My robots.txt: User-agent: * Disallow: / A page two directories below the root is being hit by a Yahoo bot and getting a 404: HTTP_REFERER: [empty string] HTTP_USER_AGENT: Mozilla/5.0 (compatible; ...
2
votes
1answer
253 views

How to use NOINDEX,FOLLOW with paginated results except index page

I am running a classified website. At the moment I am using lazy loading with jquery to load content while scrolling. I got to know that using jquery loading is not good for search-engine bots ...
1
vote
2answers
176 views

Fake Links Showing up in Google Webmaster Tools

Google Webmaster tools is reporting a large increase in 'not found errors' on my site, and I see approximately 1700 'Not Found Errors over the past 2 months. All of them are for a url that is ...
2
votes
4answers
510 views

Google crawl speed — how fast can it go?

I have a huge website with 5 million pages. Currently Google indexes about 10,000 pages per day. This is very slow, I still have lots of pages that I can't get indexed. Does anyone know what is the ...
0
votes
2answers
204 views

Does Submit to Index on a page with new content update Content Keywords for the site?

Using Google Webmaster Tools I'm trying to update the Content Keywords of my site. I'm confused about the relationship between Submit to Index and Content Keywords Does Fetch as Google --> Submit to ...
0
votes
2answers
125 views

Opening forum for search engine bots and SEO

We develop and sell components for other software developers through our website. The main products are ActiveX and WinFroms.NET grid controls. Some years ago we established a forum dedicated to our ...
0
votes
0answers
77 views

Is it OK to use translations obtained via crawling a translation service with advertising?

Google and Bing not only have their API-based translation, but also their websites which allow for one-off translation. While these are intended for use by humans, this also allows for a webcrawler ...
1
vote
1answer
42 views

How to write robots.txt for one hosting which has several websites in different directories?

I have several websites on one hosting account. The main website is in the root. Other websites are in sub-folders off the root directory. In the robots.txt file for the main website, do I need to ...
2
votes
2answers
2k views

Google keeps crawling pages that don't exist

I made a simple new website which went live about a month and a half ago (replacing the last one). The site contains new pages, like for example the navigation markup: <div class="header"> ...
2
votes
1answer
246 views

Is there a reason msnbot crawls in spikes?

I've been experiencing high RPM spikes recently. Something like this: When I debugged, I've found reasons to believe the reason is the msnbot suddenly makes a massive crawl and then stops. I assume ...
3
votes
2answers
165 views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 ...
0
votes
1answer
21 views

Indexing a page in which the content is discovered in a map

We want to improve our site's SEO. The page is a GWT (javascript based) web page and its content is discovered through a map or a GPS. Now we have just a normal landing html page, and the map and ...
0
votes
1answer
101 views

News like pages get index, but not showing up in SERP

Well I have some crawlers to scrape news from different news agency websites and auto-publish it to my own website, the thing is my pages get index in Google, but even if I search the exact same ...
1
vote
0answers
70 views

SEO strategy for multiple pages per entity [closed]

I'm trying to decide a good SEO strategy for a web-site with multiple pages per entity. For example, there are main pages for companies which summarize the main information and each sub-page and ...
0
votes
0answers
16 views

Redirecting all 404 pages in Wordpress is not good SEO, but what is? [duplicate]

Maybe a silly question, but if you have a 404 error, than Google does not want to index your site. Of course you are going to say: 'who said that, that's not true', I know... But Google does not like ...
1
vote
2answers
400 views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
1
vote
1answer
106 views

Server overhead caused by bots?

I have one customer website causing overhead (http://www.modacalcio.it/en/by-kind/football-boots.html). With htop opened, I am trying navigate the website and the much load of the website is done by ...
0
votes
2answers
93 views

Google not crawling my website [duplicate]

I own a website http://citrusbug.com. We did not have proper content before so to prevent search engine from crawling our site we added a robots.txt with code to prevent crawling. Now we had ...
2
votes
2answers
227 views

How to do a JavaScript redirection to the same page without hurting my SEO?

I did something that does that when people land on my page for the first time, they get a JavaScript redirection to the same page, but with PHP $_GET URL queries, then this page redirect back to the ...
0
votes
1answer
49 views

Hidden page, but still crawlable?

Is it possible to hide a page from users, but still have search engines crawl the content? I have RSS feeds which are really messy and don't fit my theme, but they have content that I would benefit ...
0
votes
1answer
114 views

Does operating a web crawler/scraper use up my website bandwidth?

I would like to operate a website crawler/scraper from my web server (doing so ethically of course). What I am wondering is whether this uses my website bandwidth up? If so how badly and is it ...
1
vote
3answers
111 views

How to implement an unspammable email address

Providing info email or contact email in plain text helps spammer (robots) to extract the email using regex or any technique, currently I provided an image like this: But this method has usability ...
1
vote
1answer
65 views

What are the future implications of blocking a search engine now?

At a question asked previously at this site regarding Baidu's overactive crawling tendencies, one user comments: I'd only block a search engine spider if you never want to receive any search ...
8
votes
3answers
1k views

Is it possible to slow the Baiduspider crawl frequency?

Much has been made of the Baidu spider crawl frequency. It's true: "Baiduspider crawls like crazy." I've experienced this phenomenon at sites I work with. In at least one instance, I've found that ...
1
vote
3answers
9k views

What does “Disallow: /search” mean in robots.txt?

In my blog's Google Webmaster Tools panel, I found the following code in my robots.txt of blocked URLs section. User-agent: Mediapartners-Google Disallow: /search Allow: / I know that Disallow will ...
2
votes
2answers
174 views

Googlebot algorithm for crawling?

Is there some general description telling how Google crawls your website? I mean, not the frequency, but which URL is it going to explore next?
2
votes
1answer
96 views

Google Adsense before/above article in Wordpress: SEO problem?

I have set a banner before my article / content in Wordpress on my single.php. Will this be a problem if we look at it in SEO terms? So that Google can't crawl my article or crawl it less good because ...
2
votes
1answer
67 views

Avoiding crawl time for noindex pages

I have pages with noindex in robots meta tag (specifically inner site search results pages). Because my site has many pages and Google hasn't crawled them all and I'm trying to help his crawler focus ...
1
vote
1answer
104 views

Google AdSense crawler unable to crawl my image-based website

I have recently set up a website and applied for Google AdSense. However my website is being not crawled properly. It is basically a image-based website with text under each image. It contains a lot ...
1
vote
3answers
209 views

Right way to allow all access to website in robots.txt?

user-agent: * disallow: Is this the right way to allow all access to all content of a website to crawlers in a robots.txt?
1
vote
1answer
73 views

What is the proper SEO way to add and remove links to our site and sitemap?

As content fluctuates on our site, we will obviously add the titles to the new pages to our sitemap, and link list for search engine indexing. Through time, certain links will become less relevant, ...
2
votes
1answer
511 views

How do I block a user-agent from Apache

How do I realize a UA string block by regular expression in the config files of my Apache webserver? For example: if I would like to block out all bots from Apache on my debian server, that have the ...
1
vote
2answers
107 views

Is it possible for Google to discover a website that is not linked to anywhere? [duplicate]

My colleague claims that he has seen cases where websites were "discovered" by Google without having a link to them anywhere on the web. For him, this means that the only way to not being indexed ...
1
vote
3answers
199 views

How does Googlebot crawl a webpage? [closed]

Today I had a silly fight with my colleague. My friend told me, "Googlebot crawls a web page only through the links present in webpage or through the number of page views (by text ads)." But I ...
3
votes
4answers
2k views

Do you really need robots.txt and sitemap.xml if I want bot to follow every link? [duplicate]

I wonder if I really need robots.txt/*sitemap.xml* files if I want bots to go through every link on my website without any restrictions? And if I do, what should I write in there?
6
votes
1answer
230 views

Will Google and other search engines follow IP address only sites (i.e., no URL)

Will Google and other search engines follow links to and index sites that don't have a URL, but instead just use their IP address's (obviously they would need a unique IP) ?
4
votes
1answer
70 views

Appropriate robots meta for “printer friendly” pages

We have many printer-friendly pages that have a link that adds ?pf=1 to the URL. On the ?pf=1 page would it be better to have noindex,nofollow or noindex,follow. I think that it would be better to ...
1
vote
1answer
296 views

Stop Google crawling links completely

I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told ...
2
votes
1answer
199 views

How can Google crawl content accessed via filter check-boxes?

I built a website that is divided into many categories. Each page displays the main category, and within that there is a check-box filter (like Amazon and eBay have when searching for items). I want ...
2
votes
1answer
394 views

Will Google crawl session based website

I have a website, it is split into 3 categories but using PHP its an all-in one kind of style. When a user chooses a category on the home page a session is set, this is then used to set the style and ...
1
vote
1answer
471 views

Bingbot Crawling URLS Like Folders, Causing Thousands Of 404 Errors

I first started seeing this a little over two months ago on a site I run but now I am seeing it on several others I manage. It appears the bot is trying to parse the URL into folders and this is ...
1
vote
1answer
139 views

Ahrefs is checking my site for backlinks all the time

I have a Magento site and I realize that the ahrefs.com crawler visits my site all the time everyday. I visit ahrefs.com and I check my backlinks (they are good). Is it good or bad, something like ...
1
vote
3answers
97 views

Using Ajax #! for Google but site is not being crawled any more

We used to have all links in a normal way (thus with parameters?), we changed to AJAX loading cause we installed a music player. Here's an example: Old link: ...
3
votes
1answer
853 views

How to make “crawl-delay” works for all “user-agent” in robots.txt or in .htaccess?

I have this line: Crawl-delay: 50 in the robots.txt file at the root of my website. But it seems that it is not working at all. I get 400 visits from the Yandex user agent in a few minutes. I ...
4
votes
2answers
269 views

Google Webmaster Tools reports fake 404 errors

I have a website where Google Webmaster Tools reports 15,000 links as 404 errors. However, all links return a 200 when I visit them. The problem is, that eventhough I can visit these pages and return ...