Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
10
votes
8answers
9k views

How long did it take for your new website to be indexed by search engines [duplicate]

Possible Duplicate: Why isn't my website in Google search results? I have added my website to the webmaster tools of some major search engines already, and supplied the sitemap for the ...
10
votes
1answer
712 views

How did Google manage to crawl my 403 pages?

I had a couple private files in a directory on my school folder. You could see that the files existed by going to myschool.edu/myusername/myfolder, but trying to access the files themselves via ...
9
votes
2answers
4k views

Ask.com sitemap crawler down (for good)?

I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
7
votes
2answers
947 views

Do search engines still crawl a noindex page

Do search engines crawl a page that has a 'noindex' meta attribute? The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex'...
5
votes
1answer
173 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
5
votes
2answers
73 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
4
votes
9answers
14k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
3
votes
2answers
5k views

Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?

I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
3
votes
1answer
770 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
3
votes
2answers
2k views

Can Angular project be indexed in Google without Angular Universal?

I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
3
votes
4answers
6k views

Does Google ignore robots.txt [duplicate]

I know that here www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1 it says spiders always check the robots.txt before going to page. However I have recently been told that Google crawls every single ...
3
votes
2answers
452 views

4 URLs lead to the same page content, are 301 redirects better or setting rel canonical better?

All the following pages have the same exact content: http://www.example.com/seo-url (rel canonical URL) http://www.example.com/ID1234 http://www.example.com/content/ID1234 http://www.example.com/...
3
votes
3answers
115 views

How long before Google indexes a new (to me) domain that previously had spam and virus problems?

I registered a new ".com" domain, but when I added it to the Google Search Console I saw that there were problems with viruses and spam problems. I submitted a request for reconsideration. Google ...
3
votes
1answer
78 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
2
votes
1answer
122 views

Does a link rel canonical tag pointing back to the page itself cause an infinite loop that wastes crawl budget?

I assume that search engine robots crawl the whole page and then the canonical tag because pages that are self canonicalized still get indexed and their contents are still parsed. However, does a ...
2
votes
1answer
1k views

Will Googlebot crawl HTTPS URLs that are encrypted by SSL?

This question was asked at StackOverflow, but it doesn't have good references. Does Googlebot crawl HTTPS? (Please provide references.)
2
votes
1answer
674 views

Can a website have multiple RSS feeds? How would the <link> and <channel> elements look?

If I wanted to, for example, create three RSS feeds on my site to better allow users to subscribe specifically to the section in which they were interested, could I have three separate XML files at ...
2
votes
1answer
790 views

Page load speeds effect on crawl rate

We've noticed a big drop in the total pages crawled per day on our site, we have no control over the crawl rate in Google Webmaster Tools so it's possible this has been changed by Google. However it'...
1
vote
1answer
164 views

How to disallow indexing but allow crawling?

In the front page of my website, I have some previews to articles (with a small introduction to them) that link to the full articles. I want to disallow the front page to prevent duplicate content. ...
1
vote
1answer
91 views

What are the future implications of blocking a search engine now?

At a question asked previously at this site regarding Baidu's overactive crawling tendencies, one user comments: I'd only block a search engine spider if you never want to receive any search ...
1
vote
2answers
477 views

Is it possible for Google to discover a website that is not linked to anywhere? [duplicate]

My colleague claims that he has seen cases where websites were "discovered" by Google without having a link to them anywhere on the web. For him, this means that the only way to not being indexed (...
1
vote
2answers
242 views

Adult website crawlability issue

I got a mission and asking for some help with this scenario for a new website. The site is an adult escorts advertising service. It is developed fine, works and looks good, but the issue is with ...
1
vote
3answers
1k views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
1
vote
4answers
342 views

Big discrepancy in number of known pages in Google Search Console and ScreamingFrog

In a technical SEO audit I used both Google Search Console and ScreamingFrog crawl to review the status errors, however there is big discrepancy in the number of pages that Google knows about, and the ...
1
vote
1answer
2k views

301 redirect causing Bingbot to crash, redirection limit reached

Inside my .htaccess file, I have the following script: RewriteEngine on RewriteCond %{HTTPS} off RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301] <IfModule mod_rewrite.c> ...
0
votes
3answers
2k views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=