Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

2
votes
0answers
160 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
3
votes
1answer
562 views

Checking site for meta refresh redirects

How do you check or crawl site (couple of urls) to explore existing meta refresh redirects? Screaming Frog doesn't handle them - it indicates a page with meta refresh as 200 (it is partly true), even ...
2
votes
1answer
130 views

How to check whether the infinite page scroll data is crawled and indexed by google or not?

I am having a music platform website where at a time 20 songs are visible right now then after scroll the other 20 are visible. So how can I confirm whether the songs which are coming after scrolling ...
2
votes
3answers
98 views

How can search engines find my RSS feed?

I have a small blog. Now, I have created an rss.xml. I put that in the root of the site, on the server. Should I do anything to make search engines like Google, Technorati, Yahoo! be able to find it? ...
3
votes
1answer
356 views

Google Search Console: 404 errors on existing pages

There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
1
vote
0answers
9 views

Should i create seperate page for every ad for search engine indexing purposes in my website [closed]

I am going to create a simple ad marketing web site. so I should use databases as my storing method. Now my question is If I want to show my customers ads in Google search result page(just like ...
2
votes
0answers
68 views

Which one is more SEO friendly Dynamic pages or static pages for blogging website

I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
2
votes
1answer
613 views

“Is robots.txt blocking important pages?” in Search Console regarding static folder

This is the contents of my robots.txt file: User-agent: * Disallow: /static/* Disallow: /templates/* Disallow: /translations/* Sitemap: https://xx.xxx.com/sitemap.xml However I don't know why I am ...
3
votes
1answer
98 views

Does a link rel canonical tag pointing back to the page itself cause an infinite loop that wastes crawl budget?

I assume that search engine robots crawl the whole page and then the canonical tag because pages that are self canonicalized still get indexed and their contents are still parsed. However, does a ...
3
votes
1answer
71 views

robots.txt for website hosted in a subdirectory

I have 2 website which is hosted in a shared hosting. 1st website example.com hosted in a root directory as /public_html/ 2nd website example2.com hosted in a sub-directory directory as /public_html/...
1
vote
3answers
69 views

How is noFollow enforced on sites like Quora and Facebook?

I'm curious to know how search engines like Google enforce their noFollow policy on social sites. It seems like it would be largely out of their control, especially for webpages that cannot be crawled....
6
votes
1answer
662 views

Block Yandex crawler

Our site has been behaving very strangely for the last few days, lots of time outs etc. Finally think I found the cause, the Yandex bot is crawling around 10,000 pages an hour! I need to stop it ...
1
vote
1answer
236 views

Crawler says, page not found, but browser says otherwise

I'm stumped. Google has been reporting increasing numbers of 404 errors for my website. But my site is static. Ok. Maybe they just finding more as different chunks are scanned. At the same time,...
0
votes
0answers
64 views

I need the subdomain in cpanel to not be followed. Help!

I have little knowledge of robot.txt but I know how meta follow and no follow tags work. The problem is I have 2 totally different websites, one being the primary and the other an addon site with ...
0
votes
2answers
163 views

Spider-Trap on a GitHub Site

I have a GitHub site and I hate web-crawlers that disobey or ignore robots.txt. How would I set up a Spider-Trap on a GitHub site that the robots.txt disallows the trap, so the good bots aren't ...
1
vote
0answers
31 views

How does a webmaster perform a proper redirect from one domain to another? [closed]

There are a few questions that are inherently present inside the main question. Those questions are: What is the best way to perform this type of redirect? (Presumably a 301 Perma for SEO Best ...
0
votes
2answers
414 views

Is it possible to block search engine indexing using DNS alone?

As per the title, is it possible to block all search engine indexing using DNS? Most guides point towards robots.txt or meta headers but I'd like to investigate the possibility of blocking all ...
5
votes
3answers
214 views

My personal website was cloned in its entirety, is this a security concern?

I was googling my (real) name yesterday (which is the name of my site) on a lark and discovered a website that had copied my personal website and its content. Like a carbon copy including links to ...
0
votes
1answer
532 views

Googlebot not respecting HTTP basic auth

I have basic auth set up and it has always worked. Suddenly Google started crawling my pages. The auth is still there (I have checked it using different browsers). I am at a loss how it's possible. ...
0
votes
2answers
20 views

Do I need an umbrella services page to match the /services/ part of our URLs?

From a usability perspective, this is not necessary. The mega menu dropdown on all pages makes it extremely clear how to navigate to the relevant services we offer. However, I am wondering whether a ...
0
votes
1answer
606 views

SEO issues with Elm?

Do search engines, particularly Google, render JavaScript created from transpiled Elm code when crawling? Can they follow links, even internal that modify the existing page?
2
votes
1answer
272 views

How does Google treat underscores in site map URLs?

Google is currently reporting that my URLs are invalid within my sitemap. Here's an example of a document that was considered erroneous by Google's validator: For each of the URLs in the map, Google ...
0
votes
1answer
365 views

Stopping Google from crawling my static domain

I use a cookieless sub domain static.example.com to serve all images, js, and css files. This static sub domain has as its root directory the same directory as the parent http://example.com. So static....
4
votes
1answer
821 views

Google Analytics traffic surge from China, not real visitor, Baidu?

UPDATE2: Seems this all comes from Baidu crawlers, here's some outputcreated by GoAccess analyzing our logs. We've restricted the the following user-agent which seems to be used by them: Mozilla/5....
3
votes
1answer
313 views

Help with robots meta tag and X-Robots-Tag

I have two questions about robots meta tag and X-Robots-Tag: If a page has both X-Robots-Tag HTTP header and robots meta tag, can this cause problems? For example: On HTTP response headers: X-...
3
votes
1answer
331 views

Can a website have multiple RSS feeds? How would the <link> and <channel> elements look?

If I wanted to, for example, create three RSS feeds on my site to better allow users to subscribe specifically to the section in which they were interested, could I have three separate XML files at ...
2
votes
1answer
353 views

Can Google find and crawl subdomains with no backlinks?

I know Google can find domains by checking with registrars, but is it able to find new subdomains with no backlinks? I suspect they are able to by checking DNS records, but can anyone confirm/deny?
2
votes
1answer
68 views

Blocking vs noindex to reduce crawl requests

I observed that GoogleBot is making a lot of duplicate requests for the same URLs from my website within a week. Amongst these requests a majority were for low/thin value pages(no or very low SERP,not ...
3
votes
1answer
404 views

Google is indexing pages as HTTP despite 301 redirects from HTTP to HTTPS

on 9/23 we migrated our entire site to HTTPS. Almost 3 months later and Google has recently cached versions of http:// URLs in the index. We have an http:// sitemap that is live with 100% 301 ...
1
vote
1answer
110 views

Block Google from crawling low value pages

I recently analysed the week-on-week crawl request made by google bot to my server. Here is the graph for the same. Duplicate requests are when more than 1 request was made by google-bot for the same ...
3
votes
1answer
169 views

Is it bad for SEO to return a 404 for a page that is temporarily unavailable instead of a redirect to the home page?

Imagine a site that contains 100's of 'homepages' for merchants. (e.g. www.example.com/cool_merchant) Some of these pages may exist for long periods, build up a nice SEO presence, and then be ...
1
vote
2answers
94 views

Allow bots to read dynamically injected content

I've got a pretty large Angular SPA, and I currently use ?_escaped_fragment_ to serve up static versions of all our pages. I've discovered, however, that this often has issues with newly deployed/...
4
votes
1answer
93 views

rel=“nofollow” on randomly generated internal links

I have a section on a website where I show a list of posts from the past. The list is randomly generated and it's changing every time a page is loaded. I was wondering, would it be better to add a ...
0
votes
0answers
66 views

Web page is not crawled completely by Google

I have a webpage which is built by converting the docx document to html page. The page is added to the google webmaster tool and the sitemap file also been submitted and maintained regularly. There ...
0
votes
1answer
168 views

Fetch as Google does not result in child pages getting indexed

I work on an exotic pet website which currently has several types/species of reptiles. It has done well in SERP for the first couple of types of reptiles, but I am continuing to add new species and ...
0
votes
0answers
54 views

Googlebot requesting for same page within short period

I was analysing analyzing week over week pages that GoogleBot is requesting for crawling. I can see that GoogleBot is requesting for same page more than 1500 times on a single day. This page is my ...
0
votes
1answer
40 views

Is it OK for SEO to create tutorial documentation pages on my website that are only shown to logged in users?

I'm thinking to create tutorial pages (visible only if logged in) for my client. Using same CMS database. My question is, is this bad for SEO? should I tell robots to ignore these pages?
4
votes
2answers
125 views

Google claims not to find my robots.txt

Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it. But there is a robots.txt file (https://soundhaus-band....
7
votes
2answers
698 views

Do search engines still crawl a noindex page

Do search engines crawl a page that has a 'noindex' meta attribute? The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex'...
4
votes
2answers
699 views

Why are Google Search Console 404 errors not showing up in my own site crawl?

I have exported a list of 404 and 403 pages from Google Search Console. I have compared these URLs against my site crawl. The 404 pages in Google Search Console are not within my site crawl pages list....
1
vote
1answer
114 views

Noncrawlable Javascript Pagination Issue

We have just completed a revamp of a website with a new design and I had noticed something when running the sitemap generator for the new site. The new site uses javascript pagination which fits the ...
3
votes
1answer
53 views

Crawling other websites - Can I do so?

I want to build a meta search engine, so I want to crawl other websites and filter/organize the info I get and present them to the user. My questions are: Can I do that(crawl) without having the ...
5
votes
3answers
975 views

Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources

I have a robots.txt file: User-agent:* Disallow:/path/page Disallow:/path/ Allow:/ The disallowed path is still getting crawled. I have searched this problem and what they said, order of precedence ...
4
votes
1answer
328 views

GoogleBot crawling a page up to 3,000 times per day

For a website, GoogleBot daily crawls one particular URL more than one thousand times (mainly between 1,500 and 3,000, sometimes much more). The Googlebot IP is a real Google one (for example : 66....
1
vote
1answer
70 views

Allow Googlebot to crawl dynamically generated product pages

I have been doing some reading into how Googlebot crawls javascript generated webpages but am still a little confused on it. Say I have a web page for company XYZ Inc that has 5 general products like ...
5
votes
1answer
1k views

Why do many websites block requests from common HTTP libraries by user-agent?

Writing spiders, I have noticed that many sites will return a 403 error if I hit them from popular HTTP software libraries, unless I manually override the default User-Agent header used by the library....
3
votes
1answer
34 views

Allow crawlers to pass link juice from similar and duplicated user created pages

My users commonly create new pages with similar/duplicate content. For example: http://example.com/post?123 http://example.com/post?456 I've therefore disallowed these pages in the robots.txt file. ...
1
vote
0answers
47 views

Dealing with Google spidering JavaScript incorrectly

Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot? For ...
1
vote
1answer
110 views

Using asteriks in robots.txt

Can spiders crawl the sites if there are no asterisks used on the robots.txt? For example: User-agent: * Disallow: .*/ping/.* then User-agent: * Disallow: /ping/ Please can someone confirm this, ...
0
votes
0answers
104 views

How to make sure crawlers and Google use an HTML search form?

I'm trying to make crawlers reference the results of a search form on a website. The form itself is written in AngularJS and the various filters trigger a jsonp call towards a solr server. The form is ...