Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
1
vote
1answer
366 views

Crawler says, page not found, but browser says otherwise

I'm stumped. Google has been reporting increasing numbers of 404 errors for my website. But my site is static. Ok. Maybe they just finding more as different chunks are scanned. At the same time,...
0
votes
2answers
180 views

Spider-Trap on a GitHub Site

I have a GitHub site and I hate web-crawlers that disobey or ignore robots.txt. How would I set up a Spider-Trap on a GitHub site that the robots.txt disallows the trap, so the good bots aren't ...
1
vote
0answers
33 views

How does a webmaster perform a proper redirect from one domain to another? [closed]

There are a few questions that are inherently present inside the main question. Those questions are: What is the best way to perform this type of redirect? (Presumably a 301 Perma for SEO Best ...
0
votes
2answers
689 views

Is it possible to block search engine indexing using DNS alone?

As per the title, is it possible to block all search engine indexing using DNS? Most guides point towards robots.txt or meta headers but I'd like to investigate the possibility of blocking all ...
4
votes
3answers
225 views

My personal website was cloned in its entirety, is this a security concern?

I was googling my (real) name yesterday (which is the name of my site) on a lark and discovered a website that had copied my personal website and its content. Like a carbon copy including links to ...
0
votes
1answer
732 views

Googlebot not respecting HTTP basic auth

I have basic auth set up and it has always worked. Suddenly Google started crawling my pages. The auth is still there (I have checked it using different browsers). I am at a loss how it's possible. ...
0
votes
2answers
21 views

Do I need an umbrella services page to match the /services/ part of our URLs?

From a usability perspective, this is not necessary. The mega menu dropdown on all pages makes it extremely clear how to navigate to the relevant services we offer. However, I am wondering whether a ...
0
votes
1answer
732 views

SEO issues with Elm?

Do search engines, particularly Google, render JavaScript created from transpiled Elm code when crawling? Can they follow links, even internal that modify the existing page?
2
votes
1answer
347 views

How does Google treat underscores in site map URLs?

Google is currently reporting that my URLs are invalid within my sitemap. Here's an example of a document that was considered erroneous by Google's validator: For each of the URLs in the map, Google ...
0
votes
1answer
473 views

Stopping Google from crawling my static domain

I use a cookieless sub domain static.example.com to serve all images, js, and css files. This static sub domain has as its root directory the same directory as the parent http://example.com. So static....
4
votes
1answer
1k views

Google Analytics traffic surge from China, not real visitor, Baidu?

UPDATE2: Seems this all comes from Baidu crawlers, here's some outputcreated by GoAccess analyzing our logs. We've restricted the the following user-agent which seems to be used by them: Mozilla/5....
2
votes
1answer
351 views

Help with robots meta tag and X-Robots-Tag

I have two questions about robots meta tag and X-Robots-Tag: If a page has both X-Robots-Tag HTTP header and robots meta tag, can this cause problems? For example: On HTTP response headers: X-...
2
votes
1answer
569 views

Can a website have multiple RSS feeds? How would the <link> and <channel> elements look?

If I wanted to, for example, create three RSS feeds on my site to better allow users to subscribe specifically to the section in which they were interested, could I have three separate XML files at ...
2
votes
1answer
421 views

Can Google find and crawl subdomains with no backlinks?

I know Google can find domains by checking with registrars, but is it able to find new subdomains with no backlinks? I suspect they are able to by checking DNS records, but can anyone confirm/deny?
2
votes
1answer
77 views

Blocking vs noindex to reduce crawl requests

I observed that GoogleBot is making a lot of duplicate requests for the same URLs from my website within a week. Amongst these requests a majority were for low/thin value pages(no or very low SERP,not ...
2
votes
1answer
510 views

Google is indexing pages as HTTP despite 301 redirects from HTTP to HTTPS

on 9/23 we migrated our entire site to HTTPS. Almost 3 months later and Google has recently cached versions of http:// URLs in the index. We have an http:// sitemap that is live with 100% 301 ...
1
vote
1answer
113 views

Block Google from crawling low value pages

I recently analysed the week-on-week crawl request made by google bot to my server. Here is the graph for the same. Duplicate requests are when more than 1 request was made by google-bot for the same ...
2
votes
1answer
198 views

Is it bad for SEO to return a 404 for a page that is temporarily unavailable instead of a redirect to the home page?

Imagine a site that contains 100's of 'homepages' for merchants. (e.g. www.example.com/cool_merchant) Some of these pages may exist for long periods, build up a nice SEO presence, and then be ...
1
vote
2answers
109 views

Allow bots to read dynamically injected content

I've got a pretty large Angular SPA, and I currently use ?_escaped_fragment_ to serve up static versions of all our pages. I've discovered, however, that this often has issues with newly deployed/...
3
votes
1answer
108 views

rel=“nofollow” on randomly generated internal links

I have a section on a website where I show a list of posts from the past. The list is randomly generated and it's changing every time a page is loaded. I was wondering, would it be better to add a ...
0
votes
1answer
175 views

Fetch as Google does not result in child pages getting indexed

I work on an exotic pet website which currently has several types/species of reptiles. It has done well in SERP for the first couple of types of reptiles, but I am continuing to add new species and ...
0
votes
1answer
41 views

Is it OK for SEO to create tutorial documentation pages on my website that are only shown to logged in users?

I'm thinking to create tutorial pages (visible only if logged in) for my client. Using same CMS database. My question is, is this bad for SEO? should I tell robots to ignore these pages?
3
votes
2answers
133 views

Google claims not to find my robots.txt

Like I said in the title, the Google Webmaster Console claims not to find the robots.txt file of my homepage and therefore it won't crawl it. But there is a robots.txt file (https://soundhaus-band....
7
votes
2answers
915 views

Do search engines still crawl a noindex page

Do search engines crawl a page that has a 'noindex' meta attribute? The reason I ask is because we have near duplicate content caused by faceted navigation and the filtered pages have stated 'noindex'...
3
votes
2answers
745 views

Why are Google Search Console 404 errors not showing up in my own site crawl?

I have exported a list of 404 and 403 pages from Google Search Console. I have compared these URLs against my site crawl. The 404 pages in Google Search Console are not within my site crawl pages list....
1
vote
1answer
124 views

Noncrawlable Javascript Pagination Issue

We have just completed a revamp of a website with a new design and I had noticed something when running the sitemap generator for the new site. The new site uses javascript pagination which fits the ...
2
votes
1answer
60 views

Crawling other websites - Can I do so?

I want to build a meta search engine, so I want to crawl other websites and filter/organize the info I get and present them to the user. My questions are: Can I do that(crawl) without having the ...
5
votes
3answers
1k views

Robots.txt with only Disallow and Allow directives is not preventing crawling of disallowed resources

I have a robots.txt file: User-agent:* Disallow:/path/page Disallow:/path/ Allow:/ The disallowed path is still getting crawled. I have searched this problem and what they said, order of precedence ...
3
votes
1answer
451 views

GoogleBot crawling a page up to 3,000 times per day

For a website, GoogleBot daily crawls one particular URL more than one thousand times (mainly between 1,500 and 3,000, sometimes much more). The Googlebot IP is a real Google one (for example : 66....
1
vote
1answer
78 views

Allow Googlebot to crawl dynamically generated product pages

I have been doing some reading into how Googlebot crawls javascript generated webpages but am still a little confused on it. Say I have a web page for company XYZ Inc that has 5 general products like ...
4
votes
1answer
2k views

Why do many websites block requests from common HTTP libraries by user-agent?

Writing spiders, I have noticed that many sites will return a 403 error if I hit them from popular HTTP software libraries, unless I manually override the default User-Agent header used by the library....
2
votes
1answer
35 views

Allow crawlers to pass link juice from similar and duplicated user created pages

My users commonly create new pages with similar/duplicate content. For example: http://example.com/post?123 http://example.com/post?456 I've therefore disallowed these pages in the robots.txt file. ...
1
vote
0answers
56 views

Dealing with Google spidering JavaScript incorrectly

Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot? For ...
1
vote
1answer
141 views

Using asteriks in robots.txt

Can spiders crawl the sites if there are no asterisks used on the robots.txt? For example: User-agent: * Disallow: .*/ping/.* then User-agent: * Disallow: /ping/ Please can someone confirm this, ...
1
vote
2answers
223 views

Excluding Google Analytics spam by referral

I've been reading up on how to filter out spam from Google Analytics. In this case, I'm trying to filter out the spam that shows up under referrals. In the Filter Pattern field of Analytics, I have ...
3
votes
1answer
819 views

Are Googlebot and other search engine crawlers able to detect if element is hidden after page load by JavaScript?

So, we all know Google hates hidden elements, and rightly so. But I am curious, what if the element is present in DOM normally in the static form, but after page load, it is hidden by JavaScript? ...
1
vote
3answers
152 views

Applebot not crawling sitemap.xml

On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages. We have no robots.txt restrictions for that bot, on the contrary we ...
3
votes
4answers
838 views

How to do I stop a bot from from excessively crawling my Blogger site?

I just bought a domain for my Blogger site. I noticed an unusual rise in my page views, and I tried to trace the source of the traffic but it's a dead end... This excessive bot traffic got my AdSense ...
1
vote
1answer
25 views

How can I increased my site traffic [duplicate]

My website releated to entertainment and movies trailer and review based site, my site viewers list actually very low, I have set keywords which is found from google keywords planner. And created ...
1
vote
0answers
48 views

Google doesn't index my pages managed by session

My website has an architecture like this: em1 sub1 sub2 ... em2 sub1 sub2 ... em3 ... ... All elements "em" has the same "sub" associated. The difference only resides on the content in the "...
1
vote
1answer
256 views

Google not re-crawling subpages after “fetch as Google” with the “directly linked pages” option

After fixing a "noindex" issue on my WordPress website, where all my pages failed to be crawled by Google, I used Google Webmaster to re-index the website (without waiting). But it worked only for the ...
0
votes
1answer
296 views

Google Search Console: 404 errors that aren't linked to

I manage an ecommerce website that slightly refreshes its product/category collection every night. Sometimes products/categories get disabled and links to them disappear from the website but despite ...
1
vote
0answers
52 views

Sitemap with an App using Codeigniter and jQuery Mobile?

We've got dynamic content that we want the google bot to search. Our app is running codeingiter and jQuery mobile. So, our site is basically 6 pages that are updated using XHR calls. The data is ...
1
vote
2answers
421 views

Should I submit both mobile and desktop domain names for indexing on non-responsive sites or is that bad for SEO?

I have a desktop version of my site using yourwebsite.com and a mobile version using m.yourwebsite.com. Should I treat them as one site and only submit the desktop version along with its sitemap and ...
5
votes
1answer
161 views

Can Googlebot discover URLs that are not in an anchor?

In my Search Console I had a sudden increase in 404's. They were linked from a webpage that listed files from the site in a table but none of these URLs were nested inside an <a> tag. This was ...
2
votes
0answers
55 views

Browser's auto request detection with X-Purpose

3 to 4 months ago, I found out that safari/chrome/firefox all have feature that automatically send request with X-Purpose header (with preview value for Safari) to get live thumbnail or top sites ...
1
vote
1answer
243 views

Unexplainable non-existing page requested for months

While I am quite familiar with bots requesting well-know pages as used for WordPress and phpMyAdmin on any site the can reach, I'm now dealing with a bot requested a page I really can't explain. For ...
2
votes
0answers
145 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
0
votes
1answer
107 views

Does Web crawler simulate user behavior?

Structuring the web pages' content or schema properly is important for SEO. I would like to know if popular web crawlers or bots (Google bot, Bingbot etc...) are able to simulate user behavior and ...
1
vote
2answers
707 views

At what resolution does Google Bot Web Crawlers consider desktop?

I'm aware that google evaluates how a site looks on mobile. At what resolution does google evaluate desktop sites? My site uses a responsive layout up until 1200px width, at which point it shows and ...