Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

43 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
1answer
793 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
2
votes
0answers
20 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
2
votes
0answers
516 views

URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
2
votes
0answers
39 views

Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?

I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds. The ...
2
votes
0answers
563 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
2
votes
0answers
29 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
2
votes
0answers
155 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
2
votes
0answers
109 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
2
votes
0answers
199 views

What % can be regarded as “normal” “not viewed traffic” reported in AWStats?

Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"? Is my site under attack when ...
2
votes
0answers
205 views

Google is adding “Archive” to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
2
votes
0answers
202 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
2
votes
0answers
72 views

Which one is more SEO friendly Dynamic pages or static pages for blogging website

I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
2
votes
0answers
54 views

Browser's auto request detection with X-Purpose

3 to 4 months ago, I found out that safari/chrome/firefox all have feature that automatically send request with X-Purpose header (with preview value for Safari) to get live thumbnail or top sites ...
2
votes
0answers
143 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
2
votes
0answers
753 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
2
votes
0answers
884 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
2
votes
1answer
448 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
0answers
11 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
1
vote
0answers
6 views

Should we apply “noreferrer” on “Read More” or “View All”?

I experienced years back a search engine crawler issue that Google started showing "Read More" as title of the page in it's search results. Because that page was linked through "read more" link on new ...
1
vote
1answer
25 views

Form-locked PDFs in JSON-LD

I'm setting up downloads functionality with form-locked PDFs - available after the user fills-out a form. I want the PDFs to be indexable by search engines though. Im trying to create a JSON-LD markup,...
1
vote
0answers
18 views

How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?

I have a hypothetical page, that contains pages like: example.com/see/cat example.com/see/mouse example.com/see/<basically anything> I can have thousands of such pages. The number of pages ...
1
vote
0answers
28 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
1
vote
1answer
120 views

Clients from Microsoft IP address ranges seem to be spoofing Bingbot

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block ...
1
vote
0answers
15 views

301 redirects to a similar domain

My company has two similar but not identical sites. Similar page hierarchy and content but different brands. We want to shut one down and 301 the pages to what we feel is relevant on the better site. ...
1
vote
0answers
22 views

Facing a challenge with the “Fetch As Google” feature in my webmaster

I am facing a challenge with the "Fetch As Google" feature in my webmaster Steps to reproduce bug:- 1. Goto webmaster , click on "Crawl" -> "Fetch As Google" 2. Select "Crawl this URL and its direct ...
1
vote
0answers
138 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
32 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
1
vote
0answers
35 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...
1
vote
0answers
54 views

Dealing with Google spidering JavaScript incorrectly

Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot? For ...
1
vote
0answers
48 views

Google doesn't index my pages managed by session

My website has an architecture like this: em1 sub1 sub2 ... em2 sub1 sub2 ... em3 ... ... All elements "em" has the same "sub" associated. The difference only resides on the content in the "...
1
vote
0answers
51 views

Sitemap with an App using Codeigniter and jQuery Mobile?

We've got dynamic content that we want the google bot to search. Our app is running codeingiter and jQuery mobile. So, our site is basically 6 pages that are updated using XHR calls. The data is ...
1
vote
0answers
162 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
1
vote
0answers
37 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is considered ...
1
vote
0answers
135 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) /Guitars/Children%C3%C6%92%E2%E2%82%AC%C5%A1%...
0
votes
0answers
9 views

Will having a sitemap ensure pages served by AJAX requests be crawled?

I have a website, where I publish articles, that I have started just 2 weeks ago. I want to keep the pages as clean as possible and load more content (links to more articles) by AJAX requests upon ...
0
votes
0answers
17 views

If all web pages are interchangeably linked by taxonomy (categories), do I still need a sitemap for SEO?

I have a mediaWiki web site with about 350 web pages. If all web pages, without exception, are interchangeably linked by taxonomy (categories), and moreover, there are also other indicators for web ...
0
votes
0answers
22 views

Google Search Console - Verification Failed

I am trying to add my website to Google Search Console but failed, it returns The connection to your server timed out The file is there, I can open it on the normal browser, all meta tag set to ...
0
votes
0answers
25 views

Invalid traffic coming from Sogou

I have not indexed my website in Sogou. I have only very less visitors. But i am getting traffic from Sogou daily. The search keyword is "site:mywebsitename" and is from China. Just visiting main page....
0
votes
0answers
14 views

Crawler web form submissions with “Crawler typing text” in field

We have a form on our public website that users can use to request an account from us. Just today we had hundreds of form submissions that said "crawler typing text..." in the email and name field, ...
0
votes
0answers
37 views

Google is marking as duplicate my unique URLs powered by Angular Universal

I have a problem in the search console, and google is marking my unique urls as duplicate The application is build with Angular Universal and is rendered in the server Note: This site is only ...
0
votes
0answers
14 views

Why would a bot net keep loading my site?

In blocking a lot of scrapping activity, I found a puzzling situation - that my site is accessed about 35,000 times per day by around 30,000 different IPs. I started blocking these IPs with a script, ...
0
votes
0answers
91 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or directory)...
0
votes
0answers
25 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...