Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

6
votes
1answer
110 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
2
votes
1answer
438 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
1answer
25 views

Clients from Microsoft IP address ranges seem to be spoofing Bingbot

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block ...
0
votes
1answer
60 views

Error in search console!

the new search console is pointing out usability errors on mobile devices on wordpress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
2
votes
0answers
36 views

Can many single page hits from a variety user agents from the same VPN with the same screen resolution be hurting my search rankings?

I am trying to clarify what is happening with the traffic on my site. Lately I am being flooded with many strange hits, mostly from US and they seem to only open 1 page or visit for few seconds. The ...
2
votes
0answers
125 views

Does Googlebot cache JavaScript files between two pages if they expire in one year?

My question is about javascript websites. Suppose GoogleBot has crawled example.com/index.html?id=1 Does it request all the javascript files again when crawling example.com/index.html?id=2? index....
2
votes
0answers
70 views

Crawling a SalesForce site fully generated by JS with Screaming Frog only gets content on some pages

I'm trying to crawl a Salesforce-generated site. This site appears to be fully generated by Javascript (if you load the page with JS disabled only a white screen shows up). For instance, Salesforce's ...
2
votes
0answers
179 views

What % can be regarded as “normal” “not viewed traffic” reported in AWStats?

Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"? Is my site under attack when ...
2
votes
0answers
163 views

Google is adding “Archive” to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
2
votes
0answers
163 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
2
votes
0answers
68 views

Which one is more SEO friendly Dynamic pages or static pages for blogging website

I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
2
votes
0answers
52 views

Browser's auto request detection with X-Purpose

3 to 4 months ago, I found out that safari/chrome/firefox all have feature that automatically send request with X-Purpose header (with preview value for Safari) to get live thumbnail or top sites ...
2
votes
0answers
117 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
2
votes
0answers
733 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
2
votes
0answers
822 views

Why is GoogleBot crawling HTML pages as JSON using as referrer the one from my android app?

Recently, I started finding in my logs errors caused by missing templates for pages that reasonably should have them, i.e. REQUEST_PATH : /foo/bar.json HTTP_HOST ...
1
vote
0answers
14 views

How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?

I have a hypothetical page, that contains pages like: example.com/see/cat example.com/see/mouse example.com/see/<basically anything> I can have thousands of such pages. The number of pages ...
1
vote
0answers
16 views

When using an image server on a separate domain, should we link directly to the image URL and restrict crawling, or redirect through our domain?

I have a large number of images stored on a separate server, these URLs currently go through a 302 redirect. I think these additional pages are a total waste of time for Google to crawl. At the ...
1
vote
0answers
12 views

301 redirects to a similar domain

My company has two similar but not identical sites. Similar page hierarchy and content but different brands. We want to shut one down and 301 the pages to what we feel is relevant on the better site. ...
1
vote
0answers
15 views

Facing a challenge with the “Fetch As Google” feature in my webmaster

I am facing a challenge with the "Fetch As Google" feature in my webmaster Steps to reproduce bug:- 1. Goto webmaster , click on "Crawl" -> "Fetch As Google" 2. Select "Crawl this URL and its direct ...
1
vote
0answers
71 views

Stop getting traffic from Googlebot on high server load

In my website, we usually get a spike in the number of requests because of search engine crawlers (like Googlebot) harvesting the website for indexing. So what I thought of; is depending on the ...
1
vote
0answers
313 views

What is user agent ;__CT_JOB_ID__:?

I've been searching the web for an explanation of the sudden influx we've gotten of user agents of this format: ;__CT_JOB_ID__:68a72eee-353c-4b0b-82ae-0cb6c8ca3e6e; Each visitor has a unique IP. So ...
1
vote
0answers
26 views

Product carousel on my site is showing up blank in Google's fetch and render for mobile

Using GSC, I am trying to fetch a page with products and content, this page also has few products in carousel format and when I render this page for desktop or mobile, I am seeing everything fine but ...
1
vote
0answers
13 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
1
vote
0answers
32 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...
1
vote
0answers
48 views

Dealing with Google spidering JavaScript incorrectly

Google spiders JavaScript now to look for possible URLs. This is mostly a good thing, but occasionally it gets it wrong. Is there any way to hide possible URLs in JavaScript from googlebot? For ...
1
vote
0answers
47 views

Google doesn't index my pages managed by session

My website has an architecture like this: em1 sub1 sub2 ... em2 sub1 sub2 ... em3 ... ... All elements "em" has the same "sub" associated. The difference only resides on the content in the "...
1
vote
0answers
51 views

Sitemap with an App using Codeigniter and jQuery Mobile?

We've got dynamic content that we want the google bot to search. Our app is running codeingiter and jQuery mobile. So, our site is basically 6 pages that are updated using XHR calls. The data is ...
1
vote
0answers
156 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
1
vote
0answers
37 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is considered ...
1
vote
0answers
125 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) /Guitars/Children%C3%C6%92%E2%E2%82%AC%C5%A1%...
0
votes
0answers
16 views

Visitors using multiple IPs and determining if they are human

I'm seeing some obvious bot/script traffic that is bloating our Google Analytics(GA) data. I figure the traffic is using something that loads JS, CSS, etc. which is why it's affecting GA. So I put in ...
0
votes
0answers
11 views

Why would a bot net keep loading my site?

In blocking a lot of scrapping activity, I found a puzzling situation - that my site is accessed about 35,000 times per day by around 30,000 different IPs. I started blocking these IPs with a script, ...
0
votes
0answers
10 views

How to make Google show my blog page which is on a different domain?

I have a website (eg. https://work.example.com ) which is being crawled by google. I have a blog for the same under https://blogs.example.com. When I search in google with keywords like "work example",...
0
votes
0answers
110 views

Is there a way to get Google search console crawl stats for larger than 90 days?

Google search console allows you to see how many pages Google is crawling per day on your site, but it only goes back 90 days. You can view your crawl stats here: https://www.google.com/webmasters/...
0
votes
0answers
19 views

Home page has a youtube video on it, unsure how to fix google's description

A website i'm working on has embedded on it's homepage a youtube video. For some reason, google is ignoring my meta description and instead it is showing as a description the following message: "If ...
0
votes
0answers
64 views

I need the subdomain in cpanel to not be followed. Help!

I have little knowledge of robot.txt but I know how meta follow and no follow tags work. The problem is I have 2 totally different websites, one being the primary and the other an addon site with ...
0
votes
0answers
66 views

Web page is not crawled completely by Google

I have a webpage which is built by converting the docx document to html page. The page is added to the google webmaster tool and the sitemap file also been submitted and maintained regularly. There ...
0
votes
0answers
55 views

Googlebot requesting for same page within short period

I was analysing analyzing week over week pages that GoogleBot is requesting for crawling. I can see that GoogleBot is requesting for same page more than 1500 times on a single day. This page is my ...
0
votes
0answers
104 views

How to make sure crawlers and Google use an HTML search form?

I'm trying to make crawlers reference the results of a search form on a website. The form itself is written in AngularJS and the various filters trigger a jsonp call towards a solr server. The form is ...
0
votes
0answers
89 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or directory)...
0
votes
0answers
25 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...