Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

1
vote
2answers
214 views

Excluding Google Analytics spam by referral

I've been reading up on how to filter out spam from Google Analytics. In this case, I'm trying to filter out the spam that shows up under referrals. In the Filter Pattern field of Analytics, I have ...
4
votes
1answer
741 views

Are Googlebot and other search engine crawlers able to detect if element is hidden after page load by JavaScript?

So, we all know Google hates hidden elements, and rightly so. But I am curious, what if the element is present in DOM normally in the static form, but after page load, it is hidden by JavaScript? ...
1
vote
3answers
123 views

Applebot not crawling sitemap.xml

On our website we noticed that the Applebot is not crawling our sitemap.xml, so it is unaware of most of our internal webpages. We have no robots.txt restrictions for that bot, on the contrary we ...
3
votes
4answers
726 views

How to do I stop a bot from from excessively crawling my Blogger site?

I just bought a domain for my Blogger site. I noticed an unusual rise in my page views, and I tried to trace the source of the traffic but it's a dead end... This excessive bot traffic got my AdSense ...
1
vote
1answer
25 views

How can I increased my site traffic [duplicate]

My website releated to entertainment and movies trailer and review based site, my site viewers list actually very low, I have set keywords which is found from google keywords planner. And created ...
1
vote
0answers
47 views

Google doesn't index my pages managed by session

My website has an architecture like this: em1 sub1 sub2 ... em2 sub1 sub2 ... em3 ... ... All elements "em" has the same "sub" associated. The difference only resides on the content in the "...
1
vote
1answer
238 views

Google not re-crawling subpages after “fetch as Google” with the “directly linked pages” option

After fixing a "noindex" issue on my WordPress website, where all my pages failed to be crawled by Google, I used Google Webmaster to re-index the website (without waiting). But it worked only for the ...
0
votes
1answer
278 views

Google Search Console: 404 errors that aren't linked to

I manage an ecommerce website that slightly refreshes its product/category collection every night. Sometimes products/categories get disabled and links to them disappear from the website but despite ...
1
vote
0answers
51 views

Sitemap with an App using Codeigniter and jQuery Mobile?

We've got dynamic content that we want the google bot to search. Our app is running codeingiter and jQuery mobile. So, our site is basically 6 pages that are updated using XHR calls. The data is ...
1
vote
2answers
320 views

Should I submit both mobile and desktop domain names for indexing on non-responsive sites or is that bad for SEO?

I have a desktop version of my site using yourwebsite.com and a mobile version using m.yourwebsite.com. Should I treat them as one site and only submit the desktop version along with its sitemap and ...
5
votes
1answer
148 views

Can Googlebot discover URLs that are not in an anchor?

In my Search Console I had a sudden increase in 404's. They were linked from a webpage that listed files from the site in a table but none of these URLs were nested inside an <a> tag. This was ...
2
votes
0answers
52 views

Browser's auto request detection with X-Purpose

3 to 4 months ago, I found out that safari/chrome/firefox all have feature that automatically send request with X-Purpose header (with preview value for Safari) to get live thumbnail or top sites ...
1
vote
1answer
209 views

Unexplainable non-existing page requested for months

While I am quite familiar with bots requesting well-know pages as used for WordPress and phpMyAdmin on any site the can reach, I'm now dealing with a bot requested a page I really can't explain. For ...
2
votes
0answers
116 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
0
votes
1answer
96 views

Does Web crawler simulate user behavior?

Structuring the web pages' content or schema properly is important for SEO. I would like to know if popular web crawlers or bots (Google bot, Bingbot etc...) are able to simulate user behavior and ...
1
vote
2answers
582 views

At what resolution does Google Bot Web Crawlers consider desktop?

I'm aware that google evaluates how a site looks on mobile. At what resolution does google evaluate desktop sites? My site uses a responsive layout up until 1200px width, at which point it shows and ...
4
votes
3answers
180 views

Googlebot sends several hundred requests in a minute

I have a website which has several million page views per month. Starting several days ago, a bot named Googlebot sends requests to the site very frequently. It sends several hundred requests to the ...
1
vote
2answers
95 views

How to hide product prices from search engines when they vary by country?

We have a site which sells items online only for US customers. The site checks the location based on IP and decides if the price should appear on not. Customers from other locations can buy the ...
2
votes
3answers
1k views

How to make transition from HTTP/1.1 to HTTP/2 with SEO in mind?

I'm aware that HTTP/2 is using only https protocol. My existing implementation on the website are all HTTP/1.1 with http protocol with thousands of existing urls. To add HTTP/2 support, without ...
3
votes
1answer
303 views

Alexa SEO audit reports that all crawlers are blocked despite “Allow: /” for specific crawlers in robots.txt

I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like User-agent: * Disallow: ...
4
votes
3answers
1k views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
0
votes
0answers
14 views

Website is not indexing in google search console? [duplicate]

My website is not indexing in Google webmaster tools (search console). I have submitted my sitemap but page_post_sitemap.xml only has two indexed URLs. I don't know what is problem with my website. ...
1
vote
1answer
80 views

Does Google Search Console track 404s from bots (i.e. spiders, crawlers, etc.)?

I have a verified web property on search console. It's been collecting traffic data for months now. I am using the crawl errors report to look at 404s on my site. I'm curious if this data collects ...
1
vote
1answer
300 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
69 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
0
votes
0answers
6 views

If I make a directory of veterinaries populated by a call to a Database would it get indexed by google? [duplicate]

Imagine I have a website to find say veterinarians. So I want to have a directory listing the NAMES of all the veterinarians that are members of our website. And if I click on any of the names it ...
8
votes
2answers
103 views

Does Google's webcrawler follow HTML imports?

Does googlebot - Google's webcrawler - follow browser-side HTML imports such as the following? <link rel="import" href="header.html"> I'm thinking of hosting web content on my web server that ...
1
vote
1answer
29 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
1
vote
1answer
383 views

Crawling Single Page Apss

Since this post in October 2015, Google claims to be able to crawl JavaScript. Does this mean that single page apps that don't have a server rendering are indexed by Google? I created a single page ...
4
votes
4answers
495 views

How can I make an AngularJS ecommerce website locally crawlable?

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding. So far, I've tried DeepCrawl and Screaming Frog, but I'...
1
vote
1answer
23 views

How to prevent page counts by search indexers

On a page I manage I store information about the most read pages and last read pages in the database. How can I manage to prevent taking into account accesses from search indexers? Note that I still ...
4
votes
3answers
16k views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
0
votes
0answers
16 views

Google isn't crawling all Sitemap URLs [duplicate]

Googlebot isn't indexing our Sitemap http://i.stack.imgur.com/WkB1J.png There are 63 links in the Sitemap only two are indexed I've tried sending some link to the index by "fetch as Google" with no ...
3
votes
1answer
441 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
3
votes
1answer
29 views

Will search engine web crawlers see urls that are changed with HTML5 history?

I have a site where the different language versions are on the same url. If the language is not set as a get variable, language is set by a session variable in php, and the url is rewritten ...
1
vote
1answer
241 views

Difference between crawling and indexing [duplicate]

If a website is cached, does it mean that it has been indexed too? In this snapshot cache date is shown as 18th may but when i did site search for this domain I found 0 pages were indexed by google ...
2
votes
4answers
429 views

How to track clicks on inbound links besides via URL parameters?

We use a custom CMS on a site and we're having various parameters on inbound links for own tracking and inbound marketing purposes. Usually these URL parameters are added to the links so the system ...
1
vote
1answer
326 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
3
votes
1answer
98 views

What do I need to be aware of when running a web crawler?

I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
1
vote
1answer
118 views

Dynamic product filter and SEO

I have a "product selector/filter" that uses an external data source (JSON) for product information. The user can adjust checkboxes or range sliders in the sidebar, and the pure front-end application ...
3
votes
2answers
346 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
1
vote
1answer
34 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: /...
3
votes
1answer
2k views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
1answer
121 views

Calculating requests per second within bing

In Google Search Console, I am able to set the rate at which google crawls my site and it tells me the details such as the requests per second and seconds between requests. In Bing Webmaster tools, I ...
2
votes
2answers
86 views

As a webmaster what is the purpose of using a web crawler for your site?

What benefits would using a web crawler give you as a webmaster?
2
votes
4answers
2k views

Google crawls some pages which do not exist on the website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as vy81TYYP/lraynb-20151123-dkd04-2290.html. It has never existed on a my developed website. This is not ...
3
votes
3answers
207 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
1
vote
1answer
102 views

How does Google crawl database driven web sites that don't have static files?

I have an index.php file on my server. Google will crawl this and retrieve the information. I also have a file named product.php. This dynamically gets content from a database to populate all the ...
7
votes
3answers
1k views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
1
vote
2answers
313 views

What is a tolerable request rate for bots?

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and ...