A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
1answer
20 views

Does Web crawler simulate user behavior?

Structuring the web pages' content or schema properly is important for SEO. I would like to know if popular web crawlers or bots (Google bot, Bingbot etc...) are able to simulate user behavior and ...
1
vote
0answers
7 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
1
vote
4answers
81 views

Google crawls some pages which do not exist on the website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as vy81TYYP/lraynb-20151123-dkd04-2290.html. It has never existed on a my developed website. This is not ...
0
votes
0answers
9 views

How to get rid of non-reputable Bots [closed]

Which can we consider the non-reputable web searching bots and which are their names? I would like to disallow them via robots.txt. Peace! Juan de Marcos
3
votes
1answer
106 views

What is the .well-known/ directory and how should I set it up?

In my webmaster's tools crawl errors I have a few access denied locations: .well-known/apple-app-site-association .well-known/assetlinks.json .well-known/.com I had a look at RFC 5785 but I didn't ...
2
votes
2answers
89 views

Mysterious visits to privileged URLs from anonymous user/bot

I maintain a Rails app. Every page except login and "forgot password" requires user authentication. Yesterday I noticed entries like this one in the access logs: 54.209.60.63 - - [03/Nov/2015:19:09:...
1
vote
1answer
19 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's dashboard....
1
vote
1answer
35 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
0
votes
1answer
132 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. http://www.example.com/segment1/segment2&sa=U&ved=...
0
votes
1answer
588 views

Is there a way to make Alexa's ia_archiver slow down its crawling of my website?

Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
2
votes
2answers
775 views

Does Bing execute Javascript when crawling and indexing web pages (like Google)?

Some content on some web pages is only displayed with Javascript when a button is clicked (for example). We know that Google executes Javascript and is (very likely) to find this content, but what ...
1
vote
2answers
31 views

At what resolution does Google Bot Web Crawlers consider desktop?

I'm aware that google evaluates how a site looks on mobile. At what resolution does google evaluate desktop sites? My site uses a responsive layout up until 1200px width, at which point it shows and ...
4
votes
3answers
78 views

Googlebot sends several hundred requests in a minute

I have a website which has several million page views per month. Starting several days ago, a bot named Googlebot sends requests to the site very frequently. It sends several hundred requests to the ...
1
vote
1answer
26 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
1
vote
2answers
30 views

How to hide product prices from search engines when they vary by country?

We have a site which sells items online only for US customers. The site checks the location based on IP and decides if the price should appear on not. Customers from other locations can buy the ...
1
vote
1answer
71 views

Does Google bot always treat dot as word separator?

One of our products is a grid control called iGrid, and we have two versions for two different platforms. One is iGrid ActiveX, the other is iGrid for .NET Framework WinForms called "iGrid.NET". At ...
1
vote
1answer
44 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
30 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
1
vote
3answers
31 views

How to make transition from HTTP/1.1 to HTTP/2 with SEO in mind?

I'm aware that HTTP/2 is using only https protocol. My existing implementation on the website are all HTTP/1.1 with http protocol with thousands of existing urls. To add HTTP/2 support, without ...
3
votes
1answer
64 views

Alexa SEO audit reports that all crawlers are blocked despite “Allow: /” for specific crawlers in robots.txt

I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like User-agent: * Disallow: ...
4
votes
3answers
924 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
4
votes
3answers
1k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
13
votes
3answers
2k views

Prevent malicious bots from posting spam

I remember a site closed due to misuse and i wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
1
vote
1answer
36 views

How does Google crawl database driven web sites that don't have static files?

I have an index.php file on my server. Google will crawl this and retrieve the information. I also have a file named product.php. This dynamically gets content from a database to populate all the ...
0
votes
0answers
8 views

How to keep unfriendly bots from accessing site? [duplicate]

I have a number of websites that I created where I have noticed that the errordoc file shows that some "bad" bots have crawled the site. This causes Webroot to flag the site as suspicious even though ...
2
votes
2answers
67 views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
5
votes
2answers
175 views

What determines the frequency the Wayback Machine crawls one's website?

I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
0
votes
3answers
210 views

Opening forum for search engine bots and SEO

We develop and sell components for other software developers through our website. The main products are ActiveX and WinFroms.NET grid controls. Some years ago we established a forum dedicated to our ...
0
votes
0answers
9 views

Website is not indexing in google search console? [duplicate]

My website is not indexing in Google webmaster tools (search console). I have submitted my sitemap but page_post_sitemap.xml only has two indexed URLs. I don't know what is problem with my website. ...
1
vote
1answer
57 views

Does Google Search Console track 404s from bots (i.e. spiders, crawlers, etc.)?

I have a verified web property on search console. It's been collecting traffic data for months now. I am using the crawl errors report to look at 404s on my site. I'm curious if this data collects ...
3
votes
2answers
2k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
4
votes
2answers
70 views

What's the best way to crawl an AngularJS ecommerce website?

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding. So far, I've tried DeepCrawl and Screaming Frog, but I'...
0
votes
0answers
5 views

If I make a directory of veterinaries populated by a call to a Database would it get indexed by google? [duplicate]

Imagine I have a website to find say veterinarians. So I want to have a directory listing the NAMES of all the veterinarians that are members of our website. And if I click on any of the names it ...
1
vote
1answer
60 views

Crawling Single Page Apss

Since this post in October 2015, Google claims to be able to crawl JavaScript. Does this mean that single page apps that don't have a server rendering are indexed by Google? I created a single page ...
3
votes
1answer
17 views

Does Google's webcrawler follow HTML imports?

Does googlebot - Google's webcrawler - follow browser-side HTML imports such as the following? <link rel="import" href="header.html"> I'm thinking of hosting web content on my web server that ...
1
vote
1answer
18 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
1
vote
1answer
20 views

How to prevent page counts by search indexers

On a page I manage I store information about the most read pages and last read pages in the database. How can I manage to prevent taking into account accesses from search indexers? Note that I still ...
1
vote
2answers
50 views

How to track clicks on inbound links besides via URL parameters?

We use a custom CMS on a site and we're having various parameters on inbound links for own tracking and inbound marketing purposes. Usually these URL parameters are added to the links so the system ...
0
votes
0answers
15 views

Google isn't crawling all Sitemap URLs [duplicate]

Googlebot isn't indexing our Sitemap http://i.stack.imgur.com/WkB1J.png There are 63 links in the Sitemap only two are indexed I've tried sending some link to the index by "fetch as Google" with no ...
3
votes
1answer
370 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
0
votes
1answer
48 views

Difference between crawling and indexing [duplicate]

If a website is cached, does it mean that it has been indexed too? In this snapshot cache date is shown as 18th may but when i did site search for this domain I found 0 pages were indexed by google ...
3
votes
1answer
25 views

Will search engine web crawlers see urls that are changed with HTML5 history?

I have a site where the different language versions are on the same url. If the language is not set as a get variable, language is set by a session variable in php, and the url is rewritten ...
2
votes
1answer
165 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
3
votes
1answer
40 views

What do I need to be aware of when running a web crawler?

I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
0
votes
0answers
4 views

Is a URL without the trailing slash considered the same as the URL with trailing slash for SEO? [duplicate]

I just have a simple doubt about URLs and as I didn't find useful infos on the net, I am asking here: If I have a URL, let's say http://stackoverflow.com/questions and http://stackoverflow.com/...
1
vote
1answer
33 views

Dynamic product filter and SEO

I have a "product selector/filter" that uses an external data source (JSON) for product information. The user can adjust checkboxes or range sliders in the sidebar, and the pure front-end application ...
2
votes
3answers
134 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
1
vote
1answer
90 views

How to keep friendly URLs consistent?

I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed. The original site ran on a combination of Drupal and a lesser known, ...
3
votes
2answers
47 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
2
votes
2answers
40 views