A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
0answers
7 views

different host path for web and android for app indexing

As per google's recommendation we use following meta tag for android application indexing: android-app://<package_name>/<scheme>/<host_path> Example : <link rel="alternate" ...
0
votes
1answer
19 views

Does Web crawler simulate user behavior?

Structuring the web pages' content or schema properly is important for SEO. I would like to know if popular web crawlers or bots (Google bot, Bingbot etc...) are able to simulate user behavior and ...
0
votes
0answers
9 views

How to get rid of non-reputable Bots [closed]

Which can we consider the non-reputable web searching bots and which are their names? I would like to disallow them via robots.txt. Peace! Juan de Marcos
3
votes
1answer
105 views

What is the .well-known/ directory and how should I set it up?

In my webmaster's tools crawl errors I have a few access denied locations: .well-known/apple-app-site-association .well-known/assetlinks.json .well-known/.com I had a look at RFC 5785 but I didn't ...
1
vote
2answers
30 views

At what resolution does Google Bot Web Crawlers consider desktop?

I'm aware that google evaluates how a site looks on mobile. At what resolution does google evaluate desktop sites? My site uses a responsive layout up until 1200px width, at which point it shows and ...
4
votes
3answers
78 views

Googlebot sends several hundred requests in a minute

I have a website which has several million page views per month. Starting several days ago, a bot named Googlebot sends requests to the site very frequently. It sends several hundred requests to the ...
1
vote
2answers
29 views

How to hide product prices from search engines when they vary by country?

We have a site which sells items online only for US customers. The site checks the location based on IP and decides if the price should appear on not. Customers from other locations can buy the ...
1
vote
3answers
31 views

How to make transition from HTTP/1.1 to HTTP/2 with SEO in mind?

I'm aware that HTTP/2 is using only https protocol. My existing implementation on the website are all HTTP/1.1 with http protocol with thousands of existing urls. To add HTTP/2 support, without ...
3
votes
1answer
64 views

Alexa SEO audit reports that all crawlers are blocked despite “Allow: /” for specific crawlers in robots.txt

I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like User-agent: * Disallow: ...
0
votes
0answers
8 views

How to keep unfriendly bots from accessing site? [duplicate]

I have a number of websites that I created where I have noticed that the errordoc file shows that some "bad" bots have crawled the site. This causes Webroot to flag the site as suspicious even though ...
4
votes
3answers
924 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
0
votes
0answers
9 views

Website is not indexing in google search console? [duplicate]

My website is not indexing in Google webmaster tools (search console). I have submitted my sitemap but page_post_sitemap.xml only has two indexed URLs. I don't know what is problem with my website. ...
1
vote
1answer
57 views

Does Google Search Console track 404s from bots (i.e. spiders, crawlers, etc.)?

I have a verified web property on search console. It's been collecting traffic data for months now. I am using the crawl errors report to look at 404s on my site. I'm curious if this data collects ...
1
vote
1answer
44 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
30 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
0
votes
0answers
5 views

If I make a directory of veterinaries populated by a call to a Database would it get indexed by google? [duplicate]

Imagine I have a website to find say veterinarians. So I want to have a directory listing the NAMES of all the veterinarians that are members of our website. And if I click on any of the names it ...
3
votes
1answer
17 views

Does Google's webcrawler follow HTML imports?

Does googlebot - Google's webcrawler - follow browser-side HTML imports such as the following? <link rel="import" href="header.html"> I'm thinking of hosting web content on my web server that ...
1
vote
1answer
18 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
1
vote
1answer
60 views

Crawling Single Page Apss

Since this post in October 2015, Google claims to be able to crawl JavaScript. Does this mean that single page apps that don't have a server rendering are indexed by Google? I created a single page ...
4
votes
2answers
70 views

What's the best way to crawl an AngularJS ecommerce website?

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding. So far, I've tried DeepCrawl and Screaming Frog, but I'...
1
vote
1answer
20 views

How to prevent page counts by search indexers

On a page I manage I store information about the most read pages and last read pages in the database. How can I manage to prevent taking into account accesses from search indexers? Note that I still ...
2
votes
2answers
67 views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
0
votes
0answers
15 views

Google isn't crawling all Sitemap URLs [duplicate]

Googlebot isn't indexing our Sitemap http://i.stack.imgur.com/WkB1J.png There are 63 links in the Sitemap only two are indexed I've tried sending some link to the index by "fetch as Google" with no ...
3
votes
1answer
370 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
3
votes
1answer
25 views

Will search engine web crawlers see urls that are changed with HTML5 history?

I have a site where the different language versions are on the same url. If the language is not set as a get variable, language is set by a session variable in php, and the url is rewritten ...
0
votes
1answer
48 views

Difference between crawling and indexing [duplicate]

If a website is cached, does it mean that it has been indexed too? In this snapshot cache date is shown as 18th may but when i did site search for this domain I found 0 pages were indexed by google ...
1
vote
2answers
50 views

How to track clicks on inbound links besides via URL parameters?

We use a custom CMS on a site and we're having various parameters on inbound links for own tracking and inbound marketing purposes. Usually these URL parameters are added to the links so the system ...
1
vote
1answer
26 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
0
votes
0answers
4 views

Is a URL without the trailing slash considered the same as the URL with trailing slash for SEO? [duplicate]

I just have a simple doubt about URLs and as I didn't find useful infos on the net, I am asking here: If I have a URL, let's say http://stackoverflow.com/questions and http://stackoverflow.com/...
3
votes
1answer
40 views

What do I need to be aware of when running a web crawler?

I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
1
vote
1answer
33 views

Dynamic product filter and SEO

I have a "product selector/filter" that uses an external data source (JSON) for product information. The user can adjust checkboxes or range sliders in the sidebar, and the pure front-end application ...
3
votes
2answers
47 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
1
vote
1answer
25 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: /...
2
votes
1answer
61 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
1answer
39 views

Calculating requests per second within bing

In Google Search Console, I am able to set the rate at which google crawls my site and it tells me the details such as the requests per second and seconds between requests. In Bing Webmaster tools, I ...
2
votes
2answers
40 views

As a webmaster what is the purpose of using a web crawler for your site?

What benefits would using a web crawler give you as a webmaster?
1
vote
4answers
79 views

Google crawls some pages which do not exist on the website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as vy81TYYP/lraynb-20151123-dkd04-2290.html. It has never existed on a my developed website. This is not ...
3
votes
3answers
90 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
1
vote
1answer
36 views

How does Google crawl database driven web sites that don't have static files?

I have an index.php file on my server. Google will crawl this and retrieve the information. I also have a file named product.php. This dynamically gets content from a database to populate all the ...
7
votes
3answers
123 views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
1
vote
2answers
38 views

What is a tolerable request rate for bots?

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and ...
2
votes
2answers
35 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
3
votes
1answer
47 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
1
vote
0answers
55 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
2
votes
3answers
69 views

Is it possible to ruin your SEO by cross-linking, overly structuring a website?

We're trying to categorise our website data more and more. This will lead us to have many many categories for differect setions of information. Can doing too much of this lead to bad SEO? My ...
0
votes
1answer
60 views

We want to change the URLs of an entire section of the website, what all can go wrong with SEO?

We have the following URL sections: http://www.movie-website.com/movies/movie-name/trailers And we want to change it to: http://www.movie-website.com/trailers/movie-name/ Of course we are looking ...
2
votes
5answers
91 views

Prevent robots from indexing specific word

I need to prevent robots from indexing specific word on my webpage. Here is what I did: Let's say I need to prevent JOHN GALLOW from being indexed. For that purpose, I wrote this pararaph: Over the ...
3
votes
2answers
74 views

Increase in 5XX errors and 404 errors on GWT, are we under attack?

A lot of URLs are suddenly giving us 404s and 500s. On closer inspection, these URLs do not exist on our website and are being linked from random sources. As more URLs were inspected on GWT, a ...
3
votes
1answer
35 views

Google blocks map API, GWT shows severity high, will this cause SEO problems?

Google blocked map API, GWT showed severity high. What is the effect on SEO?
3
votes
1answer
103 views

If I rename a sitemap, does it affect my SEO?

I would like to rename my sitemap: Would it affect my SEO at all? Would I have to resubmit it on GWT? If I have a sitemap-index.xml file, that remains the same, but I change the name of one of the ...