A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
1answer
14 views

How to hide product prices from search engines when they vary by country?

We have a site which sells items online only for US customers. The site checks the location based on IP and decides if the price should appear on not. Customers from other locations can buy the ...
1
vote
1answer
43 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
29 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
1
vote
3answers
23 views

How to make transition from HTTP/1.1 to HTTP/2 with SEO in mind?

I'm aware that HTTP/2 is using only https protocol. My existing implementation on the website are all HTTP/1.1 with http protocol with thousands of existing urls. To add HTTP/2 support, without ...
0
votes
3answers
57 views

Google crawl some pages, but not exist on website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as "vy81TYYP/lraynb-20151123-dkd04-2290.html" page which has never existed on a my developed website. This ...
3
votes
1answer
61 views

Alexa SEO audit reports that all crawlers are blocked despite “Allow: /” for specific crawlers in robots.txt

I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like User-agent: * Disallow: ...
4
votes
3answers
878 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
4
votes
3answers
1k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
13
votes
3answers
2k views

Prevent malicious bots from posting spam

I remember a site closed due to misuse and i wonder if bots have a part of it. If the bot is POSTing something to my site what are ways I can combat it? I was thinking of setting some cookies and ...
1
vote
1answer
35 views

How does Google crawl database driven web sites that don't have static files?

I have an index.php file on my server. Google will crawl this and retrieve the information. I also have a file named product.php. This dynamically gets content from a database to populate all the ...
0
votes
0answers
8 views

How to keep unfriendly bots from accessing site? [duplicate]

I have a number of websites that I created where I have noticed that the errordoc file shows that some "bad" bots have crawled the site. This causes Webroot to flag the site as suspicious even though ...
0
votes
1answer
33 views

Generate static SEO pages for AngularJS + Spring MVC website [closed]

I have a project using Spring MVC + AngularJS. All the data is dynamic. Have some big database of locations in this app. For SEO purposes, need to generate a static page for each location and put ...
2
votes
2answers
46 views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
5
votes
2answers
165 views

What determines the frequency the Wayback Machine crawls one's website?

I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
0
votes
3answers
207 views

Opening forum for search engine bots and SEO

We develop and sell components for other software developers through our website. The main products are ActiveX and WinFroms.NET grid controls. Some years ago we established a forum dedicated to our ...
0
votes
0answers
9 views

Website is not indexing in google search console? [duplicate]

My website is not indexing in Google webmaster tools (search console). I have submitted my sitemap but page_post_sitemap.xml only has two indexed URLs. I don't know what is problem with my website. ...
1
vote
1answer
54 views

Does Google Search Console track 404s from bots (i.e. spiders, crawlers, etc.)?

I have a verified web property on search console. It's been collecting traffic data for months now. I am using the crawl errors report to look at 404s on my site. I'm curious if this data collects ...
3
votes
2answers
2k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
4
votes
2answers
56 views

What's the best way to crawl an AngularJS ecommerce website?

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding. So far, I've tried DeepCrawl and Screaming Frog, but I'...
1
vote
1answer
18 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's dashboard....
0
votes
0answers
5 views

If I make a directory of veterinaries populated by a call to a Database would it get indexed by google? [duplicate]

Imagine I have a website to find say veterinarians. So I want to have a directory listing the NAMES of all the veterinarians that are members of our website. And if I click on any of the names it ...
1
vote
1answer
55 views

Crawling Single Page Apss

Since this post in October 2015, Google claims to be able to crawl JavaScript. Does this mean that single page apps that don't have a server rendering are indexed by Google? I created a single page ...
3
votes
1answer
17 views

Does Google's webcrawler follow HTML imports?

Does googlebot - Google's webcrawler - follow browser-side HTML imports such as the following? <link rel="import" href="header.html"> I'm thinking of hosting web content on my web server that ...
1
vote
1answer
17 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
1
vote
1answer
20 views

How to prevent page counts by search indexers

On a page I manage I store information about the most read pages and last read pages in the database. How can I manage to prevent taking into account accesses from search indexers? Note that I still ...
1
vote
2answers
36 views

How to track clicks on inbound links besides via URL parameters?

We use a custom CMS on a site and we're having various parameters on inbound links for own tracking and inbound marketing purposes. Usually these URL parameters are added to the links so the system ...
0
votes
0answers
15 views

Google isn't crawling all Sitemap URLs [duplicate]

Googlebot isn't indexing our Sitemap http://i.stack.imgur.com/WkB1J.png There are 63 links in the Sitemap only two are indexed I've tried sending some link to the index by "fetch as Google" with no ...
3
votes
1answer
369 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
0
votes
1answer
44 views

Difference between crawling and indexing [duplicate]

If a website is cached, does it mean that it has been indexed too? In this snapshot cache date is shown as 18th may but when i did site search for this domain I found 0 pages were indexed by google ...
3
votes
1answer
24 views

Will search engine web crawlers see urls that are changed with HTML5 history?

I have a site where the different language versions are on the same url. If the language is not set as a get variable, language is set by a session variable in php, and the url is rewritten ...
2
votes
1answer
151 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
0
votes
0answers
14 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
3
votes
1answer
38 views

What do I need to be aware of when running a web crawler?

I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
0
votes
0answers
4 views

Is a URL without the trailing slash considered the same as the URL with trailing slash for SEO? [duplicate]

I just have a simple doubt about URLs and as I didn't find useful infos on the net, I am asking here: If I have a URL, let's say http://stackoverflow.com/questions and http://stackoverflow.com/...
1
vote
1answer
33 views

Dynamic product filter and SEO

I have a "product selector/filter" that uses an external data source (JSON) for product information. The user can adjust checkboxes or range sliders in the sidebar, and the pure front-end application ...
2
votes
3answers
126 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
1
vote
1answer
87 views

How to keep friendly URLs consistent?

I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed. The original site ran on a combination of Drupal and a lesser known, ...
3
votes
2answers
46 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
2
votes
2answers
40 views
1
vote
2answers
38 views

What is a tolerable request rate for bots?

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and ...
8
votes
6answers
8k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
2
votes
1answer
59 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
1answer
25 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: /...
0
votes
1answer
57 views

We want to change the URLs of an entire section of the website, what all can go wrong with SEO?

We have the following URL sections: http://www.movie-website.com/movies/movie-name/trailers And we want to change it to: http://www.movie-website.com/trailers/movie-name/ Of course we are looking ...
1
vote
1answer
15 views

What is a good approximation for a page's (Google) search rank to come back to it's original rank, after it has fallen due to technical errors?

Say we had a web page, which initially had a google rank on the first page. Say these errors occurred after some time: load time Meta title changes Some data not being visible to the crawler etc. ...
1
vote
1answer
38 views

Calculating requests per second within bing

In Google Search Console, I am able to set the rate at which google crawls my site and it tells me the details such as the requests per second and seconds between requests. In Bing Webmaster tools, I ...
4
votes
5answers
1k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
3answers
85 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
7
votes
3answers
120 views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
2
votes
2answers
34 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...