A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

2
votes
2answers
9 views

Googlebot sends several hundred requests in a minute

I have a website which has several million page views per month. Starting several days ago, a bot named Googlebot sends requests to the site very frequently. It sends several hundred requests to the ...
1
vote
1answer
15 views

How to hide product prices from search engines when they vary by country?

We have a site which sells items online only for US customers. The site checks the location based on IP and decides if the price should appear on not. Customers from other locations can buy the ...
1
vote
3answers
24 views

How to make transition from HTTP/1.1 to HTTP/2 with SEO in mind?

I'm aware that HTTP/2 is using only https protocol. My existing implementation on the website are all HTTP/1.1 with http protocol with thousands of existing urls. To add HTTP/2 support, without ...
3
votes
1answer
61 views

Alexa SEO audit reports that all crawlers are blocked despite “Allow: /” for specific crawlers in robots.txt

I had got done a SEO Audit by Alexa for my website. It reported that all the crawlers are blocked from my website because of my robots.txt. My robots.txt looks something like User-agent: * Disallow: ...
0
votes
0answers
8 views

How to keep unfriendly bots from accessing site? [duplicate]

I have a number of websites that I created where I have noticed that the errordoc file shows that some "bad" bots have crawled the site. This causes Webroot to flag the site as suspicious even though ...
0
votes
1answer
34 views

Generate static SEO pages for AngularJS + Spring MVC website [closed]

I have a project using Spring MVC + AngularJS. All the data is dynamic. Have some big database of locations in this app. For SEO purposes, need to generate a static page for each location and put ...
4
votes
3answers
878 views

How does Googlebot find a new page on a website?

I was wondering how Google identifies a new article or piece of content on a website. What is the method it uses to do so for all websites?
0
votes
0answers
9 views

Website is not indexing in google search console? [duplicate]

My website is not indexing in Google webmaster tools (search console). I have submitted my sitemap but page_post_sitemap.xml only has two indexed URLs. I don't know what is problem with my website. ...
1
vote
1answer
55 views

Does Google Search Console track 404s from bots (i.e. spiders, crawlers, etc.)?

I have a verified web property on search console. It's been collecting traffic data for months now. I am using the crawl errors report to look at 404s on my site. I'm curious if this data collects ...
1
vote
1answer
43 views

Google is unable to crawl both my UK and USA sites that use IP detection redirects

The URL's of my website open as per IP detection. That is, for UK it is example.com and for USA it is example.com/usa. But Google is unable to crawl example.com/usa. It is crawling just example.com. ...
1
vote
1answer
29 views

Making google forgive us right away when accidentally publishing incorrect links (noarchive doesn't work)

So this is what happened. One day, I changed my code on my site to try to make things more compliant with adsense. Several hours later, I check the crawl errors report GSC only to see a string of ...
0
votes
0answers
5 views

If I make a directory of veterinaries populated by a call to a Database would it get indexed by google? [duplicate]

Imagine I have a website to find say veterinarians. So I want to have a directory listing the NAMES of all the veterinarians that are members of our website. And if I click on any of the names it ...
3
votes
1answer
17 views

Does Google's webcrawler follow HTML imports?

Does googlebot - Google's webcrawler - follow browser-side HTML imports such as the following? <link rel="import" href="header.html"> I'm thinking of hosting web content on my web server that ...
1
vote
1answer
17 views

Google - aliasing two pages with URLs that are generated client-side

I'm working on a website that dynamically replaces URLs during state changes so that if a user copy/pastes the URL at any give time, they will be able to land on a page that reflects the state when ...
1
vote
1answer
55 views

Crawling Single Page Apss

Since this post in October 2015, Google claims to be able to crawl JavaScript. Does this mean that single page apps that don't have a server rendering are indexed by Google? I created a single page ...
4
votes
2answers
56 views

What's the best way to crawl an AngularJS ecommerce website?

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding. So far, I've tried DeepCrawl and Screaming Frog, but I'...
1
vote
1answer
20 views

How to prevent page counts by search indexers

On a page I manage I store information about the most read pages and last read pages in the database. How can I manage to prevent taking into account accesses from search indexers? Note that I still ...
2
votes
2answers
46 views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
0
votes
0answers
15 views

Google isn't crawling all Sitemap URLs [duplicate]

Googlebot isn't indexing our Sitemap http://i.stack.imgur.com/WkB1J.png There are 63 links in the Sitemap only two are indexed I've tried sending some link to the index by "fetch as Google" with no ...
3
votes
1answer
369 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
3
votes
1answer
24 views

Will search engine web crawlers see urls that are changed with HTML5 history?

I have a site where the different language versions are on the same url. If the language is not set as a get variable, language is set by a session variable in php, and the url is rewritten ...
0
votes
1answer
44 views

Difference between crawling and indexing [duplicate]

If a website is cached, does it mean that it has been indexed too? In this snapshot cache date is shown as 18th may but when i did site search for this domain I found 0 pages were indexed by google ...
1
vote
2answers
36 views

How to track clicks on inbound links besides via URL parameters?

We use a custom CMS on a site and we're having various parameters on inbound links for own tracking and inbound marketing purposes. Usually these URL parameters are added to the links so the system ...
0
votes
0answers
14 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
0
votes
0answers
4 views

Is a URL without the trailing slash considered the same as the URL with trailing slash for SEO? [duplicate]

I just have a simple doubt about URLs and as I didn't find useful infos on the net, I am asking here: If I have a URL, let's say http://stackoverflow.com/questions and http://stackoverflow.com/...
3
votes
1answer
38 views

What do I need to be aware of when running a web crawler?

I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
1
vote
1answer
33 views

Dynamic product filter and SEO

I have a "product selector/filter" that uses an external data source (JSON) for product information. The user can adjust checkboxes or range sliders in the sidebar, and the pure front-end application ...
3
votes
2answers
46 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
1
vote
1answer
25 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: /...
2
votes
1answer
59 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
1answer
38 views

Calculating requests per second within bing

In Google Search Console, I am able to set the rate at which google crawls my site and it tells me the details such as the requests per second and seconds between requests. In Bing Webmaster tools, I ...
2
votes
2answers
40 views

As a webmaster what is the purpose of using a web crawler for your site?

What benefits would using a web crawler give you as a webmaster?
0
votes
3answers
57 views

Google crawl some pages, but not exist on website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as "vy81TYYP/lraynb-20151123-dkd04-2290.html" page which has never existed on a my developed website. This ...
3
votes
3answers
85 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
1
vote
1answer
35 views

How does Google crawl database driven web sites that don't have static files?

I have an index.php file on my server. Google will crawl this and retrieve the information. I also have a file named product.php. This dynamically gets content from a database to populate all the ...
7
votes
3answers
120 views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
1
vote
2answers
38 views

What is a tolerable request rate for bots?

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and ...
2
votes
2answers
34 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
3
votes
1answer
45 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
1
vote
0answers
53 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
2
votes
3answers
68 views

Is it possible to ruin your SEO by cross-linking, overly structuring a website?

We're trying to categorise our website data more and more. This will lead us to have many many categories for differect setions of information. Can doing too much of this lead to bad SEO? My ...
0
votes
1answer
57 views

We want to change the URLs of an entire section of the website, what all can go wrong with SEO?

We have the following URL sections: http://www.movie-website.com/movies/movie-name/trailers And we want to change it to: http://www.movie-website.com/trailers/movie-name/ Of course we are looking ...
2
votes
5answers
91 views

Prevent robots from indexing specific word

I need to prevent robots from indexing specific word on my webpage. Here is what I did: Let's say I need to prevent JOHN GALLOW from being indexed. For that purpose, I wrote this pararaph: Over the ...
3
votes
2answers
73 views

Increase in 5XX errors and 404 errors on GWT, are we under attack?

A lot of URLs are suddenly giving us 404s and 500s. On closer inspection, these URLs do not exist on our website and are being linked from random sources. As more URLs were inspected on GWT, a ...
3
votes
1answer
33 views

Google blocks map API, GWT shows severity high, will this cause SEO problems?

Google blocked map API, GWT showed severity high. What is the effect on SEO?
3
votes
1answer
92 views

If I rename a sitemap, does it affect my SEO?

I would like to rename my sitemap: Would it affect my SEO at all? Would I have to resubmit it on GWT? If I have a sitemap-index.xml file, that remains the same, but I change the name of one of the ...
2
votes
1answer
61 views

Does Google prefer HTML sitemaps or XML sitemaps? And does it hurt to implement both?

I just read that Google prefers HTML sitemaps. We have so far, only implemented an XML sitemap. Would it hurt to add an HTML sitemap, and not remove the XML sitemap? Also, can you submit the HTML ...
0
votes
1answer
23 views

What affect, if any, would buying domain names with high volume search terms have on SEO?

Assume the following: I have a website: www.example.com. I want to rank www.example.com/searc-term-1 on the first page of Google. Would it help the SEO of www.example.com/searc-term-1 if I bought ...
1
vote
2answers
50 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
1answer
15 views

What is a good approximation for a page's (Google) search rank to come back to it's original rank, after it has fallen due to technical errors?

Say we had a web page, which initially had a google rank on the first page. Say these errors occurred after some time: load time Meta title changes Some data not being visible to the crawler etc. ...