A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

1
vote
0answers
14 views

Dynamic product filter and SEO

I have a "product selector/filter" that uses an external data source (JSON) for product information. The user can adjust checkboxes or range sliders in the sidebar, and the pure front-end application ...
3
votes
2answers
41 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
1
vote
1answer
21 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: ...
2
votes
1answer
44 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
1answer
28 views

Calculating requests per second within bing

In Google Search Console, I am able to set the rate at which google crawls my site and it tells me the details such as the requests per second and seconds between requests. In Bing Webmaster tools, I ...
2
votes
2answers
38 views

As a webmaster what is the purpose of using a web crawler for your site?

What benefits would using a web crawler give you as a webmaster?
0
votes
3answers
38 views

Google crawl some pages, but not exist on website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as "vy81TYYP/lraynb-20151123-dkd04-2290.html" page which has never existed on a my developed website. This ...
3
votes
3answers
75 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
0
votes
1answer
23 views

How google crawls webpages without actual page files

For example, I have an index.php file on my server, google will crawl this and retrieve the information. I also have a file named product.php, this dynamically gets content from a database to ...
7
votes
3answers
109 views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
1
vote
2answers
33 views

What is a tolerable request rate for bots?

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and ...
2
votes
2answers
32 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
3
votes
1answer
43 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
1
vote
0answers
51 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
2
votes
3answers
66 views

Is it possible to ruin your SEO by cross-linking, overly structuring a website?

We're trying to categorise our website data more and more. This will lead us to have many many categories for differect setions of information. Can doing too much of this lead to bad SEO? My ...
0
votes
1answer
51 views

We want to change the URLs of an entire section of the website, what all can go wrong with SEO?

We have the following URL sections: http://www.movie-website.com/movies/movie-name/trailers And we want to change it to: http://www.movie-website.com/trailers/movie-name/ Of course we are looking ...
2
votes
5answers
88 views

Prevent robots from indexing specific word

I need to prevent robots from indexing specific word on my webpage. Here is what I did: Let's say I need to prevent JOHN GALLOW from being indexed. For that purpose, I wrote this pararaph: Over the ...
3
votes
2answers
67 views

Increase in 5XX errors and 404 errors on GWT, are we under attack?

A lot of URLs are suddenly giving us 404s and 500s. On closer inspection, these URLs do not exist on our website and are being linked from random sources. As more URLs were inspected on GWT, a ...
3
votes
1answer
28 views

Google blocks map API, GWT shows severity high, will this cause SEO problems?

Google blocked map API, GWT showed severity high. What is the effect on SEO?
3
votes
1answer
65 views

If I rename a sitemap, does it affect my SEO?

I would like to rename my sitemap: Would it affect my SEO at all? Would I have to resubmit it on GWT? If I have a sitemap-index.xml file, that remains the same, but I change the name of one of the ...
2
votes
1answer
47 views

Does Google prefer HTML sitemaps or XML sitemaps? And does it hurt to implement both?

I just read that Google prefers HTML sitemaps. We have so far, only implemented an XML sitemap. Would it hurt to add an HTML sitemap, and not remove the XML sitemap? Also, can you submit the HTML ...
0
votes
1answer
23 views

What affect, if any, would buying domain names with high volume search terms have on SEO?

Assume the following: I have a website: www.example.com. I want to rank www.example.com/searc-term-1 on the first page of Google. Would it help the SEO of www.example.com/searc-term-1 if I bought ...
1
vote
2answers
41 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
1
vote
1answer
14 views

What is a good approximation for a page's (Google) search rank to come back to it's original rank, after it has fallen due to technical errors?

Say we had a web page, which initially had a google rank on the first page. Say these errors occurred after some time: load time Meta title changes Some data not being visible to the crawler etc. ...
1
vote
2answers
97 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
0
votes
1answer
24 views

How can I find out if my website is being scraped? And how do I stop it? [duplicate]

Is there a way to identify that someone is scraping my website?
0
votes
2answers
37 views

Is finding and using expired domains for backlinks considered Black-Hat?

http://www.matthewwoodward.co.uk/tutorials/build-expired-web-2-0-networks-increase-rankings/ This guide shows how to identify and generate backlinks for a website. If one was to follow this guide: ...
3
votes
1answer
33 views

How to exclude Bots from Google Analytics (sudden rise of them in the last weeks)

Since a few weeks, one of my Google Analytics Accounts shows a continued rise of Visitors and Pageimpressions what seams to be bots. They have a Average Session Duration of «<00:00:01» and by now ...
1
vote
2answers
43 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?
0
votes
2answers
62 views

SEO effects of H1 carousel slider

I maintain a website for a law firm. Originally their main desired keyword was "zoning attorneys." So I created a small banner that has an h1 "ZONING ATTORNEYS" on every page with the attorneys' names ...
3
votes
1answer
30 views

cookies and useragent is not enough ? to emulate browser?

I am building kinda a search engine for online shops, and several website are blocking me although I check the robots.txt Let's take an example, I need to get this page: ...
2
votes
1answer
41 views

Does site structure/URL structure impact SEO? [duplicate]

We would like to change the following URLs as follows: (old versions to the left, being changed to new versions on the right) www.ex.com/xxx/full-seo-term -> www.ex.com/half-seo-term1/half-seo-term2 ...
3
votes
2answers
40 views

4 URLs lead to the same page content, are 301 redirects better or setting rel canonical better?

All the following pages have the same exact content: http://www.example.com/seo-url (rel canonical URL) http://www.example.com/ID1234 http://www.example.com/content/ID1234 ...
2
votes
1answer
101 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
1answer
56 views

Is it OK for SEO and user experience to 301 redirect self-generated param URLs?

I am using a CMS which offers a wide range of options which lead to many additional special pages mainly in a way duplicating the same content as the original article. These generated pages are ...
3
votes
1answer
67 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. ...
4
votes
3answers
45 views

Preventing a link on a page from being indexed and followed

I have read the post about how nofollow value impacts on crawlers/indexing. However the information from the post answers end on 2012. In my case, a have a file page that is indexed by Google. On ...
1
vote
0answers
114 views

Fetch as Google not rendering single-page application (AngularJS) - how to show console logs output

I have an SPA application, and it is not being rendered correctly by Google. I tried going to Google Web Master toolkit, and use the Fetch as google feature. This is showing just the HTML output ...
1
vote
1answer
27 views

Bots and image caching

Once again, I'm trying to optimize my site because one test showed the time to first byte is slightly over 200ms, but I need it under 200ms to make google happy. I was looking at code to my site and ...
8
votes
2answers
806 views

Robots.txt should be in the root-directory or can be in sub-directory?

I have a sub-directory that I would like not to be visible to the search engine Web crawlers. One way to do that is to use a robots.txt in the root directory of the server but is something that I ...
1
vote
1answer
15 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's ...
0
votes
0answers
10 views

Spiders cannot crawl all my website pages [duplicate]

I'm struggling with googlebot webpages indexing; in particular I have a page with many links to other subpages that aren't indexed. The strange thing is that, if you submit in any online crawler the ...
0
votes
2answers
92 views

Stop Google from crawling proxied pages but still allow the proxy itself to be found via search engines

I have a proxy at http://rahul2001.com/proxy/ I site can be 'proxied' by like this: http://rahul2001.com/proxy/proxy.php/http://site-to-be-proxied.example.com/ The problem is that Google seems to be ...
5
votes
2answers
366 views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
2
votes
2answers
59 views

Is there a way to block wget from crawling and downloading my website?

I do not want ANY of my alleged website to be crawled and downloaded (css, images, html, etc). Is there a way to block wget completely?
3
votes
1answer
35 views

WordPress: Many IPs issuing wp-login redirect + wp-login register + xmlrpc. Problem or not?

My WordPress instance is hosted at my own domain and is installed in subdirectory xxxx (i.e. not in the root directory). 5 days ago I added a WP blog, and also started commenting in social media about ...
4
votes
1answer
56 views

Using robots.txt to deny access to MediaWiki special pages using substring matching

I am running a Mediawiki at the domain someurl.com/wiki/. Unfortunally it generates a bunch of automatically generated Special pages which mainly are of low quality but nevertheless are massively ...
5
votes
3answers
62 views

Google shows the wrong title for my website - robots.txt issue

Let's cut to the chase. I have this website all set up and everything, but when I type "lissa mariage" on google it shows this in translation from Romanian it says: "pages_rss_title" I noticed the ...
6
votes
3answers
144 views

How to handle SEO for brand that has a special character in the name

I am building a website (small online retail store) for a company whose brand name has a special character in it. The name of the brand is actually a legitimate Polish word that has an Ł, which is an ...
1
vote
2answers
42 views

Is there SEO value or danger when creating product comparison functionality?

I am building a Product comparison web/application into my website. It will compare products by specs, price etc. I am wondering if there any dangers of not making a user log in before using the web ...