A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
1answer
15 views

How to prevent page counts by search indexers

On a page I manage I store information about the most read pages and last read pages in the database. How can I manage to prevent taking into account accesses from search indexers? Note that I still ...
1
vote
2answers
26 views

How to track clicks on inbound links besides via URL parameters?

We use a custom CMS on a site and we're having various parameters on inbound links for own tracking and inbound marketing purposes. Usually these URL parameters are added to the links so the system ...
0
votes
3answers
47 views

Google crawl some pages, but not exist on website

Google Webmaster Tools is reporting a crawling error. 403 Access Denied for an ".html" such as "vy81TYYP/lraynb-20151123-dkd04-2290.html" page which has never existed on a my developed website. This ...
3
votes
2answers
26 views

What's the best way to crawl an AngularJS ecommerce website?

I'm trying to crawl our ecommerce website that runs on AngularJS in an effort to create a content inventory. I'm not really very good at coding. So far, I've tried DeepCrawl and Screaming Frog, but ...
2
votes
1answer
16 views

Do I really have to block MJ12Bot (as the prevailing visitor on my site)?

I am all for allowing any legitimate search engines to visit my site, but I've noticed that on my business-card-style website about every other request comes from MJ12Bot, yet for well-known reasons ...
0
votes
0answers
13 views

Google isn't crawling all Sitemap URLs [duplicate]

Googlebot isn't indexing our Sitemap http://i.stack.imgur.com/WkB1J.png There are 63 links in the Sitemap only two are indexed I've tried sending some link to the index by "fetch as Google" with no ...
0
votes
1answer
26 views

How google crawls webpages without actual page files

For example, I have an index.php file on my server, google will crawl this and retrieve the information. I also have a file named product.php, this dynamically gets content from a database to ...
2
votes
1answer
355 views

How to disallow Google crawling out of my website through images?

I'm taking care of a website which has multiple images coming from exterior links. The thing is, those links are outgoing ones, and I want to avoid that as much as possible. I tough of one solution ...
0
votes
1answer
40 views

Difference between crawling and indexing [duplicate]

If a website is cached, does it mean that it has been indexed too? In this snapshot cache date is shown as 18th may but when i did site search for this domain I found 0 pages were indexed by google ...
3
votes
1answer
23 views

Will search engine web crawlers see urls that are changed with HTML5 history?

I have a site where the different language versions are on the same url. If the language is not set as a get variable, language is set by a session variable in php, and the url is rewritten ...
2
votes
1answer
131 views

Is it possible to make a crawlable AJAX web application WITHOUT using HTML snapshots?

Is it possible to make single page application (SPA) crawlable without HTML snapshots with help of HTML5 History API pushState? I have found contradictory meanings. Can you support or refute one of ...
1
vote
1answer
16 views

Commission Junction links are registering tons of (probably false) clicks

I have a sidebar of commission junction ads, and they are plain text links (a-tags) with rel="nofollow" attributes in them I'm getting hundreds of "clicks" per day in my daily stats on CJ's ...
0
votes
0answers
10 views

Cookie-based A/B testing on same URL - SEO impact?

we'll run a domain-wide A/B test comparing our old design with our new responsive design. Though the content itself remains almost unchanged, the code structure and especially the design differ much. ...
3
votes
1answer
38 views

What do I need to be aware of when running a web crawler?

I am getting into more holistic data analysis and would like to use a web crawler to extract data from websites for use in my analytics. To be clear, I don't want to mirror data and republish, at most ...
0
votes
0answers
4 views

Is a URL without the trailing slash considered the same as the URL with trailing slash for SEO? [duplicate]

I just have a simple doubt about URLs and as I didn't find useful infos on the net, I am asking here: If I have a URL, let's say http://stackoverflow.com/questions and ...
1
vote
1answer
31 views

Dynamic product filter and SEO

I have a "product selector/filter" that uses an external data source (JSON) for product information. The user can adjust checkboxes or range sliders in the sidebar, and the pure front-end application ...
2
votes
3answers
114 views

How is my SEO affected if I make all outbound links on my website rel=nofollow?

We have a lot of outbound links on our website to various high DA websites. Would it help or damage our SEO if I make all of the links rel=nofollow?
1
vote
1answer
83 views

How to keep friendly URLs consistent?

I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed. The original site ran on a combination of Drupal and a lesser known, ...
4
votes
2answers
1k views

Screen resolution of Googlebot mobile?

Does Googlebot-Mobile have a viewport resolution it sends across? If so, what is it? It's a general question with broad relevance, but I am asking with reference to responsive design: particularly ...
3
votes
2answers
41 views

Optional query string specification with “Disallow” in robots.txt

I'm creating a robots.txt file for my website which now has a lot of pages. I'm disallowing all the pages that I do not require to crawl by so: Disallow: /folder/file.aspx I've quite a lot of ...
2
votes
2answers
40 views
1
vote
2answers
33 views

What is a tolerable request rate for bots?

I'm writing an indexing crawler for my hobby search engine. What would be a safe figure for requests per second so I wouldn't be mistaken for a DOS attack and I wouldn't get blocked by firewalls and ...
8
votes
6answers
8k views

What is the best way to exclude bots from view counts?

My website is counting visitor views on certain pages. I noticed that Google and other Bots are "clicking" to my site like crazy and some of the pages get unrealistic view counts (compared to those ...
2
votes
1answer
53 views

Can I safely block Google AdsBot from spidering specific links?

My question is similar to Google Adsbot suddenly crawling odd pages, but my use case is different. I'm running a price comparison website. I have ads directing users to my home page, and other ads ...
1
vote
1answer
22 views

Homepage not being crawled by search engines [duplicate]

My website is using WordPress with the following robots.txt User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/cache Disallow: /trackback Disallow: ...
0
votes
1answer
54 views

We want to change the URLs of an entire section of the website, what all can go wrong with SEO?

We have the following URL sections: http://www.movie-website.com/movies/movie-name/trailers And we want to change it to: http://www.movie-website.com/trailers/movie-name/ Of course we are looking ...
1
vote
1answer
14 views

What is a good approximation for a page's (Google) search rank to come back to it's original rank, after it has fallen due to technical errors?

Say we had a web page, which initially had a google rank on the first page. Say these errors occurred after some time: load time Meta title changes Some data not being visible to the crawler etc. ...
1
vote
1answer
32 views

Calculating requests per second within bing

In Google Search Console, I am able to set the rate at which google crawls my site and it tells me the details such as the requests per second and seconds between requests. In Bing Webmaster tools, I ...
4
votes
5answers
1k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
3
votes
3answers
81 views

Is buying domain names for property addresses and redirecting them to a realty website a viable SEO strategy? [duplicate]

I'm a Linux engineer, and I know quite a bit of programming languages, and can create a website pretty easily, but what I suck at, is SEO. That's an art all by itself, and that's what I need some help ...
7
votes
3answers
113 views

Should I disallow crawling of HTTP after moving to HTTPS?

I am migrating my well indexed website from HTTP to HTTPS only. Steps planned: All HTML pages canonical to be HTTPS now All HTTP pages will 301 redirect to HTTPS with the same URL All sitemaps will ...
2
votes
2answers
33 views

How to tell search engines to lower the robots.txt scanning rate on obsolete subdomains?

After putting my site into one domain instead of multiple subdomains, I checked my logs and noticed that search engines (particularly google) are scanning robots.txt several times a day for each ...
1
vote
0answers
52 views

Incorrect backlinks listed in Google Webmaster Tools (Search Console) from my other website

I have two websites hosted on the same server (same IP), say www.example1.com and www.example2.com. Google Webmaster Tools has been set for both the websites years back and everything was fine till ...
3
votes
1answer
43 views

Googlebots request types

Does Google's bot send HEAD requests as well as GET requests? I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in ...
2
votes
3answers
66 views

Is it possible to ruin your SEO by cross-linking, overly structuring a website?

We're trying to categorise our website data more and more. This will lead us to have many many categories for differect setions of information. Can doing too much of this lead to bad SEO? My ...
1
vote
1answer
2k views

How to properly (dis)allow the archive.org bot? Did things change, if so when?

I have a website that I mostly don't want to be indexed by search engines, but I do want to preserve it for eternity on archive.org. So my robots.txt starts with this: User-agent: * Disallow: / ...
3
votes
2answers
72 views

Increase in 5XX errors and 404 errors on GWT, are we under attack?

A lot of URLs are suddenly giving us 404s and 500s. On closer inspection, these URLs do not exist on our website and are being linked from random sources. As more URLs were inspected on GWT, a ...
0
votes
2answers
85 views

SEO effects of H1 carousel slider

I maintain a website for a law firm. Originally their main desired keyword was "zoning attorneys." So I created a small banner that has an h1 "ZONING ATTORNEYS" on every page with the attorneys' names ...
2
votes
5answers
90 views

Prevent robots from indexing specific word

I need to prevent robots from indexing specific word on my webpage. Here is what I did: Let's say I need to prevent JOHN GALLOW from being indexed. For that purpose, I wrote this pararaph: Over the ...
3
votes
1answer
76 views

If I rename a sitemap, does it affect my SEO?

I would like to rename my sitemap: Would it affect my SEO at all? Would I have to resubmit it on GWT? If I have a sitemap-index.xml file, that remains the same, but I change the name of one of the ...
3
votes
1answer
32 views
2
votes
1answer
49 views

Does Google prefer HTML sitemaps or XML sitemaps? And does it hurt to implement both?

I just read that Google prefers HTML sitemaps. We have so far, only implemented an XML sitemap. Would it hurt to add an HTML sitemap, and not remove the XML sitemap? Also, can you submit the HTML ...
1
vote
2answers
44 views

Why do some websites hide their sitemaps and exclude mentioning it from the robots.txt file? Does this affect SEO at all?

A lot of websites hide their sitemap.xml file. What I mean by this is that they make the URL something that a person cannot guess and then do not mention them in the robots.txt file. So my question ...
0
votes
1answer
23 views

What affect, if any, would buying domain names with high volume search terms have on SEO?

Assume the following: I have a website: www.example.com. I want to rank www.example.com/searc-term-1 on the first page of Google. Would it help the SEO of www.example.com/searc-term-1 if I bought ...
3
votes
1answer
353 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
2
votes
3answers
108 views

Using Ajax #! for Google but site is not being crawled any more

Our pages used to be formulated using the ? string but since we changed to AJAX loading we now use #!, for example: Old link: /post.php?idPost=5326 New link: /post.php#!idPost=5326 The snapshot ...
0
votes
1answer
27 views
0
votes
2answers
39 views

Is finding and using expired domains for backlinks considered Black-Hat?

http://www.matthewwoodward.co.uk/tutorials/build-expired-web-2-0-networks-increase-rankings/ This guide shows how to identify and generate backlinks for a website. If one was to follow this guide: ...
1
vote
2answers
48 views

Can you find out when Google will next send Googlebot to crawl your website?

I can find the last date when Google crawl of my website but I can't find the next crawl date. Is there some place I can look to find out when Googlebot is scheduled to return?
3
votes
1answer
35 views

How to exclude Bots from Google Analytics (sudden rise of them in the last weeks)

Since a few weeks, one of my Google Analytics Accounts shows a continued rise of Visitors and Pageimpressions what seams to be bots. They have a Average Session Duration of «<00:00:01» and by now ...