A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

-1
votes
0answers
6 views

How is this?http://nightatthemuseum2.tumblr.com/

But why does goes to http://www.zcspc.edu.ph/index.php/kunena/welcome-mat/25766-hdfull-night-at-the-museum-secret-of-tomb-2014 and https://sites.google.com/site/f3ngshui2/ It really bothers me how ...
2
votes
1answer
23 views

Fixing website title on Bing

I have recently created my own portfolio website, and the title meta tags specify the title of my front page as: "Thomas Russell's Portfolio and Blog", but when I search for "Thomas Russell's ...
1
vote
1answer
15 views

Are “embed” HTML tags significant for crawlers and preview generators?

I'm wondering if <embed> tags get used to crawl and/or affect SEO rankings. Say I'm making a "widget" of HTML (maybe eventually a Polymer web widget) but let's say it's just a plain old HTML5 ...
1
vote
1answer
34 views

How can Iframes be optimized for search engines?

If my site has an Iframe. What is the best way to optimize this for search engines? We want to keep the Iframe. I have read that blocking Iframes in robot.txt is a good way to go, but this seems ...
1
vote
2answers
60 views

Is it okay to block all user agents that start with “PHP/5.”?

I'm getting a lot of spambots from China (and elsewhere, likely botnets) accessing my blog which have a user agent string starting with PHP/5.. These include such nonsense values as PHP/5.3.70 and ...
5
votes
1answer
66 views

Is it expected behaviour that Google also crawls #! urls?

I'm a developer for an AJAX webapp and, following Google's specification for crawlable webapps, we use #! to indicate that it's a AJAX application such that we can serve a static page to Google ...
4
votes
1answer
60 views

Does robot traffic visiting your website have any advantages?

What are the advantages of having bot traffic on your website? Will it raise your SEO ranking on Google? Real visitors would raise awareness of your product and site, but the visitors I'm seeing ...
1
vote
1answer
40 views

Notify search engine crawlers about user generated content so that our rankings are not biased by keywords used by users

Is there a meta tag or a way to notify crawlers that a page is generated by the user? To clarify the issue, We have a website already promoted by SEO and we want to add dynamic user generated content ...
0
votes
0answers
7 views

Is there a way to block off a single element from web crawlers? [duplicate]

Instead of using a meta no-follow no-index is there a way to block a particular piece of a website, say a span tag from being indexed? For example to include a company name while greatly reducing the ...
2
votes
1answer
47 views

Website not fetched by Bingbot: Unexpected Error

My website has been online for a couple years. It's on the first result for Google but it doesn't seem to be indexed by Bing. I tried to fetch it with bing in their webmaster tools and it always ...
2
votes
3answers
72 views

Having issues getting new pages indexed in Google

We are having issues getting our new pages crawled by Google and cannot figure out why. We've done lots of research and are still having trouble finding answers. Here's some information regarding our ...
1
vote
2answers
130 views

What are spiders trying to achieve on my site by fetching lots of non-existant urls such as various versions of phpMyAdmin?

I've recently set up a build server that compiles Java source on Github webhooks. I created a public page for downloading latest .jars with a top level domain and I'm logging requests to it. What are ...
1
vote
2answers
29 views

Private folders which are not indexable

I have a requirement where I need to distribute about 400 copies of a pdf document (about 6 MB) and .pub files, that are personalized for each user. Every user must have access to only his/her set of ...
3
votes
1answer
90 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
1
vote
0answers
14 views

Sudden decrease in pages crawled per day (from 20k/day to 300/day) [duplicate]

I've seen large drops before (like the drop mentioned in this question), but I've never seen a drop like this before and am concerned that something is seriously wrong with our site's relationship ...
1
vote
1answer
39 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
0
votes
1answer
15 views

google webmaster is crawling both file urls

webmaster is crawling both the urls, one which is SEO friendly and the one .php urls for the same page thus showing duplicate title and description for those pages. urls of .php were changed through ...
7
votes
7answers
3k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
1
vote
1answer
26 views

What would prompt multiple web crawlers to scan a site?

I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's mobile bots and others. Is this just bad luck or could something ...
0
votes
0answers
14 views

Site's pages hasn't been indexed for more than 10 days, why? [duplicate]

I checked out and robots.txt doesn't block the pages that I've been reviewing. The cached versions of the pages I reviewed in Google are made 10 days ago or more. What could be the cause that a site ...
1
vote
1answer
51 views

Can Google crawlers/googlebot read CSS content in before and after elements?

I have a div and I am placing some text in it via content in :before and :after. Can Google crawlers read this? Example: div:after { content: 'This is from CSS'; display:block; } Working ...
1
vote
0answers
16 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) ...
2
votes
1answer
83 views

How to Keep “Good” Bots That Ignore Robots & Nofollow From Getting Into Honeypots?

We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
1
vote
2answers
43 views

After the home page was redirected by a hack, it is no longer indexed. How long until it comes back?

My site was hacked, and the index page was 302 redirected. It took me a few months to get all this sorted. In the meantime, the home page lost its PageRank and was no longer included in the index. I ...
0
votes
2answers
107 views

BingBot hitting multiple subdomains all at the same time, causing panic

I have a site with multiple subdomains. On certain hours of the day, Bingbots would gather at my site and do a massive scan like this: 01:23:11 a.example.com GET /index HTTP/1.1 200 Bot.A 01:23:11 ...
0
votes
1answer
50 views

AOL search engine bot name?

Can somebody tell me the name of the AOL search engine bot? I need to allow only this bot in robots.txt.
1
vote
2answers
43 views

What to do about crawler errors after removing non-essential parts of my website?

So, recently I was re-building my website, putting on another CMS, optimizing and so on. Now I visited Google Webmaster tools and it tells I have a lot of Web crawler 404 not found errors. I am ...
0
votes
1answer
85 views

DNS mistake. Google crawled the wrong server. What to do?

Problem Exists Between Keyboard And Chair I have a couple of similar looking domains on Coudflare where I manage their DNS. An old forum and my new still MVP site. As many of you may have ...
0
votes
1answer
30 views

Understanding the maximum hit-rate supported by a web-server

I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the crawler, I gathered that my program hits the server with a new ...
0
votes
1answer
94 views

Will making my AJAX app serve HTML snapshots improve SEO?

Currently, a small ajax web app of mine uses an anchor hijacking method to display content. The user clicks an anchor, I cancel the default browser event and funnel the link into an AJAX call, after ...
2
votes
2answers
581 views

What is Google's tolerance about cloaking with unwanted crawlers (fraudsters, scammers, etc)?

We have a classified ad website. Some fraudsters, scammers and kids crawl the content. Some can be: aggressive, running several EC2 with multi-threaded curl requests resulting in 100'000s requests ...
1
vote
1answer
132 views

Google WebMaster Tools Reports a Significant Increase in 500 Response Code Errors

Google's Webmaster tools shows lots of Crawl Errors. They are reported as 500 response code. How can I go about diagnosing this issue? The website is eCommerce business using the Magento ...
2
votes
2answers
61 views

Google Webmaster Tools linked from lists dead links

How long will Google Webmaster Tools try to crawl old links? We are seeing errors in the crawler that show "linked from" URLs that haven't existed in months. Will it just keep trying to crawl the link ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
11
votes
2answers
834 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
59 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
1
vote
3answers
72 views

How do I remove a LOT of indexed pages from Google?

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
0
votes
0answers
3 views

Will I get penalised by google for overflow hidden? [duplicate]

I have 2 divs next to each other, div A and div B. Div A displays a whole wordpress post and div B displays about 15 excerpts of posts with the same tag. To get this looking nice I set the hight of ...
2
votes
2answers
114 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
1
vote
1answer
126 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
1
vote
1answer
61 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
1answer
44 views

How long is the penalty for Duplicate ecommerce content after it has been ressurected

I am fixing all of the duplicate content on my ecommerce site with all orignal descriptions etc. How long does it take google to start ranking it again? I used to have a good ranking that converted ...
1
vote
1answer
526 views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
1
vote
1answer
161 views

How to block a spam bot .htaccess proper formatting?

I have downloaded my access log file from my cPanel and found I have a bot I would like to stop. In the access log, there User-Agent string reads the following; 88.80.205.219 - - ...
0
votes
1answer
51 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
2
votes
1answer
83 views

Is there a way to opt out of Alexa rankings or prevent them from gathering data about my site?

My competitor is using Alexa to see what I am doing. I need to get stop Alexa from including my sites in their rankings. How can I do so? Is it possible to deny access to any visitor with the Alexa ...
2
votes
2answers
41 views

How do I get Google to index two URLs that differ only in the value of a URL parameter?

I have a site which has variable ID's which bring up different content For example: http://example.com/university.php?id=3 http://example.com/university.php?id=4 Technically these are two ...
0
votes
1answer
32 views

How to force search engines not to crawl my site, is there any special mark on them? [duplicate]

I want to block search engines from crawling my websites. Atleast for Msn, Yahoo, Google, and Yandex. i dont really trust robot.txt. Because they can simply ignore it and continue crawling. I would ...
6
votes
1answer
362 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
0
votes
1answer
155 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...