Tagged Questions

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

2
votes
1answer
13 views

Website not fetched by Bingbot: Unexpected Error

My website has been online for a couple years. It's on the first result for Google but it doesn't seem to be indexed by Bing. I tried to fetch it with bing in their webmaster tools and it always ...
2
votes
3answers
53 views

Having issues getting new pages indexed in Google

We are having issues getting our new pages crawled by Google and cannot figure out why. We've done lots of research and are still having trouble finding answers. Here's some information regarding our ...
1
vote
2answers
47 views

What are spiders trying to achieve on my site by fetching lots of non-existant urls such as various versions of phpMyAdmin?

I've recently set up a build server that compiles Java source on Github webhooks. I created a public page for downloading latest .jars with a top level domain and I'm logging requests to it. What are ...
1
vote
1answer
18 views

Private folders which are not indexable

I have a requirement where I need to distribute about 400 copies of a pdf document (about 6 MB) and .pub files, that are personalized for each user. Every user must have access to only his/her set of ...
2
votes
1answer
33 views

Semalt ignores robots.txt, does their own form actually do what they promise?

Semalt is blatantly ignoring robots.txt and the best way to block them as a webmaster seems to be blocking Semalt referral traffic through e.g. .htaccess. I just found out that they're also have a ...
1
vote
0answers
14 views

Sudden decrease in pages crawled per day (from 20k/day to 300/day) [duplicate]

I've seen large drops before (like the drop mentioned in this question), but I've never seen a drop like this before and am concerned that something is seriously wrong with our site's relationship ...
1
vote
1answer
25 views

Prevent image hotlinking from Amazon S3 and while allowing social networks

I'm using Amazon S3 to host some assets (mostly images) on a small project. I'm trying to find a way to prevent leeching (which can be very serious should someone tried to use a network of bots to ...
0
votes
1answer
14 views

google webmaster is crawling both file urls

webmaster is crawling both the urls, one which is SEO friendly and the one .php urls for the same page thus showing duplicate title and description for those pages. urls of .php were changed through ...
7
votes
7answers
3k views

Will banning bots make a site harder to find on search engines?

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing. I want to parse the logs and get accurate statistics as to how many ...
1
vote
1answer
25 views

What would prompt multiple web crawlers to scan a site?

I ask because a few of my servers are currently being scanned by 5+ bots: Bing, Google, Baidu, Sougu, and Yandex, along with Google's mobile bots and others. Is this just bad luck or could something ...
0
votes
0answers
13 views

Site's pages hasn't been indexed for more than 10 days, why? [duplicate]

I checked out and robots.txt doesn't block the pages that I've been reviewing. The cached versions of the pages I reviewed in Google are made 10 days ago or more. What could be the cause that a site ...
1
vote
1answer
45 views

Can Google crawlers/googlebot read CSS content in before and after elements?

I have a div and I am placing some text in it via content in :before and :after. Can Google crawlers read this? Example: div:after { content: 'This is from CSS'; display:block; } Working ...
1
vote
0answers
11 views

multiple encoded requests from crawlers

Some crawlers repeatedly try to access URLs on my page that are multiple encoded in a strange way. An example: Bing bot tries to access (which gives a 404) ...
2
votes
1answer
67 views

How to Keep “Good” Bots That Ignore Robots & Nofollow From Getting Into Honeypots?

We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today ...
1
vote
2answers
40 views

After the home page was redirected by a hack, it is no longer indexed. How long until it comes back?

My site was hacked, and the index page was 302 redirected. It took me a few months to get all this sorted. In the meantime, the home page lost its PageRank and was no longer included in the index. I ...
0
votes
2answers
91 views

BingBot hitting multiple subdomains all at the same time, causing panic

I have a site with multiple subdomains. On certain hours of the day, Bingbots would gather at my site and do a massive scan like this: 01:23:11 a.example.com GET /index HTTP/1.1 200 Bot.A 01:23:11 ...
0
votes
1answer
42 views

AOL search engine bot name?

Can somebody tell me the name of the AOL search engine bot? I need to allow only this bot in robots.txt.
1
vote
2answers
41 views

What to do about crawler errors after removing non-essential parts of my website?

So, recently I was re-building my website, putting on another CMS, optimizing and so on. Now I visited Google Webmaster tools and it tells I have a lot of Web crawler 404 not found errors. I am ...
0
votes
1answer
65 views

DNS mistake. Google crawled the wrong server. What to do?

Problem Exists Between Keyboard And Chair I have a couple of similar looking domains on Coudflare where I manage their DNS. An old forum and my new still MVP site. As many of you may have ...
0
votes
1answer
29 views

Understanding the maximum hit-rate supported by a web-server

I would like to crawl a publicly available site (and one that's legal to crawl) for a personal project. From a brief trial of the crawler, I gathered that my program hits the server with a new ...
0
votes
1answer
91 views

Will making my AJAX app serve HTML snapshots improve SEO?

Currently, a small ajax web app of mine uses an anchor hijacking method to display content. The user clicks an anchor, I cancel the default browser event and funnel the link into an AJAX call, after ...
2
votes
2answers
572 views

What is Google's tolerance about cloaking with unwanted crawlers (fraudsters, scammers, etc)?

We have a classified ad website. Some fraudsters, scammers and kids crawl the content. Some can be: aggressive, running several EC2 with multi-threaded curl requests resulting in 100'000s requests ...
1
vote
1answer
97 views

Google WebMaster Tools Reports a Significant Increase in 500 Response Code Errors

Google's Webmaster tools shows lots of Crawl Errors. They are reported as 500 response code. How can I go about diagnosing this issue? The website is eCommerce business using the Magento ...
2
votes
2answers
51 views

Google Webmaster Tools linked from lists dead links

How long will Google Webmaster Tools try to crawl old links? We are seeing errors in the crawler that show "linked from" URLs that haven't existed in months. Will it just keep trying to crawl the link ...
1
vote
0answers
39 views

Subdomains for UK and USA [duplicate]

I have a site which is multilingual. I need to target it for visitors from different countries, including United Kingdom and USA. The problem is that if I make the main site for USA, and subdomain ...
11
votes
2answers
824 views

What bots are really worth letting onto a site?

Having written a number of bots, and seen the massive amounts of random bots that happen to crawl a site, I am wondering as a webmaster, what bots are really worth letting onto a site? My first ...
1
vote
1answer
46 views

Why does bingbot access my sites the most even if I set crawl rate to the minmum?

Recently my sites got overloaded and it turned out bingbot was the culprit. I set the crawl rate for the minimum for my sites in bing webmaster tools, but regardless bingbot has the most access (about ...
1
vote
3answers
68 views

How do I remove a LOT of indexed pages from Google?

A few weeks ago we have figured out that Google has indexed some information we would rather keep in some confidentiality, in the format of individual PDF files. Our assumption was that this was a ...
0
votes
0answers
3 views

Will I get penalised by google for overflow hidden? [duplicate]

I have 2 divs next to each other, div A and div B. Div A displays a whole wordpress post and div B displays about 15 excerpts of posts with the same tag. To get this looking nice I set the hight of ...
2
votes
2answers
105 views

Webmaster Tools word count

Is there a way to somehow verify that the Googlebot finds the headings and the content, for example by word count? I'm asking this because I tried a program called Screaming Frog, which fails to even ...
1
vote
1answer
111 views

Is this a good WordPress robots.txt configuration?

I am about to launch a new WordPress site and below is what I came up with for my robots.txt file. Should I combine some of these directives with a wildcard or should I include them both ways? I know ...
1
vote
1answer
55 views

Google Webmaster crawler showing internal links I cannot find

I've got the crawler showing links that appear to be double URL encoded links. Specifically links that are %2520 instead of %20 or . As far as I know I've fixed this a while ago by removing special ...
1
vote
1answer
43 views

How long is the penalty for Duplicate ecommerce content after it has been ressurected

I am fixing all of the duplicate content on my ecommerce site with all orignal descriptions etc. How long does it take google to start ranking it again? I used to have a good ranking that converted ...
1
vote
1answer
396 views

Redirect Google crawler to different robots.txt via .htaccess

I have googled for the answer all day and still couldn't find an answer. I have a virtual subdomain www.static.example.com which is a mirror site of www.example.com. It means I have just one root ...
1
vote
1answer
134 views

How to block a spam bot .htaccess proper formatting?

I have downloaded my access log file from my cPanel and found I have a bot I would like to stop. In the access log, there User-Agent string reads the following; 88.80.205.219 - - ...
0
votes
1answer
50 views

Does hosting the pre-release website on domain affect SEO?

Until now and still going strong I've always hosted my pre-release websites (my own and customers) (test builds / beta versions) on a test domain, before uploading the whole content to the official ...
2
votes
2answers
40 views

How do I get Google to index two URLs that differ only in the value of a URL parameter?

I have a site which has variable ID's which bring up different content For example: http://example.com/university.php?id=3 http://example.com/university.php?id=4 Technically these are two ...
0
votes
1answer
32 views

How to force search engines not to crawl my site, is there any special mark on them? [duplicate]

I want to block search engines from crawling my websites. Atleast for Msn, Yahoo, Google, and Yandex. i dont really trust robot.txt. Because they can simply ignore it and continue crawling. I would ...
6
votes
1answer
353 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
0
votes
1answer
123 views

Google Webmaster Tool: “Severe health issues are found on your site” [closed]

I just saw the "Severe health issues are found on your site. Is robots.txt blocking important pages?" message on Google Webmaster Tools. This is what I have in robots.txt: User-agent: * Allow: ...
3
votes
2answers
462 views

Serve a different robots.txt file for every site hosted in the same directory

We have a global brand website project for which we are only working the LATAM portion. There is a website installation process here that allows to have one website installation with several ccTLDs, ...
0
votes
1answer
85 views

Baidu spider does not obey DNS TTL

I moved my website from one VPS to another on 16th April. The A record of the domain name has a TTL of 86400. I made the IP address change at 19:00 GMT 16th April. I do not want the site to face ...
3
votes
1answer
111 views

How to make page content crawlable , which is requested in script from ajax on page load

I hope that the contents which is pulled on page load could be crawled by google robot, how to do that ? I went to Google Webmasters tool to ensure that whether Google crawler could see the content ...
0
votes
1answer
69 views

Why does bot traffic have a high (e.g. 100%) bounce rate in GA?

Since bots may request many pages at a time, why do they so often have a 100% bounce rate? It seem unlikely that they would only request one page, then leave for over 30 minutes, then request another ...
1
vote
1answer
84 views

Replacing url parameters tool usage with canonical / robots.txt to avoid crawl time

For a long time I've been getting Googlebot found an extremely high number of URLs warnings, which I know don't necessarily indicate there is a problem, but I am trying to reduce the amount of links ...
5
votes
5answers
1k views

Should I block agent Wget/1.12 (linux-gnu)?

Is that a scraper? It tried accessing my site which is currently password protected for testing purposes. shall I block it?
1
vote
2answers
118 views

What is better for SEO robots.txt file or dynamically generated robots.txt URL?

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule ...
5
votes
3answers
209 views

Does meta name=robots content=nofollow prevent indexing iframes?

Let's say I have a publicly accessible page that contains an iframe back to a different page on my own domain. By default, google (and other crawlers) will index that page and the iframe's content. It ...
1
vote
1answer
78 views

Should my site be AJAX crawable?

I'm developing a website that uses a lot of AJAX. Now this is the first time I use AJAX and read something about making an AJAX website crawable I don't have any experience of SEO, but I read about ...
2
votes
1answer
77 views

Google bots and iframe

Here is the problem: Let say I have mydomain.com. The externaldomain.com/article/mydomain has an iframe with mydomain.com inside but the externaldomain.com robots.txt contains: Disallow: /article/ ...