Tagged Questions
2
votes
1answer
39 views
What does “all” mean in meta robots tag?
Several websites have the following meta tag aimed at search engines like Google, Bing, etc.:
<meta name="robots" content="all" />
What does it do?
5
votes
2answers
140 views
Which meta “robots” tag gets preference?
My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
1
vote
1answer
34 views
Different Blocked URLS Stats in Google Webmaster
Just want to know i got different blocked urls stats at webmaster.
I blocked entire site for revamp and latest builds implementation. Here are different stats
Under Health tab> Blocked URls>2,580
...
5
votes
2answers
415 views
Should I block bots from my site and why?
My logs are full of bot visitors, often from Eastern Europe and China.
The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on.
Should I block these bots from my site and ...
1
vote
1answer
49 views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
0
votes
1answer
119 views
mod_rewrite and SEO friendliness
My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes.
I have a unique, large PHP script, and I use the Apache ...
2
votes
3answers
310 views
User-agent identification and SEO crawler database
I am requested to analyze the traffic log of a site. In particular I have to identify the crawlers starting from the collected user agent values.
I know there are 'trap' links that you can use to ...
3
votes
4answers
1k views
Why deny access to website for msnbot/bingbot?
I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for ...
1
vote
1answer
110 views
Do or can robots cause considerable performance issues?
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Our setup:
Lets say I have site ...
1
vote
1answer
421 views
Question about robots.txt Google shows urls are blocked
I am working on this site dealsin.us and the Google webmasters tool shows that there are about 9800+ urls blocked by robots.txt. You can view the robots.txt here I have blocked some directories which ...
1
vote
1answer
285 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
1
vote
2answers
66 views
SEO: How to stop certain web-pages from “having importance” [duplicate]
Possible Duplicate:
SEO - Reducing significance of irrelevant kewords.
I wrote "having importance" since I do not really know the right way to say this.
I have certain pages in my site ...
2
votes
1answer
413 views
Is it possible to block an entire site to Google then list exceptions?
I usually disallow subdirectories in the robots.txt file and I was wondering if it's possible to do it the other way: block everything and list the main index file and the others pages I'd like search ...
0
votes
2answers
249 views
Menu hidden links and bots!
I want to make a menu with a list with hidden links and when the user points the menu with he mouse the links will appear, my problem is that the bots may think this as a deceiving tactic and lower my ...
