Tag Info

New answers tagged

0

Nofollow does not prevent crawling What you probably need to do is one of the following:- Have all the links on the page be redirects that are blocked using robots.txt Encode all links on the page using javascript, and have the decoding function in an included script that is blocked using robots.txt to be 100% sure Google doesn't crawl the links That ...


1

If a page has content="nofollow" , google will not index the page and links inside that page also . so your iframe page also will not be indexed . Unless the iframe link is there from some other page also which doesnt have a no follow tag . or create a sitemap with the pages in that iframe and submit to google for indexing.


1

There is some argument about how bounce rate is measured. Google has even changed how it looks at bounce rates by adjusting how it views a visit through the years. In short, Google says that the bounce rate is the rate where a user has left your site from the entrance page. This page seems to explain some of what Google sees as a visitor: ...


1

Specifying a canonical on the filter pages to the parent (non-filter) page is perfectly fine but you can also serve a meta noindex, nofollow on these filter pages as well and instruct Google what to do with these filter URL's (if they contain parameters) from the section of Google WMT you've already mentioned. Disallowing them in robots.txt is also another ...


1

Because it could then be abused to index a single page 100 times per minute, which would waste Google's server power and bandwidth. You may look at Google as a machine with limitless resources but it is not so, every byte and cpu cycle counts :) As for the 500 and 10 links submission limit per week - it is set reasonably high, most smaller sites do not ...


1

Every time I read questions like this I think of Kevin Spacey's character in Henry and June. The fellow who was always writing his greatest Novel, but was so worried about someone stealing his ideas he kept it locked away in a briefcase, carried close to his chest... Every linux user is a "legit" wget user. I use it often for grabbing debs, videos, ...


5

wget has legitimate uses, yes, but it's also quite useful for Web scraping. However, I don't think you should try to block it (or any other agent) by using the user agent string. wget respects, by default, your robots.txt file. It's true that a scraper can just switch that option off, but guess what -- it's just as easy to use --user-agent ...


-1

Why another answer? Because both are right but with caveats. I study these things as a subject area regarding security. WGet is an offline web browser. It is not interactive like Linx which is a text based browser that works something like any other browser. WGet is most often used to capture resources such as downloads, videos, audio files and capture and ...


0

If i understand your question correctly then you want to really apply a no follow on the links that are inside the iframe? Personally i would have said this to be more a job on SO then webmasters however, i'm not sure on an exact way of doing this but these may provide a temporary fix but you may want to investigate them before trying. My first approach ...


13

Wget is just a command line tool for linux that fetches resources over HTTP - all this tells you is that someone accessed your site via a command line, it could have been a bot scraping you, but there's no way of knowing for sure If your site is password protected properly, there shouldn't be any need to block particular user agents :) x


1

wget is often used for scraping. It's a command-line tool to download webpages and their assets. If your website isn't being publicized, you can almost be sure that it's a bot doing scraping. So yes, you could block it, but also be aware you may need to do something more sophisticated than blocking it with robots.txt since wget can easily be told to ignore ...


0

No difference at all as far as crawlers are able to access them (with all your valid rules) with HTTP 200 response.


2

There is no difference to search engines between those two methods. You can generate it like you want. Moreover, you need to know blocking a page with robots.txt is not the perfect way to prevent indexing of pages. You can read this question for more information.


3

I think that you may be interested in this article: Design AJAX-powered sites for accessibility. It's a very basic article, but it should give you some pointers about what to do and what not to do. AJAX should be used to improve the experience on specific pages and elements of the site, but not to build a whole site. For the user perspective, it's a bad ...


1

Disallow: /article/ will block the URL (or page) article/ and all subfolders or subpages. That's why, if the robots.txt of externaldomain.com applies this directive for the user agent is googlebot, don't worry. Here are Google instructions on this.


4

Most bots don't accept cookies (including Googlebot), however, some bots do. You send a Set-Cookie header in the response, but the bot does not send back a Cookie header in subsequent requests - so the cookie is effectively lost. Whether Google is monitoring whether the site is setting cookies, we don't know for sure, but I'd wager they probably are. ...


8

Yes. Assuming that your agent names are specified correctly, it looks like this should work. Here is a resource if you want to read more. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt



Top 50 recent answers are included