Tag Info

Hot answers tagged

6

Both of those would work, but they would also block anything else in those directories. If you only want to block that specific URL, you could do: Disallow: */blah/assets/ico/favicon.png Do you have a Google Webmasters Account setup? If so they have a new robots.txt test page where you can see if a URL is blocked or not based on your robots file. EDIT: ...


5

You could put the robots meta tag in all of those pages: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">


3

It is not possible to use robots.txt (as defined by the original specification) in your case. A line like Disallow: /profile/1 will block all URLs whose paths start with /profile/1. So this applies to the profiles 1, 10-19, 100-185 (as intended), but also to the profiles 186-199, 1000-1999, 10000, … (not intended). Workaround: Add a character as suffix, for ...


2

Firstly, you shouldn't block your favicon image unless you have a very good reason. What exactly is the 'blah' supposed to represent in your example? If the URL is as you state in the question, you should just use: Disallow: /assets/ico/favicon.png That will prevent crawling of the specific image. To block the whole ico directory, use: Disallow: ...


2

You could delete the sitemap and it would not mess up the current SEO. The purpose of a sitemap is to "tell Google about pages on your site we might not otherwise discover". They help fix indexing problems, but are not required for a site to be indexed. Google (et al) are not going to de-index pages because they are not listed in a sitemap. Realistically, ...


2

Google will fetch your updated robots.txt within the next few days and update their crawlers accordingly. Keep in mind this doesn't mean they will start crawling your site immediately and doesn't mean your pages will be indexed soon or rank well. It just means will know your pages are available for crawling but that is the first step to getting your pages ...


2

Never use robots during maintenance You've created the problem by blocking the site using the robots.txt, you should only ever use robots to inform search engines what to index and what not to index. Because you blocked all URLS it's likely it dropped some from its index meaning you lost your rankings on those pages. Normally this process takes a couple of ...


2

Google will fetch the robots.txt file itself from your site every 24 hours. If you make changes to your robots.txt file, you must wait a day to ensure that Googlebot picks up your changes. After it has the correct robots.txt file, Googlebot will start crawling and indexing your entire site properly. As a general rule, I expect to see changes to the ...


2

24 hours is not enough time to wait. Google will almost certainly not be crawling your site that frequently. It could take days or weeks before they re-index your robots.txt and crawl your site again. And to answer the question you will have next, no, there is nothing you can do to speed that up.


1

Are you able edit the code on the ?mobile=mobile indepently from the main page? If so rather than using just the canonical tags, maybe you should also be using the markup for mobile pages. On the desktop page, add a special link rel="alternate" tag pointing to the corresponding mobile URL. This helps Googlebot discover the location of your site's mobile ...


1

Create a subdomain example www.mobi.yoursite.com. and place the robots & duplicate web page files in the .mobi folder. Disallow robots to not index that sub domain folder. Redirect them from www.yoursite.com to www.mobi.yoursite.com. with JavaScript (you then know that they have a tablet or android device. you should look at using bootstrap for ...


1

Canonical links to the rescue There's no need to use robots to block urls that are used by your users instead what you should be using is canonical links on the desktop version, by doing so you are informing Google that the content originates from this page this will resolve any duplicate problems. Responsive design is the way forward Nowadays it's ...


1

I do not recommend a redirect but a rewrite. Google will follow a certain number of redirects (up to 5 per RFC specifications), but you cannot expect this of all bots. The Google Developer's site provides details on Robots.txt Specifications. And specifically about how they process the results. A Redirect would send the bot to a different URL. Using a ...



Only top voted, non community-wiki answers of a minimum length are eligible