Tag Info

New answers tagged

0

Why not set canonical URL's to suggest to Google which content you want indexed. https://support.google.com/webmasters/answer/139066?hl=en


1

Are you able edit the code on the ?mobile=mobile indepently from the main page? If so rather than using just the canonical tags, maybe you should also be using the markup for mobile pages. On the desktop page, add a special link rel="alternate" tag pointing to the corresponding mobile URL. This helps Googlebot discover the location of your site's mobile ...


1

Create a subdomain example www.mobi.yoursite.com. and place the robots & duplicate web page files in the .mobi folder. Disallow robots to not index that sub domain folder. Redirect them from www.yoursite.com to www.mobi.yoursite.com. with JavaScript (you then know that they have a tablet or android device. you should look at using bootstrap for ...


1

Canonical links to the rescue There's no need to use robots to block urls that are used by your users instead what you should be using is canonical links on the desktop version, by doing so you are informing Google that the content originates from this page this will resolve any duplicate problems. Responsive design is the way forward Nowadays it's ...


2

Firstly, you shouldn't block your favicon image unless you have a very good reason. What exactly is the 'blah' supposed to represent in your example? If the URL is as you state in the question, you should just use: Disallow: /assets/ico/favicon.png That will prevent crawling of the specific image. To block the whole ico directory, use: Disallow: ...


6

Both of those would work, but they would also block anything else in those directories. If you only want to block that specific URL, you could do: Disallow: */blah/assets/ico/favicon.png Do you have a Google Webmasters Account setup? If so they have a new robots.txt test page where you can see if a URL is blocked or not based on your robots file. EDIT: ...


0

This is not an issue if you don't want Google to index these 450 pages. But you may get these warnings forever, or until you get control on the sitemap.


3

It is not possible to use robots.txt (as defined by the original specification) in your case. A line like Disallow: /profile/1 will block all URLs whose paths start with /profile/1. So this applies to the profiles 1, 10-19, 100-185 (as intended), but also to the profiles 186-199, 1000-1999, 10000, … (not intended). Workaround: Add a character as suffix, for ...


0

You are creating a file to be read by a robot, so create it with a robot: <?php ob_start(); ?> User-agent: * <?php header("Content-Type:text/plain"); $limit = 185; for($i = 1; $i < $limit ; $i++) echo "Disallow: /profile/$i\n"; ?> # rest of robots.txt here Or if you are using leading zeros (better sorting) replace the echo ...


2

You could delete the sitemap and it would not mess up the current SEO. The purpose of a sitemap is to "tell Google about pages on your site we might not otherwise discover". They help fix indexing problems, but are not required for a site to be indexed. Google (et al) are not going to de-index pages because they are not listed in a sitemap. Realistically, ...


5

You could put the robots meta tag in all of those pages: <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">


2

Google will fetch your updated robots.txt within the next few days and update their crawlers accordingly. Keep in mind this doesn't mean they will start crawling your site immediately and doesn't mean your pages will be indexed soon or rank well. It just means will know your pages are available for crawling but that is the first step to getting your pages ...


2

Never use robots during maintenance You've created the problem by blocking the site using the robots.txt, you should only ever use robots to inform search engines what to index and what not to index. Because you blocked all URLS it's likely it dropped some from its index meaning you lost your rankings on those pages. Normally this process takes a couple of ...



Top 50 recent answers are included