Skip to main content
10 votes
Accepted

How to avoid robots from indexing pages of my app through alternate URLs?

It happen that Googlebot discovers the alternate URL. The official way if you want your application to be accessible through all those URLs for internal purposes, is to use canonical links as you ...
Itai's user avatar
  • 5,997
10 votes
Accepted

How to allow Googlebot unlimited crawling but to have a limit on individual users?

Googlebot will be able to crawl your site fully because it never sends any cookies. There is, however, something else you need to do. Google expects sites to implement limits on the amount of content ...
Stephen Ostermiller's user avatar
10 votes
Accepted

Could a brand name called "Error 404" cause SEO issues?

You will be able to get your most of your content indexed, but there will probably be some SEO issues associated with that brand name. Google uses heuristics to determine if the page is an error page. ...
Stephen Ostermiller's user avatar
7 votes

How to get tens of millions of pages indexed by Google bot?

I've had experience of just this sort of site. I ran an article directory many years ago and the % of pages indexed and more importantly actually performing was pretty much directly correlated to the ...
Adrian Lawrence's user avatar
7 votes
Accepted

Ampersand (&) in actual URL and sitemap

if this difference of ampersand in URL and sitemap will cause any issue. tl;dr No issue, because the URLs are the same. Since in sitemap & has to be escaped I replaced & with & ... ...
MrWhite's user avatar
  • 42.9k
7 votes
Accepted

Do SEO analysis bots have any advantage in terms of website traffic?

If you aren't using these services, then you can definitely block them. The data collected by these bots are only used within those specific tools and wouldn't affect your performance in organic ...
Matthew Edgar's user avatar
6 votes
Accepted

Is it possible for Google to disregard my canonical tag?

Early 2018 update to Google Search Console will now tell you if it's overridden a page with a different canonical: Google chose different canonical than user: This URL is marked as canonical for a ...
grg's user avatar
  • 3,207
6 votes
Accepted

How to tell bots to forget a website and reindex it from scratch

That isn't possible. You need to map your old URLs to the new with redirects for SEO and user experience. Google never forgets about old URLs, even after a decade. When you migrate to a new CMS, ...
Stephen Ostermiller's user avatar
5 votes

Best way to let Google index page with tabs - SEO

This is a very common problem faced these days. This can be easily solved as follows: If you want to have same page /pets but also drive /pets/cats, /pets/dog to index separately, then ideally they ...
abhinsit's user avatar
  • 1,714
5 votes
Accepted

Redirecting robots.txt when moving to a new domain

This is really interesting question, It took me few minutes to search official statement from gary tweet.. When Google see 404/30X status, then Google will ignore your robots.txt completely. It is ...
Goyllo's user avatar
  • 5,507
5 votes

Why Google is indexing Thumbnails and not Large images?

The issue is that Google only sees one ALT tag therefore the other image is without description and will not rank in image search results. If using a lightbox then preferably you should code or use ...
Simon Hayter's user avatar
5 votes
Accepted

How can I remove this URL from Google?

It looks like that you have affiliate links which have 302 redirected to the Amazon or another affiliate site. This is a common issue for affiliate websites. The issue with 302 redirects is that ...
gintsg's user avatar
  • 546
5 votes

Google indexed my home page with a completely wrong domain

There are two ways this can happen: Another domain name points to your server. Your server is doesn't have a specific rule for that domain and defaults to serving the content of your main site. ...
Stephen Ostermiller's user avatar
5 votes
Accepted

Google CSE has indexed robots.txt itself

You could add this to robots.txt: Disallow: /robots.txt In What if robots.txt disallows itself? Google's John Mueller says: The only thing this would affect is if a link were pointing to the ...
Stephen Ostermiller's user avatar
5 votes

Indexing pages with few text (like tags with just one article) is penalizing for the SEO?

The main issue with having too many tags is that CMS such as Wordpress or Blogger will create tag pages similar to category pages which can result to many similar thin content duplicate template pages ...
Alston Antony's user avatar
5 votes

Is setting the width and height of an SVG important for indexing?

While SVG's can be a purpose of their own, websites, accessibility readers and much of what you need to account for with SEO, is based on the standard IMG schema. Because of this, it's best practice ...
Electron's user avatar
  • 314
5 votes
Accepted

Is it good for SEO to change an article's slug when changing its title?

Check out the URL of this very post on Stack Exchange, it's a great example of SEO-friendly slug+id addressing: /questions/119694/is-it-good-for-seo-to-change-an-articles-slug-when-changing-its-title ...
Maximillian Laumeister's user avatar
5 votes

How do i add custom fields in google search results?

That happens automatically. Google extracts what it thinks are key bits of information from the page, without needing any special markup or configuration on the site owner's part. It's likely an ...
GDVS's user avatar
  • 7,795
5 votes
Accepted

How should I choose which URLs I want indexed to include in my sitemap?

The pages you most want indexed are your post pages such as /blog/post-1. Those text-heavy pages with article-like content on them are the most attractive to search engines. Collectively, those ...
Stephen Ostermiller's user avatar
5 votes
Accepted

How should we handle old URLs from a recently acquired expired domain that currently redirect to empty pages?

As long as your "empty" pages are correctly returning a 404 (or 410) status code, you should not have to worry about any ranking drop in the new domain. Google will treat the URLs as removed ...
Maximillian Laumeister's user avatar
5 votes

Screaming frog runs out of memory: will this affect crawling and ranking?

Screaming Frog uses your computer's memory, so lowering the amount of URLs crawled per second will likely help you. Will that get you through the whole site? I don't know. Perhaps just run it off a ...
Mike Ciffone's user avatar
  • 6,435
5 votes

Stop any search engine from indexing/crawling images in a dedicated folder in wp

In your robots.txt file, assuming this folder is at the root, include the following: User-agent: * Disallow: /donotindex/ ... The ellipsis on line 3 simply stands in for any other Disallow or Allow ...
Henry Visotski's user avatar
4 votes

Google is displaying a URL in my site's index with the cache of that result being completely different

You have an error in your robots.txt file. On line 11 you have Allow: /, a robots.txt file doesn't say what files and directories you can allow, only what you can disallow. The only supported ...
Chris Rutherfurd's user avatar
4 votes

How to get tens of millions of pages indexed by Google bot?

There are two Possible options I know of thay be of aome assistance. One: A little trick I tried with a website that had three million pages which worked surprisingly well was what my colleague ...
Zak's user avatar
  • 41
4 votes

Will meta setting Meta="NOINDEX" eventually remove page from Google index?

I am not sure if your theory is good because it is based on a very big assumption that your new website will take the credit for your old site after deindexing. Except rel="canonical", you don't have ...
Aakash's user avatar
  • 1,316
4 votes

Should I optimize back end pages?

Here's my answer considering your comments. Back end pages shouldn't be indexed by search engines because they are useless for visitors. It means that optimizing SEO of these pages is not necessary. ...
Zistoloen's user avatar
  • 10k
4 votes

Google is reporting only 5% of our site's pages as indexed after one year

What can prevent Google from Indexing Pages? Low quality (low interaction, thin or duplicate) Robots.txt Incorrect header responses Blocked resources It's about the quality of articles While an ...
Simon Hayter's user avatar
4 votes
Accepted

Google is indexing different pages of the same list

It is fine to index your paginated content in Google search result, you can set different title for that, for example for www.myshop.com/category-list?p=2 you can set title as "Page 2 - product ...
Goyllo's user avatar
  • 5,507
4 votes
Accepted

Does rel="next" have to be in the HEAD?

The specifics of Rel next and Rel Prev can be found on the google blog here: https://webmasters.googleblog.com/2011/09/pagination-with-relnext-and-relprev.html The link attributes, <link rel="...
DEVPROCB's user avatar
  • 304
4 votes

How long before Google indexes a new (to me) domain that previously had spam and virus problems?

This is the very interesting situation as you didn't know the history of the domain. There are several things you may need to look. Virus vs Spam: Check if it had infection issue or it was penalised ...
TopQnA's user avatar
  • 3,173

Only top scored, non community-wiki answers of a minimum length are eligible