Tagged Questions
1
vote
2answers
33 views
Robots.txt Blocking Blog Tags?
Tags are causing duplicate content on my website and I want to make sure that I block them from being indexed. Is this the right syntax to use for blocking those Tags from being crawled?
User-agent: ...
0
votes
2answers
54 views
Googlebot ignore my new site [duplicate]
I have the following situation: I had a blog (made using Joomla) that in some month was pretty indexed by Google.
For some technical problem I have delete it and I have newly created using WordPress ...
0
votes
1answer
39 views
Addon domains and SEO
I have a shared hosting account with a main domain and many add-on domains. I don't actually use the main domain, it just has a landing page.
Can I tell bots not to index the main domain WITHOUT ...
1
vote
3answers
100 views
Google Indexing Pages with noindex meta tag
I got a message in Google Webmaster Tools about "Googlebot found an extremely high number of URLs on your site" with a long list of example URLs.
For some of the pages listed, there was nothing to ...
2
votes
1answer
110 views
Is there any advantage/disadvantage to using robots.txt to disallow access to legal pages such as terms, privacy policy, etc.?
As I understand, having repetitive content is a detriment to search engine placement.
Given that many websites that use similar or even identical "Terms and Conditions" and "Privacy Policy" pages due ...
3
votes
2answers
185 views
Robots.txt on one line
Can I combine robots.txt rules on one line, separated by whitespace?
Will this work?
User-agent: * Disallow: /
6
votes
2answers
77 views
What kind of redirect (301 or 302) for an email links tracker?
We are developing an email sending application ("à la" Mailchimp). Hyperlinks inserted by our users, in the emails they want to send, are replaced by a tracking URL on our application ...
5
votes
1answer
233 views
should I block wordpress archive pages from search engines?
I use WordPress and /sample-post/ URLs for my posts and /yyyy/mm/ for the archives.
Google has indexed the site fully.
Since the archives currently display full posts I've thought that I shouldn't ...
0
votes
1answer
535 views
SEO not indexing my site: “A description for this result is not available because of this site's robots.txt – learn more.” [duplicate]
When googling my site I get following text under the result for my domain:
"A description for this result is not available because of this site's
robots.txt – learn more."
My robots.txt, ...
2
votes
2answers
88 views
Will duplicate international (i18n) content hinder SEO rankings?
Google clearly states that duplicate content within a single, or multiple, domains is not advised. This is understood, but I am not sure of any exceptions for sites with region-specific content that ...
4
votes
3answers
72 views
SEO Removal Time?
I was searching about removing the a domain/site being listed in Web Search Results (like, Google Search) and i found solution, at here.
I believe it will be the solution but what i still need to ...
2
votes
2answers
156 views
How to fix 404 crawl error of a website
I have a navigational links in a php page nav.php, Which is included in all pages of my website. I have got folder structure like below
+top_folder
-config.php
+includes_folder
...
1
vote
1answer
49 views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
0
votes
1answer
49 views
If I blocked a directory, is a 301 error expected?
I blocked a directory in my site that is not secret but meant for internal discussions that are not relevant for search engine results. After running "Search as Google" at Google Webmaster Tools, the ...
2
votes
2answers
106 views
How to get search engines to properly index an ajax driven search page
I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial ...
3
votes
3answers
111 views
SEO consideration for duplicate sites
I am building a brochure-ware website for a company that sells products all across the world. They need the site to ask the user what region they are in before using the site; there are 5 regions. ...
1
vote
3answers
108 views
robots.txt file Help needed
What if I put same robots.txt file in multiple directories and folders?
User-agent: *
Disallow: /otherdirname/dirname2/
Can it'll affect anyhow ??
And how much time it generally takes to refine ...
4
votes
3answers
280 views
Google indexing page with parameters but page is Disallowed in robots.txt
I have the following in robots.txt:
User-agent: *
Disallow: /refer.php
User-agent: NinjaBot
Allow: /
Sitemap: http://www.mysite.com/sitemap.xml
The refer.php file does various things depending on ...
3
votes
1answer
89 views
Handling SEO for Infinite pages that cause external slow API calls
I have an 'infinite' amount of pages in my site which rely on an external API. Generating each page takes time (1 minute). Links in the site point to such pages, and when a users clicks them they are ...
0
votes
1answer
247 views
How to secure robots.txt file?
I would like for User-agents to index my relative pages only without accessing any directory on my server.
As initial thought, i had this version in mind:
User-agent: *
Disallow: */*
Sitemap: ...
5
votes
3answers
274 views
Does loading images from a server that disallows robots affect SEO?
Lets say a page on example.com has all of its images coming from img.example.com:
<img src="http://img.example.com/keyword.jpg" alt="keyword"/>
while robots.txt on img.example.com disallows ...
1
vote
1answer
110 views
Do or can robots cause considerable performance issues?
At work we are in a discussion with team members who seem to think bots will cause us problems relating to performance when running on our services website.
Our setup:
Lets say I have site ...
4
votes
3answers
2k views
In Robots.txt, how do I disallow all pages except for the index page?
I want google to index my index page, but do not index any other pages.
User-agent: *
Disallow: /
So far I have this. However...when I do this, Google does not index my index page. When I search ...
2
votes
1answer
173 views
What methods are effective/necessary to quickly remove URLs from the Google index? [duplicate]
Possible Duplicate:
How to Remove URLs from Google Search Engine
In short, I have ~1000 URL's that need removing from the Google index ASAP, this is as of a week or so ago, and over that ...
2
votes
2answers
163 views
Would a 403 page, a blank page and a page redirect be indexed by search robots?
I am wondering if a disallow directive needs to be issued to robots on pages that will generate:
a 403 forbidden error;
a blank page, or;
a page redirect (to log in page for example).
Would not ...
0
votes
1answer
107 views
Hiding a particulat page from search engines not to index [duplicate]
Possible Duplicate:
Blocking Search Engines to Crawl Parts of a Website?
I have a page which i don't want search engines to index or crawl. I am not sure hat should i put in my robots.txt ...
0
votes
1answer
184 views
Is “robots.txt” necessarily needed? [duplicate]
Possible Duplicate:
Do I need robots.txt?
From my understanding robots.txt is only needed to do something different from the norm like exclude a certain file directory and such, but if this ...
0
votes
3answers
112 views
Robots are still disallowed even without a robots.txt file
I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt rile has ...
0
votes
1answer
147 views
Stop searches showing tld domain sites appearing as subdirectory of main holding site domain?
I'm not sure if this is a problem specific to how my hosting is structured or not.
Let's say the main domain for my account is maindomain.com (for purpose of discussion)
I am building other sites, ...
0
votes
2answers
1k views
How to control what appears on google search results?
So I have just built a site and all is well... but I want to control what appears on google search results.
Lets say I google for "thefwa", first what comes is the link to www.thefwa.com, then ...
7
votes
2answers
153 views
Robots.txt Issue
So I've been assigned to take a look at our SEO (an area I have some, but not amazing competence in), and the first thing I noticed is that our robots.txt file says the following:
# go away
...
1
vote
2answers
420 views
Should a mobile-only site use robots.txt?
I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a ...
1
vote
6answers
77 views
Uploading a non-finished website
I have a pretty basic question. I developed a neet little website wich I'm ready to upload, but still needs a bit of work. The designer needs the html to do his work so the website needs to be ...
2
votes
2answers
242 views
Dynamic URLs are Restricted by Robots.txt; does it really matter for SEO?
Today, I was searching for Robots.txt on Google and I found this such a great thread regarding it. I checked Google webmaster tools and found that, there are 117 URLs are restricted by Robots.txt. ...
1
vote
1answer
285 views
Facebook and Crawl-delay in Robots.txt?
Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
2
votes
2answers
120 views
Pointing to Google that a specific website is for mobiles
I have 2 websites: let's call them A and B.
A is a Desktop optimized website.
B is a mobile version of A, but it's not on the same domain. It also has a smaller amount of content/pages than A
How ...
4
votes
1answer
131 views
Do I need robots.txt?
I want to allow all bots to crawl everything, do I need to create a robots.txt?
If the file is missing will it have a negative influence on some crawlers?
If I need to create it, what should it look ...
1
vote
2answers
107 views
Robots.txt and specific file inclusion
Is it possible to use robots.txt to disallow crawling of a folder, but allow crawling a specific file in that folder?
1
vote
1answer
69 views
Search and Tags for robots.txt
After setting up a blog at blogspot.com I added the site to my webmasters tools of google.
I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the ...
2
votes
1answer
240 views
How do I disallow an entire directory with robots.txt?
I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example
http://someurl.com/first_url
...
1
vote
1answer
46 views
Blocking URLs and canonical questions
We are running a Magento store set up and was looking to block all pages with the exception of a select few.
It seems the only way we can do this is with blocking direct paths to files that are in ...
3
votes
2answers
133 views
How to temporarely remove a website from search engines?
we are working on a new website and we forgot to disallow robots after putting the test version online, on a subdomain.
now we are being already referenced by bots (google, yahoo...) and we dont want ...
3
votes
1answer
327 views
Quick Question, robots.txt Disallow: /*/ does what exactly?
A SEO firm suggested changing the robots.txt to:
User-agent: *
Disallow: /*/
Allow: /ims/
I'm not sure what that would do, but my guess is that is would tell all robots to index nothing but the ...
3
votes
2answers
192 views
How can I allow search engines to index my invite only website in ruby on rails?
I have a ruby on rails website that will be in invite-only mode for the next couple of months. Currently I have it set up so visits to any page performs an authentication:
before_filter ...
2
votes
3answers
298 views
Google is indexing unintended content that is either unpublished or a secret part of site
Google is indexing parts of my site that I haven't linked to, it's a mobile version of website that I'm working on - www.domain.com/m/ <-- note the M.
How is that even possible? Only thing I can ...
8
votes
4answers
420 views
Robots denied by domain is still listed in search results
So, on all of our sites that are not search facing we've applied a robots.txt file (per How to exclude a website from real-time Google search results ?, or any other similar question).
However, if ...
3
votes
2answers
196 views
Should I disallow robots on A/B test pages?
I'm going to be setting up an A/B test using Google's WebsiteOptimizer on my homepage, and I plan to name my two test variants index_a.php and index_b.php.
Should I be putting in rules to my ...
1
vote
2answers
520 views
SEO: disallowing Google from indexing forms?
From a SEO point of view, would you exclude forms from being indexed/crawled by Google or not?
I mean my forms hardly ever contains keyword/keyphrases.
So I'm wondering what's the point of letting ...
1
vote
1answer
104 views
To disallow indexing the category and tag listings in a blog
Mark Wilson says that category and tag listings in a blog should be disallowed in order to prevent duplicate content. I understand this.
However, I want to put internal links on keywords in the blog ...
3
votes
3answers
4k views
How to stop certain urls from being indexed
When I type site:example.com (using my domain obviously),
I get several link errors showing up in the listing. Typically, they are of the form:
www.example.com/some/fixed/path/admin/unblockUser/11
...
