6
votes
1answer
317 views

robots.txt blocking specific files also block unnecessary URLs

I am using Magento for one of my site. In Magento there is a file mage (no extension file name is mage only) to block this file I write robots.txt as # Files User-agent: * Disallow: /mage But this ...
1
vote
2answers
98 views

Is using !# for AJAX URLs the standard when it comes to SEO?

Will URLs like this site.com/!#users only be crawled by Google's search engine bots, or is this a standard adopted by all search engines (e.g., Yahoo, Bing, DuckDuckGo, etc...)?
5
votes
2answers
184 views

Tell Google the list of URLs to crawl

Consider the case of Quora, which is really SEO-friendly. The problem is that there are no links to questions from Quora homepage. Essentially nothing (even via an indirect URL). So I could never find ...
4
votes
3answers
203 views

Do i need to have a separate page for each article on my website?

I am using a database to store data such as articles, titles, meta and so on. Do I need to Have a separate page for each article so that my articles show up in the list when searched on Google? Is is ...
0
votes
1answer
117 views

Spiders crawling broken links

Tl:Dr Unknown bot crawling the same broken (HTTP 400) URL over and over again. Different User Agent and different country of origin. The Problem It seems at least once a week we're getting a big ...
2
votes
1answer
180 views

How to remove Ever crawled web Pages by Google

I know that we can use noindex, nofollow and robots for excluding the pages from Google Index. This process takes a long long time to clean up website from Google Index. In Webmaster Tools => Index ...
0
votes
1answer
228 views

mod_rewrite and SEO friendliness

My website has an atypical structure and I'm not sure if this could create problems in the long run, specially for SEO positioning purposes. I have a unique, large PHP script, and I use the Apache ...
4
votes
2answers
531 views

Understanding Ajax crawling of search site

I have a couple of questions about Ajax crawling of a website, which is kind of search engine itself. The base article explains the mechanism of making AJAX applications crawlable. All this stuff ...
2
votes
1answer
150 views

What's the extent of the “damage” to having a broken URL found by the spiders (showing in Google Webmaster Tools)?

An error with our URL rewrite rules caused some pages to be not found (404 errors) when Google crawled our site. I regularly watch for errors and fix database or coding issues when they arise. We ...