Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm working on a website for someone which sells their products on their web site. There is an item.php page which displays the item for sale. I switched all item pages/inks from dynamic URLs to static ones, and Google indexed the pages just fine. However, about a month ago, they stopped indexing all newly added pages. All old pages, previous to a month ago, are still in their search results, in their rewritten URL format.

Things to note:

  • There is a sitemap.xml which has all item links written in their static format(rewritten format). The newly added products are in the start of the sitemap. So it goes from newer, to older items.

  • Google webmaster tools knows about the sitemap(i've added it manually)

  • Sitemap tests show no errors

  • The sitemap is in robots.txt

  • All new products are also linked from site's home page under as "new items" list(so bots can find items there as well, without the sitemap)

  • New items are always added to the sitemap.

  • Google's index number on the sitemap seems to be stuck at a number such as 2500 submitted, but only 2350 indexed.

  • When items are no longer for sale, they are removed from sitemap and all pages on the website. And if someone tries to access the page, page reports as a 410 GONE to browsers/spiders.

  • Google doesn't show any site errors in the webmaster tools

  • Google fetch utility fetches all the pages and links fine with success! Which makes it even more weird!

Why I believe this has to do with dynamic URL being the issue: I also have newly added categories which are NOT rewritten in the URLs, so they will be like www.example.com/category.php?id=Shoes, and Google will always find those newer pages! Yet, not the specific new items which have rewritten URLs.

I'm thinking of switching back to dynamic URLs, since it seems static URLs are not being added to index now...

Any ideas? Thank you all.

This is the .htaccess

AddHandler application/x-httpd-php53s .php 
ErrorDocument 404 /404.shtml
RewriteEngine On 
RewriteRule ^buy-used/[^/]+/(.*)$ buy-used/item.php?id=$1 [L] 

<Files 403.shtml> 
  order allow,deny 
  allow from all
</Files>

deny from 5.52.34.246

robots.txt file here:

# robots.txt generated at http://www.mcanerin.com
User-agent: *
Disallow: /cgi-bin/
Sitemap: http://www.ahparts.com/sitemap.xml
share|improve this question
    
Any chance you can provide the URL of the live site please, there may be more too it than just htaccess. –  bybe Apr 6 '13 at 19:59
    
@bybe Before I do that, look at the robots.txt file I'm about to post it in the edit. –  Dave Fes Apr 6 '13 at 20:11
    
@bybe the site is in the last line of the post now. check it out. –  Dave Fes Apr 6 '13 at 20:27
    
TA, can you give me an example of what URL is not indexed, and how long you have waited on the URL provided. –  bybe Apr 6 '13 at 20:46
    
@bybe so for example, the home page lists all new car pages and the dates they are posted. So click on any of those, then scroll to bottom of page and see all items. none of those items get indexed. However, I just did a google fetch of all the links on the home page, in a few days maybe it will show up ? But that isn't good because I had to do a manual fetch, and those are limited. –  Dave Fes Apr 6 '13 at 20:53

1 Answer 1

up vote 0 down vote accepted

Google has indexed your site perfect as expected and rather quickly too as mentioned in comment. Google will always try to return results from other sites if you do not use the site:youdomain.com your-url-or-keyword-here so at the moment I am seeing pages being returned from your site, if it doesn't pop up then you define the search more, Since Google is treating your search a broad search and often will return results from what it feels is the best result to the person searching.

Using your own search term as an example http://www.ahparts.com/buy-used/1991-Acura-NSX-HOOD-BLACK-60100-SL0-A90ZZ/57923

I search using ahparts.com HOOD BLACK 60100-SL0-A90ZZ

Google Search Webmasters

Now as you can see above you was hoping that the URL of the actual black hood would be returned but instead of has returned the parent page... This is purely because Google believes that the parent page is more valuable to the searcher, this is purely because the actual hood page is being treated as 'thin content', you have some pictures and very little text for Google to go off on... So if you want these pages ranking better it's time to write more content on those pages as the parent pages are giving better siginals to Google and is being preferred by Google. (Using a broad search such as ahparts.com often will only return 1-2 results max of your site).

Now let's try doing this search with site:ahparts.com HOOD BLACK 60100-SL0-A90ZZ

Google Search Webmasters

As you can see both pages are now found this is because Google now knows that you only want results from your domain and it's disgrading what they believe is best for you and just matching everything possible on the site.

Fatter Content

If you want your pages ranking better then its simply the case of beefing up your pages, the parent page in comparision to the hood page that you would think Google would prefer is not because of the thin content, you can have a million pictures and Google will still prefer the text content page since its able to verify the page contents more. So I understand theres so much you can write about a hood but maybe list the things like the metal type, and some other details from the stickers on the back, its not my field so I wouldn't have a clue but for sure beefing up the pages will only increase your rankings.

Page Problems

You also have some major page problems which could be causing Google to rank the pages incorrectly, on the front page for example you have JavaScript before the HTML and HEAD tags, this can cause all types of problems for search engines nevermind the end customers. This problem actually gets worse thoughout the site:

Referring to the example page we used take a look at this:

<html>
   <head>
    <title>Buy $850 1991 Acura NSX       trim(HOOD BLACK 60100-SL0-A90ZZ) Sale # 57923</title>
<LINK REL='SHORTCUT ICON' HREF='http://www.ahparts.com/images/ahparts.ico'>

       <meta name='description' content='57923 For sale - Buy 1991 Acura NSX       HOOD BLACK 60100-SL0-A90ZZ - Used Category: Hoods - Online OEM Honda & Acura auto wrecker recyclers! We search our scrap yard & pull for free. Computerized dismantlers inventory & WORLDWIDE SHIPPING! From body to mechanical parts, we salvage almost anything. Buy from our junkyard & save big by avoiding dealer pricing!'>
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD>
<LINK REL='SHORTCUT ICON' HREF='http://www.ahparts.com/images/ahparts.ico'>
<title>Used Honda Acura Parts 877-859-0023 Auto Wreckers Recyclers - Discounted Cheap</title>
<meta name="description" content="Your specialized OEM Honda & Acura auto wrecker recyclers! We search our scrap yard & pull for free. Computerized dismantlers inventory & WORLDWIDE SHIPPING! From body to mechanical parts, we salvage almost anything. Buy discounted & cheap from our junkyard. Save big by avoiding dealer pricing!">
<meta name="keywords" content="specialized, used, OEM, Honda, Acura, auto wreckers, recyclers, scrap yard, dismantlers, inventory, WORLDWIDE SHIPPING, body, mechanical parts, salvage, junkyard, save, avoid dealer pricing">

You should never have more than one head and content must always be wrapped within the HTML tag, also you should take w3c validation very seriously - not only can this help search engines rank your pages easier it ensures that your end customers hardly get any problems. 1-2 errors is generally not a problem depending if they are not major ones which you have but at present you have over 298 errors which you should get resolved.

http://validator.w3.org/check?uri=http%3A%2F%2Fwww.ahparts.com%2Fused-parts.php%3Fid%3D13015&charset=%28detect+automatically%29&doctype=Inline&group=0

share|improve this answer
    
Thanks for the answer. So the question about the duplicate I haven't been able to fix. Maybe you can help me with that. Basically the page references a header.html with an "include" statement. But the header also has a <head> and a <title>. How can I not include a second <head>& title? I am guessing I have to make a second header page for items that I want to have unique title and head tags? I can do that to avoid duplicate head and title! And the javascript style tags are there because it wont work if i put it later on in the code. So maybe I need to do external css? Advise? –  Dave Fes Apr 6 '13 at 22:06
    
oh wait, not sure what the javascript is doing. I think it maybe some old facebook thing.. hmm let me try to remove! –  Dave Fes Apr 6 '13 at 22:15
    
And let me send you some $20 for helping me :) –  Dave Fes Apr 6 '13 at 22:19
    
This site is free don't worry about it. The javascript on the front page is actually 4 elements, facebook, jquery, fadeslideshow x2. The problems vary from page to page. –  bybe Apr 6 '13 at 22:26
    
Hey I just updated some stuff, check it out. Fixed duplicate title and head tags. made sure body tags are in place. Moved jscript I think. I know this site is free, but I still want to send you something. Send me your email for paypal. –  Dave Fes Apr 6 '13 at 22:34

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.