I have a robots.txt for my google-hosted (gae) webapp with the following contents
User-agent: * Disallow:
I also have a sitemap.xml.gz (currently 49.3 kB) that is generated by a python function.
class SiteMap(webapp2.RequestHandler): def get(self): days = 60 url = (os.environ['HTTP_HOST'] if os.environ.get('HTTP_HOST' ) else os.environ['SERVER_NAME']) i = \ '<?xml version="1.0" encoding="UTF-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" >' for ad in \ Ad.gql('where published = True and modified >:1 order by modified desc' , datetime.now() - timedelta(days=days)): i = \ '%s<url><loc>http://%s/vi/%d.html</loc><lastmod>%s</lastmod><changefreq>daily</changefreq><priority>0.8</priority></url>' \ % (i, url, ad.key().id(), #filters.slugify(ad.title), ad.modified.date()) i = '%s</urlset>' % i self.response.headers['Content-Type'] = 'gzip' self.response.headers['Content-Length'] = str(len(i)) self.response.out.write(compressBuf(i))
GWT says there are "no maual actions needed for junk removal" for the content but still I think that it doesn't index as much as it should looking at the graph for what's indexed and why the drops?
Can you tell me why the index fluctuates the way it does on the graph and if there is something I can do to improve PR and/or search index status? I don't target any spec keywords but I just want google to index everything.