1

I submitted an updated sitemap via Google Webmaster and I want Google to rescan URLs in it, as some of them are not yet found in Custom Search. I found an instruction for for CSE https://support.google.com/customsearch/answer/115959 but this seems to be obsolete as this functionality has moved to Webmaster Tools, but I can't find it anywhere. Just to clarify, I already refreshed a sitemap, I want to ask Google to index the URLs in it.

update

yesterday I submitted my main page to index (using fetch as Googlebot) with all its links, including "newest entities" section. today all these links are searchable with CSE. they were searching just fine with regular Google search (using "site: mydomain.tld" filter). so I'm guessing my problem is not with Google indexing per se, but rather Custom Search Engine. previously CSE had an option to rescan an entire sitemap (please see a link above). this feature seems to be moved to Webmaster Tools. does somebody know if this feature is still available and how to call it?

  • 2
    You cannot do this. Do not try and hurry up Google. If you have updated your sitemap, Google will see it soon enough, download it, and submit to the fetch queue any new URL and will hit your other pages at it's normal rate. – closetnoc Apr 25 '16 at 13:03
  • I have some weird issue - regular search in Google (using site:mydomain.tld) finds new entities, while CSE doesn't. when I submit a link directly with Google Webmaster - it's found in a minute. I want to rebuild CSE index completely, is there an option to do this? – Oleg Shemetov Apr 25 '16 at 13:11
  • These are not two indexes. When a site creates a CSE for their site, the regular googlebot will index the entire site fairly quickly so that CSE can be populated. However, after that, it all runs at the regular rate. When you submit a URL via the Search Console, it is not really indexed for a couple of days when the regular googlebot fetches the page. What appears in the SERPs is considered temporary. – closetnoc Apr 25 '16 at 13:25
  • I didn't know this, thanks. but thing it, the page is not really new like days new, it was added more than a week ago, I can find it in regular Google search "stuff site:mydomain.tld" but when I search with CSE on site it's not found. – Oleg Shemetov Apr 25 '16 at 13:37
  • I know that the CSE result set is different. Not sure why exactly. We have been getting several questions about CSE lately. And to be truthful, I am not sure there is an answer as to why you are experiencing this issue. How are you searching CSE? What is the query? It is possible that the site: search option does not work in CSE. – closetnoc Apr 25 '16 at 13:54
1

You can't ask crawler the frequency with which it should crawl your page. Though you can define the change frequency in your sitemap but that too does not ensure any defined behavior.

Suppose your sitemap had 100 links and now it has 200 links. The crawling of delta of these links depends on the actual content change on your pages, the domain authority of your site and a lot of other factors.

But the one of the most important is the content change frequency. If you have some dynamic crawlable content or user generated content on your pages, it has a very high probability of getting crawled with a high frequency.

For now you can manually submit links within your defined quota through webmaster tool and wait for the pages to get indexed.

Useful Resources:

https://support.google.com/webmasters/answer/34441?hl=en
https://moz.com/ugc/8-reasons-why-your-site-might-not-get-indexed

  • sorry, that's not exactly the answer I was looking for. I have a very specific problem, more likely with Custom Search Engine rather than with whole Google indexing. please see question update. – Oleg Shemetov Apr 26 '16 at 7:22
  • Google doesn't use the changefreq field because they find that webmasters don't put reliable values in it. See: stackoverflow.com/questions/31349345/… – Stephen Ostermiller Apr 13 '18 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.