1

I'm trying to crawl my website using the google webmaster tools, but I'm curious as to if I've setup my parameters correctly. On my website I have different pages that's content is relative to a url parameter that is a date. (ex. example.com/date?=5-2-2018), I am wanting to crawl all of the possible combinations, so that if someone searches "example.com 4-2-2018", the corressponding website appears. I listed the date parameter in the crawl settings, but I don't know if it will be able to detect all possible date combinations. My URL structure is also controlled by HTACCESS, which changes the GET format to just "Example.com/day/5-2-2018". Anyone have some suggestions?

Improve this question
2
  • You will need to create links in the format you prefer. I suggest using pretty URLs such as example.com/day/5-2-2018. You can also create a sitemap file, however, only linked pages will ultimately be indexed. This is a catch 22 scenario. It may be impossibility to link all of the possibilities. That is the problem with sites such as yours. I ran into the same problem years ago. There maybe other options available, but for the moment nothing is coming to mind. Cheers!!
    – closetnoc
    Commented Mar 31, 2018 at 18:43
  • How do users navigate to the various dates? Do you have a calendar on the home page with a couple years of dates listed on it? Does the page for each date have a three month calendar with previous, current, and next months? If you have such calendars just make sure the links are regular <a> anchors and search engines will crawl and index your dynamic content just fine.
    – Stephen Ostermiller ♦
    Commented Jul 29, 2018 at 23:57

1 Answer 1

Reset to default
1

If all of the pages have duplicate or very similar content, it will be difficult to get Google to crawl all of the pages, and Google probably won't index them all.

You can use rel canonical tags, but I think it's risky to have too many duplicate urls. Google only has so much network capacity to crawl the web, and having too many urls with the same content could cause the bot to get tired of crawling you.

If the content is different and unique for each url, simply linking to all of the pages properly will allow for Google to crawl your site.

If on the other hand the pages are all duplicate then I think it's risky to structure your site this way.

Improve this answer
2
  • They all have different content relative to the date specified.
    – Will
    Commented Mar 31, 2018 at 18:29
  • My main kinda question is how for example, Youtube has urls listed for each youtube channel, how is that structure achieved?
    – Will
    Commented Mar 31, 2018 at 18:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.