3

I own a website and I'd like to be able to have an history of how some page and keywords ranked in google over time, to be able to understand if certain changes I made have positive or negative effect.

Now, I think that doing it by hand is a bit error prone and time consuming and I'd like to understand if I can write a simple tool to do that.

Unfortunately, from what I can read from the Google TOS (Term of services) is not allowed to crawl their result, instead they suggest to use the Google Custom search API.

And here is the question. As the custom search API

(from https://support.google.com/customsearch/answer/70392?hl=en)

are unlikely to match those returned by Google Web Search

how can I acquire accurate informations about keyword positioning without violating the google TOS?

  • I am confused. Is there a reason why the Google products cannot help? As well, please know that most of the online rank/keyword sites are junk. There is some value in some here and there, but it is mostly minor. Keep in mind that Google, Bing, and others are semantic search engines. In particular, Google does not make keyword matches and any apparent search term match is incidental as the last step in many where search terms are highlighted. So doing the keyword chase can potentially semantically score you within a narrow window and not increase search potential. – closetnoc Apr 23 '16 at 17:17
  • @closetnoc Can you point me to a google product that offer this feature? Also, I'm aware that most of the online rank tools are junk but as a matter of fact, the SEO industry rotate around few key keywords. And tracking their position over time is critical. – snovelli Apr 23 '16 at 17:26
  • I would be wary of what the SEO industry says. Many of them are not technical people and are guessing without having a clue as to the technologies used and how they work. Occasionally, one or two gets something right here and there. ;-) There is some good info of course. However, the notion of keywords is just plain wrong. I often talk about semantic weighting and linguistics. It is about topic influence. I was thinking about Google Search Console and Analytics. Google is, as far as their search engine is concerned, the authority in how your site actually performs and the potential it has. – closetnoc Apr 23 '16 at 17:47
  • @closetnoc I understand that from Google perspective, the keyword match is incidental, and the fact that the notion of topic takes over the keyword, is the only logical evolution of search engine. Still, from the user point of view (especially in the online shop business) keywords ranking are still a solid value and correlates very closely with traffic. That's why (at least in my opinion) tracking a keyword position over time, does make sense. – snovelli Apr 23 '16 at 18:34
  • I am not arguing with you. Not at all! In fact, for e-commerce, there is less opportunity for very strong semantics signals. I was just warning you only because we get sooooooo many that are convinced that is how tings work and go about doing weird things. ;-) As far as tracking, I mostly stay away from most online sites. SEMRush at least does something worth while. Googles Search Console list of site keywords is pure junk. Don't get me started on the [not provided] thing. Still, I was wondering if these tools did not do something for you since only Google really knows how your site performs. – closetnoc Apr 23 '16 at 18:50
2

Use Google Webmaster Tools: https://www.google.com/webmasters/tools

After adding all your sites (http - www, non-www, https - www, non-www) you can find your rankings Under Search Traffic -> Search Analytics.

Only one site should show keyword rankings, you do have everything 301 redirected to a single site, right?

As User Snovelli pointed out, Google Webmaster Tools can also be accessed programmatically via their API at https://developers.google.com/webmaster-tools/v3/quickstart/quickstart-java

| improve this answer | |
1

As noted by others, tools like Google Search Console and Bing Webmaster Tools will give you a limited insight into your rankings (and other metrics) over time.

For more complete data, you'll need 3rd party rank-tracking software. There are many out there, each with their pros and cons, but some popular industry standards are Searchmetrics, SEMRush and Pi Datametrics.

Bear in mind that these tools are limited in that they can't account for personalisation from, e.g., search history, location, etc.

| improve this answer | |
  • Thank you for your answer. I was aware of third party software providing this data and the question was more of a usecase to understand how the hell this services gather data if not by violating google TOS. Turns out google expose a fairly complete API to retrieve the query/position data, so there's actually a chance that they use it ;) – snovelli Apr 29 '16 at 9:51
  • @snovelli Those services don't use Google's API. They have their own methods that may or may not skirt around TOS issues. – GDav Apr 29 '16 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.