As long as I can remember, Google has always capped the number of results that are given on any on search, so if you was to search for a "keyword" then the limit would be 1,000 this rule also applies to
site: does report the total number of pages found within the search at the top of the search.... But.... This number a lot of the time is incorrect and shouldn't be trusted, its often bugged and you can test by looking at the number at the top and then clicking another page such as 10 and compare the results (a lot of the times the number changes dramatically.)
Google Webmaster Tools
You should use Google Webmaster Tools to work out the indexes, site: command is generally good for checking if a page has been indexed i.e
site:yourdomain.com/url-of-page/ but its not designed for checking all the pages at once.
Google Site: Currently Bugged
Google has just recently released Penguin v2, this had a cluster filter update applied which prevents to many results from the same sites being displayed in search for results, for example you search for keyword
cats and one site has 5 results on the front page or the results.. This is called clustering, anyhow! this update has caused many problems and currently the
site: command is bugged and only returning a maximum of 3 pages (30 Results) for the majority of sites. This should be temporary and result when Google get around to fixing it.