Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm a webmaster and I have a question about how google or other search engines index site that are search engines. In a normal site, from the homepage if I created a link to all the section and pages of my websites, google index all the pages of my sites.

But sometimes I see sites that they don't have this link to internal pages, for example:

search "waitress london" in google, you find this result:

www.jobrapido.co.uk/?q=waitress%20london

But if you go on the site homepage

www.jobrapido.co.uk

you have only a search form like google, there is no link to internal search, and I can't find external site linkink to

www.jobrapido.co.uk/?q=waitress%20london

So my question is:

How google know this link?

Any ideas?

share|improve this question
add comment

migrated from stackoverflow.com Aug 25 '11 at 20:19

This question came from our site for professional and enthusiast programmers.

5 Answers

All search engines consist of three parts: a database of webpages, a spider operating on that database, and a series of search engine software that decide how search results are displayed.

First, a spider visits a webpage, reads it and then follows the links to other pages within the website. All the data that the spider has gathered is stored in the database, which contains a copy of every webpage that the spider has found. The spider will often return regularly to a website to look for any changes and update the database accordingly.

The search engine sotware sifts through the millions of webpages stored in the database, identifying the body text, links and other content on the page. It does this so it can find matches to a search and ranks them in order of their relevancy. Page titles, meta-descriptions and other various elements play a part in determining where a website should be ranked.

One advantage of having spiders revisit your site is that you can make changes to your webpage in order to appear higher in their relevancy rankings, then see if your changes worked. This is what search engine optimization is all about.

share|improve this answer
add comment

Google has a bot... the googlebot in which it crawles a webpage ang like a bug scatters itself and explores link within a website and copies it to it's database. so if you ask how they do it, it is because the whole website is already in there storage devices in their company. :))

share|improve this answer
add comment

Google says that is not good to set indexable other SERP exception if in that page there's some unique content of quality. I've a search engine based on CSE and - for exit from hard penalization - I set disallow serp pages on robots.txt.

share|improve this answer
add comment

just do a ctrl+u and you will see that they have a few links to subpages on the homepage; the one you see if you click on "Most popular cities" and "Most popular jobs" under the search fields. Google can follow those links since they are only hidden via css. And from any of those pages on you can discover more and more pages, keeping the google bot busy for quite a while.

share|improve this answer
add comment

As long as someone linked to that URL, and Google found that link while crawling that website, then they will know about it and index it.

That page can also be submitted via an XML sitemap.

share|improve this answer
    
I get the feeling this isn't the right answer. I'm with the OP on this: I've searched for things that led me to other sites that had absolutely nothing to do with the search terms, but did contain them in a link, and the text looked like it had simply substituted my search term in a body of unrelated text –  Chris Lively Aug 25 '11 at 20:23
3  
How they are found and how those pages are constructed are two different things. –  John Conde Aug 25 '11 at 20:45
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.