Hot answers tagged

3

An internal web crawler can be used for the following purposes: Creating a Localized Search Engine. Scanning Pages to Detect Dead Links, where the target page was deleted or unpublished from a CMS, but the links to that page still exist in other pages. Finding Breadcrumb Click Trail Paths in massive websites. Finding Orphaned Pages in CMS generated ...


2

Using a crawler on your own site can be good if you get a lot of pages on your site which don't necessarily have an entry in a database to search, this is effectively a way to implement site search for your site. It can also be used where you need a single unified search for a number of separate web properties. I implemented this for a university which had ...


2

No what you are asking unfortunately is not possible. The site description which is shown in search results isn't based on the description meta tag anymore, instead Google and other search engines use content from the page which the user can see as the description in search results. If you want description text your search engine results you need to have ...


2

Crowded markup is not a huge issue these days. Once long ago it was considered a sign of a badly designed site and Google acted accordingly but these days with the rise of dynamic interactive database driven websites and rich user interfaces Google has stopped ranking based on HTML clutter. As long as all the HTML is not causing really long page load times ...


1

Crowded markup is advantageous these days for a couple of reasons: Anyone who wants to steal part of your code to make it their own will have a harder time doing so when all the HTML is crowded together as one instead of html with nice comments between blocks of code. The amount of bytes required for crowding your HTML is often less than that of pretty ...


1

Disallow: /folder/file.aspx What you had in the beginning is all that's required - in order to block both /folder/file.apsx and /folder/file.aspx?foo=bar. If in doubt check it with the robots.txt testing tool in Google Search Console. robots.txt is prefix matching, so there is never any need to include the wildcard (*) at the end of the URL path. The ...


1

See page 8-9 on the Google page SEO guidelines http://static.googleusercontent.com/media/www.google.com/en//webmasters/docs/search-engine-optimization-starter-guide.pdf Opinion based: Matching URL's with titles & menus is best practice in my view. Avoid hashes - URL's - same as titles should be human readable and in best case helpful to the ...


1

From Webmaster Tools FAQ: Q: The Search Queries feature says that my site ranks #5 for [cookies], but when I search for [cookies] I see it ranking #8 (or I don't see it at all). What gives? A: Several things: Different searchers can see different results, based on things like where they're searching from (geographic differences), personalized ...


1

Here's how consumer facing domains were originally meant to work: domain.com didn't serve anything necessarily. mail.domain.com hosted your email server. ftp.domain.com hosted your FTP server. irc.domain.com hosted your Internet Relay Chat server. www.domain.com hosted your World Wide Web server. All of these are subdomains. The World Wide Web was ...



Only top voted, non community-wiki answers of a minimum length are eligible