Hot answers tagged

3

An internal web crawler can be used for the following purposes: Creating a Localized Search Engine. Scanning Pages to Detect Dead Links, where the target page was deleted or unpublished from a CMS, but the links to that page still exist in other pages. Finding Breadcrumb Click Trail Paths in massive websites. Finding Orphaned Pages in CMS generated ...


2

Just keep it like that, clean. It's quite easy to remove index.html from a URL with mod_rewrite. Let's say we want to redirect www.example.com/index.html to www.dense13.com: RewriteEngine On RewriteRule ^index\.html$ / [R=301,L] If you're not familiar with .htaccess syntax the RewriteRule directive has three parts, a pattern (^index.html$), a ...


2

Using a crawler on your own site can be good if you get a lot of pages on your site which don't necessarily have an entry in a database to search, this is effectively a way to implement site search for your site. It can also be used where you need a single unified search for a number of separate web properties. I implemented this for a university which had ...


1

I've written a search engine bot before. Building it was fun! There is a lot of cleanup that you'll have to do on the URLs, as munged URLs will constantly crash your search engine bot when it runs. I'd set it to a 5 second sleep timer. Websites probably won't care as long as you aren't trying to hit them with 10K requests in <1 second. It's pretty easy ...


1

Disallow: /folder/file.aspx What you had in the beginning is all that's required - in order to block both /folder/file.apsx and /folder/file.aspx?foo=bar. If in doubt check it with the robots.txt testing tool in Google Search Console. robots.txt is prefix matching, so there is never any need to include the wildcard (*) at the end of the URL path. The ...


1

We have also faced similar issue on our website. This happened because initially you must have disallowed your home page in robots.txt. Now even if you allow, it takes sometime around a week-two to reflect in google indexes. I have tested your robots.txt and it looks fine.


1

The Google AdsBot does not only check one level deep to assess the page and it is not simply for assessing if the page is a good landing page or not. The AdBot has multiple roles, firstly to check the actual landing page for the ad to see what is on the landing page, this improves the quality of ad delivery to end users by more accurately targeting ads to ...


1

Aside from setting the "Crawl Control" settings within Bing Webmaster tools you could add a crawl-delay: directive to your robots.txt file. Bing supports whole number values ranging from 1 – 20. Each number maps to the length in seconds of time slices in which we divide a 24-hour crawl cycle. In this context, the value 1 means you allow us a ...



Only top voted, non community-wiki answers of a minimum length are eligible