Hot answers tagged web-crawlers
You can't. You have to wait for Google to re-crawl the page on whatever schedule it has determined they will crawl that page. It could be every hour, it could be once a month. Unfortunately you have no control over that.
For some reason Google seems to be indexing the URLs of the iframes As covered here, Google will indeed try to crawl and associate framed content with the page containing the frames. Would rel="nofollow" work for iframes? Within the header section of the iframe page (not the parent page containing the iframe), use: <meta name="robots" ...
Yes. Search engines and any crawler that is coded properly can crawl pages with parameters and index them as separate pages. The fact that the crawler you referenced/used cannot handle this is an indication that I would not use this tool again. I often recommend using http://www.screamingfrog.co.uk/seo-spider/ if you can access your site directly. This tool ...
Only top voted, non community-wiki answers of a minimum length are eligible