A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

0
votes
1answer
21 views

Screaming Frog displaying deleted subdomain

In Screaming Frog, when I do a basic crawl of one of my website's in the 'Internal' tab it shows a sub-domain at the very top. This sub-domain was one I had previously created, but I removed it quite ...
0
votes
2answers
45 views

How do search engines behave when encountering noindex?

I have a question about the behaviour of search engines when they encounter meta name="robots" content="noindex" If I use the meta tag format of robots, I know that search engines will not index the ...
0
votes
2answers
69 views

Should sitemap contain URL for all categories of Product?

I am having open cart eCommerce website. Here I need to add one product under multiple Categories. By default what opencart produces SITEMAP.xml, it includes URL for all the categroies for that ...
0
votes
0answers
13 views

Fetch as Google AngularJS Duplicate Entries

I have pages built in AngularJS that when I fetch as Google in WMT or when I switch my user-agent to Googlebot using a browser extension, duplicate entries are displayed. For example, if there is an ...
0
votes
0answers
49 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
0
votes
1answer
23 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
1
vote
1answer
23 views

How do canonical tags work when you have multiple versions of a website?

I apologize for how vague the question sounds; I was not able to write the complete question in the title because I ran out of characters. Before the duplicate monsters close the question, please read ...
0
votes
2answers
29 views

SEO: How to stop specific sections of a page from being crawled? [duplicate]

I would like to display customer reviews from another website (with their agreement) own my own site for certain product pages. Using these 3rd party reviews would be helpful for users, though ...
0
votes
0answers
12 views

Would use of IFrame in my ecommerce site affect SEO ranking? [duplicate]

I Use an Iframe to show product summary, which is saved HTML Data from ckEditor. (As the summary have it's own html and styles I can't use it in product page directly). <iframe ...
0
votes
0answers
18 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
2
votes
2answers
49 views

Does Google index images declared using 'srcset'?

As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
0
votes
0answers
12 views

robots.txt that allows only Google and Bing [duplicate]

Is it possible to create a robots.txt file that disallows everything except for Google and Bing? Something like: User-agent: * Disallow: / User-agent: Googlebot Allow: / User-agent: Googlebot-News ...
1
vote
1answer
43 views

How to keep friendly URLs consistent?

I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed. The original site ran on a combination of Drupal and a lesser known, ...
1
vote
1answer
31 views

What is this bot in server logs?

Noticed this bot in the top ten bots in the server logs: dwbot Coming from the United States. Does any one have any idea what it is?
0
votes
1answer
23 views

Updated and published website - changes not taking effect in search results?

Question Background: I've recently updated my website and republished it using Azure. This is fine and the changes have taken effect when I open and view the site from a Google search. This is the ...
0
votes
1answer
45 views

Trick to index infinite scroll content on Google?

I have a Question & Answer website, sort of like stackoverflow. When you open it, it displays 20 latest questions. When scroll to bottom of page, and it fetches 20 more, and so on, infinitely. ...
2
votes
1answer
101 views

Does Bing execute Javascript when crawling and indexing web pages (like Google)?

Some content on some web pages is only displayed with Javascript when a button is clicked (for example). We know that Google executes Javascript and is (very likely) to find this content, but what ...
1
vote
1answer
117 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
1
vote
1answer
87 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
0
votes
2answers
44 views

What is the best way to detect if the traffic is legit or (bot) not?

We are planning to start program something similar to youtube where users get paid as per views. But problem we are facing is that people will game the system and send traffic using botnets to gain ...
3
votes
1answer
53 views

What determines the frequency the Wayback Machine crawls one's website?

I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
0
votes
0answers
22 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
2
votes
1answer
51 views

Does e.preventDefault on link prevent the page from being crawled?

I have a couple of links in navigation on homepage that lead to new pages. When clicked on link, i want to prevent going on that page (e.preventDefault()), and instead load that page in popup with ...
0
votes
1answer
40 views

Format of _escaped_fragment_ for complex states

I understand that for a url such as www.example.com/about, Google's crawler will look for www.example.com?escaped_fragment_=about. However, what will google use as the value for the ...
0
votes
0answers
19 views

URL Dynamically changes after first edit

We are developing a magento eCommerce website with main category pages, sub-category pages and details page along with other static pages. Our condition is if any miss-spelled URLs entered directly ...
1
vote
1answer
49 views

How to prevent search engine bots from crawling specific pages?

My website crashes my server when a certain page is crawled. I've tried implementing a robots.txt file to disallow that page and to implement a crawl delay, but web crawlers are ignoring it. Is it ...
0
votes
2answers
58 views

can google index/crawl random subdomains that aren't linked to from anywhere

Me and a friend are having a debate where she claims that every and all subdomains will get crawled/indexed by google unless you specifically tell it not to, and I'm saying that if a page isn't linked ...
0
votes
1answer
25 views

Crawl errors for pdf download links in my website

We are building a local business classified website which has about 6000 unique listingsunder huge number of categories and sub-categories. For each listing there is a details page in which we have ...
1
vote
1answer
58 views

Hiding links via AJAX on pages disallowed for indexation for SEO. Good or Bad & Why?

I have pages that I've disallowed in robots.txt and if I don't want those pages to leak link juice to external and internal links that appear on those pages. Questions: If I "hide" these links via ...
0
votes
1answer
89 views

Is it a good practice to block spam bots by IP ranges?

I'm getting too many hits on my website from Yandex. It doesn't obey robots.txt and by some way it can bypass .htaccess rules. So I'm thinking to block all Yandex IP ranges in my system firewall. Is ...
0
votes
0answers
14 views

spam referers bypass htaccess rules [duplicate]

Here is a sample code I'm using in .htaccess to block spam referers: RewriteEngine On RewriteCond %{HTTP_REFERER} ^http://.*buttons-for-your-website\.com/ [NC,OR] RewriteCond %{HTTP_REFERER} ...
2
votes
3answers
81 views

How to block a user-agent that has a spacing in its name?

I got a hit from a crawler with a user-agent called DV CRAWLER which is an abvious a spam-bot. I tried to block it in both .htacess and nginx configuration as I'm running nginx as a reverse proxy in ...
1
vote
1answer
33 views

Is there anything unusual or nefarious about private IP addresses in server logs?

A site I help take care of has just started getting crawled by somebody from 172.31.13.241 and 172.31.42.227 . According to WHOIS info, these are part of a block reserved by IANA for special purposes, ...
0
votes
1answer
39 views

How to find out which links of website posted on Twitter

I recenlty noticed many requests from Twitterbot user agent in my website's logs which I believe it doesn't crawl by itself unless someone is posting links of my website on Twitter. These posts might ...
1
vote
0answers
9 views

proper use google webmaster tools URL Parameters [duplicate]

There are some urls like http://domain.com/user/:username and http://domain.com/post/:id on my website, the username and id is the parameter from database. I try to let google crawler more efficiently ...
0
votes
2answers
54 views

Generating a list of URLs including metadata [closed]

Basically I want to do an editorial sweep on my website, I need a program to crawl my website and generate a list of URLS and their metadata. A program that produces something like a CSV would be ...
1
vote
2answers
55 views

Why does Google Analytics report that I have traffic at a calculator.php URL for my blog?

I frequently see this URL having traffic via Google Analytics although I don't have any page and with that strucuter either. URL - ...
0
votes
3answers
68 views

Robot.txt disallow *?s=

Looking on the robot file of our ,soon to be, website. I want to know what prevent the site to be crawled. Is it this line ? If not, what will it disallow ? Disallow: *?s=
1
vote
2answers
62 views

Pages with dynamically loaded content and SEO

This question is not bound to a particular web technology. If I load a page that is basically a blank shell and on page load I start getting all the content from the server with ajax requests, won't ...
0
votes
0answers
6 views

How does google discover new hosts? [duplicate]

Assume I register a new host, foobar.com, and assume that Google doesn't know about foobar.com at all. Never in the history of internet was foobar.com mentioned in any web page. The two ways I know ...
-1
votes
3answers
19 views

Is there any special tag for web master to crawler page

My page is based on HTML and I need to subscribe for google webmaster and bing so I know how to give short description about my page So that it easy to crawl my page my page is full of html code and ...
1
vote
2answers
42 views

Can I limit content scrapers by counting the number of hits from an IP?

I would like to ban aggressive scrapers that access x number of pages in an hour, say 1000. I plan to implement such a check via fail2ban and count the hits from the same ip via access logs. Should I ...
1
vote
2answers
119 views

SEO on AJAX based search site

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table. I'm having trouble finding out the correct information about crawlability ...
0
votes
0answers
7 views

Why is googlebot trying to fetch click.php? [duplicate]

I see the following in my access log: access.log.1:66.249.69.81 - - [02/Apr/2015:23:03:13 -0400] "GET /click.php?c%3Ec4C%7B%5BHG%7BbD6kc31%3E'dmjdl(..more such snipped..) HTTP/1.1" 404 19 "-" ...
8
votes
4answers
736 views

Google doesn't crawl CDN files

I've noticed that Google Webmaster Tools is reporting a lot of blocked resources in my website. Right now all the "blocked resources" are .css, .js and images (.jpg, .png) that I serve from Cloudfront ...
17
votes
2answers
2k views

Is there a difference between an empty robots.txt and no robots.txt at all?

On a webserver I now have to admnistrate I noticed that the robots.txt is empty. I wondered if there's a difference between an empty robots.txt and no file at all. Do crawlers behave differently in ...
1
vote
3answers
279 views

500 error returns website content

I'm using CloudFlare on my shared host. Recently all search engines and is-it-up sites have been getting a 500 error. I'm not sure who's to blame, my host or CloudFlare. I don't remember changing ...
0
votes
0answers
14 views

Trigger Google (re)crawl with ping [duplicate]

I have a site A with daily updated content. Right after the update, I send a webblog ping using various ping services, like http://rpc.pingomatic.com. Moments later, the updated homepage lands in the ...
4
votes
1answer
124 views

If I dynamically add a robots meta tag (via JS) will Google and other search engines recognize it?

I want to have a chunk of JS in the middle of my page insert a meta-robots element into the <head> based on some arbitrary condition. Will Google and other search engines recognize this tag the ...
2
votes
1answer
368 views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...