A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

learn more… | top users | synonyms (7)

4
votes
2answers
37 views

Should we still care about ajax-crawling specification?

It seems google crawlers run js. But documentations of ajax-crawling techniques are still available on Google Developers site. Are they still valid and necessary to be followed?
1
vote
1answer
92 views

How to block the most popular spider crawlers via robots.txt?

I want do disallow my website from indexing via robots.txt by MSN/Bing, Yahoo, Ask Jeeves, Baidu and Yandex spider bots. I want to disallow content and media (images, videos) crawlers. The reason ...
1
vote
1answer
52 views

Website being bombarded by non-existent URLs

I noticed a newly registered domain is being hit by non-existent URLs like /pithy/597/47363-1117.doc /pithy/597/47363-1117.doc /2015/150728.html etc... It's being hit for thousands per day which ...
2
votes
2answers
129 views

Disallow Crawling of All Search Pages

Using robots.txt I am attempting to stop all crawling of search URLs Disallow: /rest_of_url/search&tour* Above is what I am using. Our URL looks like the following for all search results. ...
3
votes
1answer
26 views

Prevent bing from crawling thousands of essentially identical pages?

I have a web page with a dozen tables of data on it, each with half a dozen columns. Every table can be sorted by a column by clicking on the relevant header, and these get appended to the ...
0
votes
0answers
36 views

Baidu requesting malformed URLs - my fault or theirs?

I often get lines like this in my error log: 2015/11/11 12:15:58 [error] 10927#0: *167859 open() "/var/www/sites/mysite/http/https:/mysub.domain.com/filename.jpg" failed (2: No such file or ...
0
votes
0answers
12 views

How to gracefully and temporarily stop search engines from crawling a website [duplicate]

I run into these moments every now and then where I want to make a major update to my website but then I feel I just cant do it because search engines crawl it at that time. Is there a way I can ...
3
votes
2answers
190 views

Will Googlebot have trouble crawling a site if we have such international redirects based on location and cookie?

So, we have an international site with multiple subdomains for each language. We have hreflang markup in place (+ sitemaps), but we also detect where users come from and redirect them to appropriate ...
1
vote
0answers
57 views

Mysterious visits to privileged URLs from anonymous user/bot

I maintain a Rails app. Every page except login and "forgot password" requires user authentication. Yesterday I noticed entries like this one in the access logs: 54.209.60.63 - - ...
0
votes
0answers
9 views

How to make sub-directories appear in Google Search Engine? [duplicate]

I want to know how to effectively SEO manage my personal websites. For example, I have http://www.janedoe.com/ and in the meta data I have: <meta name="keywords" content="jane doe"> along with ...
0
votes
1answer
94 views

Using robots.txt to block sessionID URLs

I've got a problem with Googlebot literally crawling millions of URLs that are the same, because ";jsessionid" keeps getting inserted into the URL (something I cannot change due to the work ...
1
vote
0answers
36 views

Google not returning an international version of the page

I run a website that duplicates some content across international editions. These are differentiated by the country codes e.g. /uk/folder/article1/ /au/folder/article1/ The UK version is ...
9
votes
2answers
303 views

Should we drop AJAX crawling scheme?

So now Google has deprecated AJAX crawling scheme. They say not to bother implementing it in new websites, because it's no longer needed since Googlebot has now no problem watching dynamic content. ...
9
votes
3answers
994 views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
0
votes
1answer
51 views

Can Searchbots Find Web Pages That Aren't Linked To? [duplicate]

I have read this which claims that the listed search engines are not able to show the unlinked doc&XML in its results. My questions are Have any one else experimented like this? This information ...
0
votes
0answers
23 views

Google INDEED crawls ajax? [duplicate]

I don't understand how this didn't make a bigger BOOM online. It is well known in the industry that google does not index AJAX (just google-search "google crawls ajax") and see dozens of articles ...
0
votes
1answer
25 views

Does disavow solve the problem of removing unnatural links in google search?

I have created the site and it got hacked by spammer. now there many unnatural links in it if I disavow it the whole domain will it solve my problem what. Also I have two version of domain ...
0
votes
1answer
18 views

Issue in webmaster tool

I have created a site with domain name https://www.example.com but the site had many problems so I created a new fresh site and removed the previous one. This time I chose the domain name as ...
1
vote
0answers
13 views

Google displaying wrong site title and snippet [duplicate]

I am having an issue where a website that displays the incorrect site title and snippet in Google Search. From the content it is clear that some spammer has infected the site, but I cannot figure out ...
1
vote
1answer
27 views

Webmaster Showing Desktop Pages in Mobile Usability errors

I have same URL for both desktop and mobile content, but changes their format according to User-agent. Google search console is showing error as "Touch elements too close" for some pages which are ...
1
vote
0answers
28 views

Page accessible but googlebots return 404 error [closed]

I'm working on a project which has Codeigniter integrated with WordPress. My website works fine, all links are accessible. When google crawls my website pages then only for wordpress pages it gives ...
4
votes
3answers
63 views

Securing images from browsers and humans

Particular parts of a website I'm working on will have pictures not intended for the general public. It is already behind a password protection and the page has no-index no-follow rules. I'm concerned ...
3
votes
2answers
909 views

How to identify if IP address is really google's IP

From what I learned in my server logs, IP addresses starting with 64.249 belong to google. When I do a host lookup on my computer for google I receive 11 different IP addresses all starting with ...
0
votes
0answers
20 views

Seo-wise way to remove a portion of website [duplicate]

I have a website with 5000+ products driven from a MySQL database, so far so good. My links are friendly structured; http://www.example.com/brand/product/ About a year ago, I decide to add reviews ...
0
votes
2answers
142 views

How to allow Googlebot and Bingbot to crawl mysite?

Recently I got email message from Google that Googlebot cant access your site. I tried to look for some articles on the internet about this issue and most them points its because of robots.txt file ...
1
vote
1answer
25 views

Crawl-error: Google searches for foo.tld/www.foo.tld

Google Webmaster Tools is reporting strange crawl errors. It reports that www.example.com can't be found, yet the domain is valid and is an exact match of our clients domain. A click on the error ...
6
votes
1answer
115 views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
0
votes
1answer
53 views

Screaming Frog displaying deleted subdomain

In Screaming Frog, when I do a basic crawl of one of my website's in the 'Internal' tab it shows a sub-domain at the very top. This sub-domain was one I had previously created, but I removed it quite ...
0
votes
2answers
68 views

How do search engines behave when encountering noindex?

I have a question about the behaviour of search engines when they encounter meta name="robots" content="noindex" If I use the meta tag format of robots, I know that search engines will not index the ...
0
votes
2answers
172 views

Should sitemap contain URL for all categories of Product?

I am having open cart eCommerce website. Here I need to add one product under multiple Categories. By default what opencart produces SITEMAP.xml, it includes URL for all the categroies for that ...
0
votes
0answers
23 views

Fetch as Google AngularJS Duplicate Entries

I have pages built in AngularJS that when I fetch as Google in WMT or when I switch my user-agent to Googlebot using a browser extension, duplicate entries are displayed. For example, if there is an ...
0
votes
0answers
198 views

redirecting bots and crawlers to another but not human via htaccess

I would to apply this diagram via htaccess I tried a lots of codes but I failed every time so I need to redirecting bots and crawlers especially from facebook via htaccess regards
1
vote
2answers
47 views

How to control old indexed page crawling rate

I have a site where user can create their profile page with their specific URL. Each specific URL will be available to public which means that it will also be ready for search engines too to crawl. ...
1
vote
1answer
37 views

How do canonical tags work when you have multiple versions of a website?

I apologize for how vague the question sounds; I was not able to write the complete question in the title because I ran out of characters. Before the duplicate monsters close the question, please read ...
0
votes
2answers
44 views

SEO: How to stop specific sections of a page from being crawled? [duplicate]

I would like to display customer reviews from another website (with their agreement) own my own site for certain product pages. Using these 3rd party reviews would be helpful for users, though ...
0
votes
0answers
13 views

Would use of IFrame in my ecommerce site affect SEO ranking? [duplicate]

I Use an Iframe to show product summary, which is saved HTML Data from ckEditor. (As the summary have it's own html and styles I can't use it in product page directly). <iframe ...
0
votes
0answers
103 views

Google Bot Crawler Issue with Codeigniter URLS

I received 6000+ crawler errors in my webmaster tools account. following is the one of the URL that identified as problematic URL. ...
3
votes
1answer
164 views

Does Google index images declared using 'srcset'?

As of Chrome 40, the srcset attribute is supported, but will Google index the images within it?
0
votes
0answers
12 views

robots.txt that allows only Google and Bing [duplicate]

Is it possible to create a robots.txt file that disallows everything except for Google and Bing? Something like: User-agent: * Disallow: / User-agent: Googlebot Allow: / User-agent: Googlebot-News ...
1
vote
1answer
80 views

How to keep friendly URLs consistent?

I downloaded a site with a crawler script (HTTrack) and now have a few hundred HTML files that need to be edited and re-deployed. The original site ran on a combination of Drupal and a lesser known, ...
1
vote
1answer
44 views

What is this bot in server logs?

Noticed this bot in the top ten bots in the server logs: dwbot Coming from the United States. Does any one have any idea what it is?
0
votes
1answer
33 views

Updated and published website - changes not taking effect in search results?

Question Background: I've recently updated my website and republished it using Azure. This is fine and the changes have taken effect when I open and view the site from a Google search. This is the ...
0
votes
1answer
98 views

Trick to index infinite scroll content on Google?

I have a Question & Answer website, sort of like stackoverflow. When you open it, it displays 20 latest questions. When scroll to bottom of page, and it fetches 20 more, and so on, infinitely. ...
2
votes
1answer
624 views

Does Bing execute Javascript when crawling and indexing web pages (like Google)?

Some content on some web pages is only displayed with Javascript when a button is clicked (for example). We know that Google executes Javascript and is (very likely) to find this content, but what ...
1
vote
1answer
122 views

What are proper process to disallow a site from google crawl

I have done google and get the two solution for disallow whole a site from google crawl. 1: User-agent: * Disallow: 2: User-agent: * Disallow: / Now can anyone tell me which proper ...
1
vote
1answer
274 views

For how long does the google bot wait after document ready before taking a snapshot of the webpage?

As the google bot lets the JS do its work on the page while crawling, for how long does it wait to let the javascript execute after document ready ? Suppose I have an AJAX response that takes longer ...
0
votes
4answers
101 views

What is the best way to detect if the traffic is legit or (bot) not?

We are planning to start program something similar to youtube where users get paid as per views. But problem we are facing is that people will game the system and send traffic using botnets to gain ...
3
votes
1answer
125 views

What determines the frequency the Wayback Machine crawls one's website?

I understand archive.org takes its website list to crawl from Alexa, but I don't understand how it decides the snapshot frequencies for each website. We can see some websites are crawled multiple ...
0
votes
0answers
28 views

Crawl Errors - ?2016-jun-764.html

In the past week, after having no crawl errors on my website at all, I'm now getting the following showing up in Google Webmaster Tools after some of the pages: [fullURL]?2016-jun-764.html The date ...
2
votes
1answer
129 views

Does e.preventDefault on link prevent the page from being crawled?

I have a couple of links in navigation on homepage that lead to new pages. When clicked on link, i want to prevent going on that page (e.preventDefault()), and instead load that page in popup with ...