Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
2
votes
0answers
361 views

URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
0
votes
1answer
20 views

Will human visitors visit through Amazon AWS?

My website gets visited at least once a day or so from IPs that give Amazon AWS domains for reverse DNS requests. The User-Agent strings are those of browsers, not bots. Could these be real people, or ...
10
votes
1answer
1k views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
1
vote
1answer
24 views

Form-locked PDFs in JSON-LD

I'm setting up downloads functionality with form-locked PDFs - available after the user fills-out a form. I want the PDFs to be indexable by search engines though. Im trying to create a JSON-LD markup,...
2
votes
4answers
305 views

Secure way to share unfinished web-page with a customer?

What are the options to share unfinished web-page with a customer without allowing third-party to see the work in progress when developer has control over the web-server? I could come up with ...
2
votes
1answer
62 views

Will adding a random letter to the end of the URL prevent content from being scraped and stolen while allowing SEO?

I have built a website which is a directory of venues / places in a country and client has asked me to make URLs more SEO friendly. There has been some misunderstanding in communication and I have ...
1
vote
1answer
100 views

Clients from Microsoft IP address ranges seem to be spoofing Bingbot

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block ...
8
votes
5answers
4k views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
6
votes
1answer
782 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
5
votes
1answer
157 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
0
votes
0answers
17 views

If all web pages are interchangeably linked by taxonomy (categories), do I still need a sitemap for SEO?

I have a mediaWiki web site with about 350 web pages. If all web pages, without exception, are interchangeably linked by taxonomy (categories), and moreover, there are also other indicators for web ...
0
votes
0answers
16 views

Google Search Console - Verification Failed

I am trying to add my website to Google Search Console but failed, it returns The connection to your server timed out The file is there, I can open it on the normal browser, all meta tag set to ...
1
vote
3answers
444 views

SEO impact of onclick handlers in place of links

I have a question similar to the one posted here. I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue: For all of our ...
1
vote
0answers
5 views

Should we apply “noreferrer” on “Read More” or “View All”?

I experienced years back a search engine crawler issue that Google started showing "Read More" as title of the page in it's search results. Because that page was linked through "read more" link on new ...
2
votes
1answer
37 views

Will blocking bots other than those from the major search engines affect SEO?

I read a lot about this but I'm a bit confused since it looks like it's not a black or white thing. In some posts you can read that the more backlinks the better and other ones claiming the opposite. ...
1
vote
1answer
24 views

We are blocking bots, crawlers, spiders and scan tools in our servers. Can this affect DA (domain authority)?

I didn't know the concept of domain/page authority in SEO until 1 hour ago. We have been blocking bots, crawlers, spiders and scanners in general directly from Apache through ModSecurity rules (custom ...
2
votes
2answers
39 views

Disallow URLs with an ID from being crawled

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example....
6
votes
1answer
3k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
0
votes
0answers
20 views

Invalid traffic coming from Sogou

I have not indexed my website in Sogou. I have only very less visitors. But i am getting traffic from Sogou daily. The search keyword is "site:mywebsitename" and is from China. Just visiting main page....
0
votes
0answers
14 views

Crawler web form submissions with “Crawler typing text” in field

We have a form on our public website that users can use to request an account from us. Just today we had hundreds of form submissions that said "crawler typing text..." in the email and name field, ...
1
vote
3answers
2k views

Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?

Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites. Do ...
2
votes
1answer
37 views

Do search engines crawl links with query strings?

If I were to have a url like https://example.com/books, that contained several links to urls such as: https://example.com/books?id=foo https://example.com/books?id=bar https://example.com/books?id=...
1
vote
2answers
37 views

How exactly does GoogleBot discover internal search result pages?

I know internal search result pages can be indexed sometimes but what I don't get is how it is crawled? Crawler only crawls links. Does that mean that someone would have to link to an internal search ...
1
vote
1answer
26 views

Visitors using multiple IPs and determining if they are human

I'm seeing some obvious bot/script traffic that is bloating our Google Analytics(GA) data. I figure the traffic is using something that loads JS, CSS, etc. which is why it's affecting GA. So I put in ...
4
votes
5answers
3k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
1
vote
2answers
223 views

Am I getting DDOS from crawlers?

My website is currently getting hit with spamming bots (Example: 66.249.73.*) that are causing high CPU usage. Is it common for Google/Bing to crawl a website 10/sec? I have done reverse look ups on ...
0
votes
1answer
15 views

SEO consequences for using IIS Rewrite rule to redirect to static content for crawlers only

I have an AngularJS site that's pretty slow which is causing the SEO to be rather poor. Not to mention, the metadata doesn't render correctly. To fix this, my thought was to use IIS rewrite rules to ...
1
vote
1answer
43 views

Robot.Txt File interpretation problem?

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages: {block name="frontend_robots_txt_user_agent"} User-agent: * {/block} {block name="...
0
votes
1answer
32 views

Can search engines index webpages with path after “.html”?

Are search engines able to index a webpage which has a path after .html? Example: http://example.com/about.html/abc/123 Is it an acceptable implementation for SEO?
2
votes
1answer
45 views

What are the potential side effects of the “nosnippet” robots meta tag?

Just recently I noticed our Search Console CTR drop from 4% to 2%. I noticed that we had an ever increasing number of impressions, but despite the increase in impressions, our clicks had decreased. ...
0
votes
1answer
26 views

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory

MediaWiki: Hiding drafts from Search engines without changing any file in the website directory, Is this possible or I must change robots.txt for drafts starting as draft:myDraft?
0
votes
1answer
96 views

Pages uploaded to WordPress as media are causing mobile usability errors in Google Search Console [duplicate]

the new search console is pointing out usability errors on mobile devices on WordPress upload pages, that is, pages where I can not change font size, viewport window and things like that, how do I ...
1
vote
1answer
50 views

Do I need to disallow mediapartners-google in robots.txt for SEO?

I found that most of the websites disallow mediapartners-google in robots.txt. Why do I need to do block mediapartners-google? Can blocking it help withp better SEO or does it not matter for SEO?
2
votes
0answers
27 views

Should a page with adsense be viewed at least once for adsense crawler to crawl?

I use plugins to insert Adsense codes on my WordPress website. The plugin has an option to disable ads for the admin user. Hence, I will not be seeing ads when creating and publishing/reviewing posts. ...
2
votes
2answers
31 views

Thousands of bot hits for retired domain

In May of last year we retired a domain and redirected it at a new one. This was done to consolidate two sites from a company merger into one site. We shutdown the CMS for site1.com and pointed its ...
1
vote
1answer
25 views

How do I access an entire website list that is continually generated when scrolling? [closed]

I am trying to acquire all the information from a list of songs on ranker, however as I keep scrolling down new songs appear in an ever growing list, as the name suggests its a voting website so I am ...
12
votes
1answer
19k views

Why does baidu crawl my site like crazy

When i'm checking my apache log I can see that baidu has been crawling my website 10 times a day for the last 2 weeks. It's not that i care much about it but I'm really curious about why he's doing ...
2
votes
1answer
472 views

Google intermittently fails to download CSS and JS from CloudFront CDN

We noticed a lot of "Mobile Usability" errors in the Search Console. When I clicked on them and chose "Check live version" option - it always said the page was mobile friendly. However, the mobile ...
4
votes
1answer
55 views

Robots.txt allow directory root while disallowing all deeper URLs

I would like to allow a path and disallow everything else in the directory. User-Agent: * Allow: /example/ Disallow: /example/* Will this work? If not what would be a better way of doing this?
0
votes
2answers
34 views

What happens if we forgot to write meta tags?

If we forgot to write meta tags, especially meta title and meta description, how will search engines crawl and index?
3
votes
9answers
13k views

Is this Google proxy a fake crawler: google-proxy-66-249-81-131.google.com?

Recently I discover that some variants of a google proxy visits my sites. I doubt these are legal Google crawlers because these crawlers are NOT always behind a proxy (like the hostname describes) and ...
2
votes
1answer
508 views

Fixing website title on Bing

I have recently created my own portfolio website, and the title meta tags specify the title of my front page as: "Thomas Russell's Portfolio and Blog", but when I search for "Thomas Russell's ...
0
votes
0answers
37 views

Google is marking as duplicate my unique URLs powered by Angular Universal

I have a problem in the search console, and google is marking my unique urls as duplicate The application is build with Angular Universal and is rendered in the server Note: This site is only ...
3
votes
2answers
390 views

Google Search Console: 404 errors on existing pages

There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
2
votes
1answer
46 views

Is blocking googlebot from accessing portions of the site with robots.txt penalized by Google?

I was reading this question. Let's say i have a div with something useful for users but i don't want it to be considered by Google. If i put this on an iframe and block the googlebot with robots.txt ...
6
votes
1answer
167 views

AdSense crawler access fails because the bot adds a slash to the login URL

I'm trying to get a site verified for AdSense. Since I want to display ads in authenticated pages too, I'm setting up Crawler Access to give a username and password to Google crawler. In this site, ...
1
vote
2answers
32 views

How to preserve ALL website's URLS when transferring to a new CMS as well hosting

I am in a position where I have to figure out how to preserve all of the website's pages URLS when transferring it to a new CMS, perhaps potentially Joomla (hasn't been decided yet). New hosting is ...
0
votes
2answers
46 views

Google keeps crawling old site, ignores new (& better) one

Searching 'Dragon Notes' in Google predominantly yields (1) rather than (2) - even for images; (2)'s almost non-existent. Differences: (2) is more up-to-date, has a better sitemap and source code (2) ...
1
vote
0answers
18 views

How to deal with great number of URLs that will not be in anchor and should be known only for friendly crawlers?

I have a hypothetical page, that contains pages like: example.com/see/cat example.com/see/mouse example.com/see/<basically anything> I can have thousands of such pages. The number of pages ...
2
votes
1answer
142 views

Two pages have same title, same description, different content. Are they duplicates?

We have release our website from our mobile app in a very short amount of time. The website is quite good, but not at all optimized for SEO. For example, there is absolutely no care for meta tags. All ...