Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
3
votes
3answers
69 views

How to properly tell Google they have no right to access the download. What status do I use?

I'm looking for the SEO way to tell Google not to download pictures. I have a script that generates one of two things based on who accesses the site: If an unauthorized robot accesses the site, they ...
1
vote
0answers
15 views

Limit search engines to indexing just one domain when multiple domains share the same web server and document root

I have multiple domains listening on same ports and sharing same document root. This is a limitation of my hosting provider that lets me have multiple domains for my GSuite mail routing without using ...
1
vote
1answer
24 views

SEO for blog with async content

I have a blog website and I want to perform Search Engine Optimization on it. My website, which is built in Angular 8, takes part of the URL route and uses that to search for the blog. For example, ...
3
votes
1answer
42 views

How to change Google crawl rate in new Google search console?

Google has now completely removed the old search console (https://searchengineland.com/the-old-google-search-console-is-no-longer-available-321650) where you used to be able to request a custom crawl ...
2
votes
1answer
53 views

Can I turn off my website at certain times to save money and schedule when Googlebot crawls to maintain SEO?

Is it possible to schedule Google crawlers' crawling at a time when my website it turned on? I host my website with Google's cloud product: Google Compute. To save costs I am thinking of turning ...
2
votes
2answers
8k views

Google keeps crawling pages that don't exist

I made a simple new website which went live about a month and a half ago (replacing the last one). The site contains new pages, like for example the navigation markup: <div class="header"> &...
1
vote
0answers
22 views

Web page crawling by Google for SEO

On Our website we have an angular component which have multiple tabs and loads all the data (of all the tabs) at once. As per our new requirement to increase the 1st load speed, we will be loading ...
3
votes
1answer
74 views

Is it problematic to include both decoded and encoded versions of the same directives in robots.txt?

This question is a follow up to this one: Should one hide RTL encoded URLs in robots.txt or not? Is it problematic to include both decoded and encoded versions of the same directives in robots.txt, ...
2
votes
1answer
12 views

Override robots.txt exclusion to allow Mediapartners-Google for AdSense

I use robots.txt to block all robots.txt compliant bots from a directory like this: User-agent: * Disallow: /sub/* How can I override this so to allow Mediapartners-Google (AdSense) to /sub/?
2
votes
1answer
44 views

Will this robots.txt file work? [closed]

User-agent: * Allow: /sitemapindex.xml Allow: /post_google_news.xml Allow: /news/ Allow: /category/news/ Disallow: / I want to exclude everything from being indexed by Google except for example.com/...
10
votes
4answers
3k views

Can the files robots.txt and sitemap.xml be dynamic through an .htaccess redirect?

I have a multilanguage and multidomain site. It runs through a unique CMS installation (Drupal), so I have a single root directory. So if I have a static robots.txt, there I only can show the files ...
3
votes
0answers
38 views

Can it be destructive to disallow all MediaWiki:Special pages in robots.txt?

I consider preventing indexation of all of MediaWiki 1.33.0's Special pages; (In Hebrew, "מיוחד" means "special"): Disallow: /מיוחד:* Disallow: /index.php?title=מיוחד:* Disallow: /index.php%3Ftitle%...
2
votes
1answer
3k views

In robots.txt, why would there be “Allow: /*” below a Disallow rule?

I want to allow full access to all Google I am specially confused about Allow: /* below Disallow: # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: ...
3
votes
1answer
48 views

Do search engines crawl JavaScript generated pages?

I'm considering building a new website using react.js and am doing quite a bit of research before embarking into anything. I have found quite a few articles stating that websites that have content ...
2
votes
3answers
55 views

Stop googlebot crawling URL more than once?

I have a site that usually creates a few thousand pages a day, which don't change after they have been created. Recently my dedicated server has crashed due to googlebot crawling the site too often. ...
18
votes
3answers
16k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
-1
votes
3answers
47 views

How to remove bulk urls from google index

I have URLs indexed by Google and I want to remove them from indexing. I found that I can use bulk remove option by providing a CSV file including all indexed URLs to Google Search Console. I have ...
4
votes
3answers
935 views

How can I make Google crawl and index multiple pages?

There are several pages on the web that are not under my direct control but contain content about a product I'm involved with. I'd like make Google robots crawl and index those pages, is there a ...
3
votes
2answers
2k views

Can Angular project be indexed in Google without Angular Universal?

I'm working on an Ionic PWAprogressive web app (PWA) and forthere are reasons iI cannot use Angular Universal. Is there any way to index with success inget Google to successfully index the site ...
0
votes
0answers
9 views

Will having a sitemap ensure pages served by AJAX requests be crawled?

I have a website, where I publish articles, that I have started just 2 weeks ago. I want to keep the pages as clean as possible and load more content (links to more articles) by AJAX requests upon ...
1
vote
1answer
23 views

Disallowed “Search” results with robots.txt and Sessions dropped

I've started working on our website and I've found millions of "Search" URL's which I don't think should be getting crawled & indexed (e.g. .../search/?q=brown&prefn1=brand&prefv1=C.P. ...
1
vote
2answers
33 views

Does decerasing crawl depth affect SEO?

We are planning to relaunch our website and I have noticed that certain pages that receive good visibility are just too deep (4 and above). I'm planning to make these pages easier to access and reduce ...
2
votes
0answers
713 views

URL will be indexed only if certain conditions are met

I've got a few pages of my site that are coming up in G Webmaster Tools with errors. The pages are fully working, not restricted by noindex or robots.txt file. The pages don'r have a lot of content, ...
0
votes
1answer
22 views

Will human visitors visit through Amazon AWS?

My website gets visited at least once a day or so from IPs that give Amazon AWS domains for reverse DNS requests. The User-Agent strings are those of browsers, not bots. Could these be real people, or ...
10
votes
1answer
1k views

How does “Noindex:” in robots.txt work?

I ran across this article in my SEO news today. It seems to imply that you can you use Noindex: directives in addition to the standard Disallow: directives in robots.txt. Disallow: /page-one.html ...
1
vote
1answer
27 views

Form-locked PDFs in JSON-LD

I'm setting up downloads functionality with form-locked PDFs - available after the user fills-out a form. I want the PDFs to be indexable by search engines though. Im trying to create a JSON-LD markup,...
2
votes
4answers
310 views

Secure way to share unfinished web-page with a customer?

What are the options to share unfinished web-page with a customer without allowing third-party to see the work in progress when developer has control over the web-server? I could come up with ...
2
votes
1answer
66 views

Will adding a random letter to the end of the URL prevent content from being scraped and stolen while allowing SEO?

I have built a website which is a directory of venues / places in a country and client has asked me to make URLs more SEO friendly. There has been some misunderstanding in communication and I have ...
1
vote
1answer
146 views

Clients from Microsoft IP address ranges seem to be spoofing Bingbot

I routinely get hits from Microsoft-owned IP addresses in the 13.x.y.z ranges of AS8075 (in these ranges) that present as Bingbot in the User Agent, but these addresses are listed on several block ...
8
votes
5answers
4k views

Deny access to Archive.is

I would like to deny archive.is from having access to my website. (I do not want this website to cache mine without my consent). Do you know if it is possible?
7
votes
1answer
810 views

Googlebot encountering JavaScript errors due to rendering outdated HTML files with newer JS files

We recently encountered an issue where we added a JavaScript variable (e.g. var GLOBAL_VAR = true;) to an HTML page (e.g. /Search/Index) and updated the separate JS file (e.g. /Scripts/search/index.js)...
5
votes
1answer
164 views

Does Google return to visit a page with 404 Error?

If a page indexed in Google shows a 404 Error, does Google return to visit it again? In other words, if a page that was indexed in Google continue to show the 404 Error for several years, will ...
0
votes
0answers
17 views

If all web pages are interchangeably linked by taxonomy (categories), do I still need a sitemap for SEO?

I have a mediaWiki web site with about 350 web pages. If all web pages, without exception, are interchangeably linked by taxonomy (categories), and moreover, there are also other indicators for web ...
0
votes
0answers
31 views

Google Search Console - Verification Failed

I am trying to add my website to Google Search Console but failed, it returns The connection to your server timed out The file is there, I can open it on the normal browser, all meta tag set to ...
1
vote
3answers
553 views

SEO impact of onclick handlers in place of links

I have a question similar to the one posted here. I'm building a React application which has very strict SEO requirements, so I'm wondering if the following approach is an issue: For all of our ...
1
vote
0answers
6 views

Should we apply “noreferrer” on “Read More” or “View All”?

I experienced years back a search engine crawler issue that Google started showing "Read More" as title of the page in it's search results. Because that page was linked through "read more" link on new ...
2
votes
1answer
38 views

Will blocking bots other than those from the major search engines affect SEO?

I read a lot about this but I'm a bit confused since it looks like it's not a black or white thing. In some posts you can read that the more backlinks the better and other ones claiming the opposite. ...
1
vote
1answer
25 views

We are blocking bots, crawlers, spiders and scan tools in our servers. Can this affect DA (domain authority)?

I didn't know the concept of domain/page authority in SEO until 1 hour ago. We have been blocking bots, crawlers, spiders and scanners in general directly from Apache through ModSecurity rules (custom ...
2
votes
2answers
42 views

Disallow URLs with an ID from being crawled

I have a website with around 100K URLs. I want to disallow crawling for all URLs that have an ID with this pattern: www.example.com/node/sport/category/id but not those without the ID: www.example....
6
votes
1answer
3k views

How can I stop Google from indexing embedded iframes?

I have iframe widgets of external sources embedded on my website. For some reason Google seems to be indexing the URLs of the iframes and showing 404 errors for such URLs in the Crawl Section of ...
0
votes
0answers
29 views

Invalid traffic coming from Sogou

I have not indexed my website in Sogou. I have only very less visitors. But i am getting traffic from Sogou daily. The search keyword is "site:mywebsitename" and is from China. Just visiting main page....
0
votes
0answers
14 views

Crawler web form submissions with “Crawler typing text” in field

We have a form on our public website that users can use to request an account from us. Just today we had hundreds of form submissions that said "crawler typing text..." in the email and name field, ...
2
votes
3answers
2k views

Do other search engines support Google's “hash bang” syntax for crawling AJAX applications?

Google has a really nice document explaining how web developers can get Google to crawl non-AJAX versions of their web applications to make it possible for Google to index AJAX-heavy websites. Do ...
2
votes
1answer
40 views

Do search engines crawl links with query strings?

If I were to have a url like https://example.com/books, that contained several links to urls such as: https://example.com/books?id=foo https://example.com/books?id=bar https://example.com/books?id=...
1
vote
2answers
38 views

How exactly does GoogleBot discover internal search result pages?

I know internal search result pages can be indexed sometimes but what I don't get is how it is crawled? Crawler only crawls links. Does that mean that someone would have to link to an internal search ...
1
vote
1answer
27 views

Visitors using multiple IPs and determining if they are human

I'm seeing some obvious bot/script traffic that is bloating our Google Analytics(GA) data. I figure the traffic is using something that loads JS, CSS, etc. which is why it's affecting GA. So I put in ...
4
votes
5answers
3k views

Google Bots Crawling my Site daily is this good or a bad thing?

Currently my site is being crawled by the Googlebot on a daily basis. Is a daily crawl from the Googlebot a good thing or bad thing?
1
vote
2answers
231 views

Am I getting DDOS from crawlers?

My website is currently getting hit with spamming bots (Example: 66.249.73.*) that are causing high CPU usage. Is it common for Google/Bing to crawl a website 10/sec? I have done reverse look ups on ...
0
votes
1answer
15 views

SEO consequences for using IIS Rewrite rule to redirect to static content for crawlers only

I have an AngularJS site that's pretty slow which is causing the SEO to be rather poor. Not to mention, the metadata doesn't render correctly. To fix this, my thought was to use IIS rewrite rules to ...
1
vote
1answer
47 views

Robot.Txt File interpretation problem?

I have a Joomla/Shopware For an eCommerce website, but I see some problems and I think google didnt crawl pages: {block name="frontend_robots_txt_user_agent"} User-agent: * {/block} {block name="...