Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
10
votes
1answer
3k views

Googlebot submitting thousands of requests to our map locater and using up the API quota

We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map. Recently we had begun to notice that the site was ...
2
votes
0answers
251 views

Google is adding “Archive” to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
2
votes
1answer
166 views

Should I prevent search engines from indexing empty user profile pages? If so, how much content is enough for indexing?

I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available: The main user page (about me, last activity, ...) Several book ...
1
vote
2answers
112 views

Would an online queueing system for a website have an effect on google's ability to crawl the site?

If I was to implement a visitor queuing system for my website, where if visitor count is higher than n, place them in a queue. Would this have an effect on google's ability to crawl the site? Thanks
3
votes
3answers
120 views

How long before Google indexes a new (to me) domain that previously had spam and virus problems?

I registered a new ".com" domain, but when I added it to the Google Search Console I saw that there were problems with viruses and spam problems. I submitted a request for reconsideration. Google ...
1
vote
0answers
20 views

Is it bad for SEO to have a URL with nothing on it? [duplicate]

I have a WordPress site and for a very complicated reason I have to set some post type's template to empty file. Nothing. Which means I have a URL on my site (let's say: example.com/postype/the-best-...
1
vote
2answers
782 views

How to Crawl a website requires cookies for audit?

Situation: My Client's website requires cookies to access it. Users should choose (Language and country) to access the website. The problem is: Whenever I try to crawl the website using any software (...
0
votes
1answer
118 views

Can we find all backlinks to a webpage? [closed]

In Page and Brin's paper on PageRank, they say that while you are guaranteed to be able to find all links that point away from a page, the reverse is not true. My impression from reading the paper was ...
1
vote
1answer
42 views

If .htaccess is used to block my bot from accessing a particular directory, will I know this?

I'm working on a research project and I have a question. Say I would like to crawl all pages of a given site. In the case that my bot is blocked from accessing a certain portion of the site, I would ...
1
vote
0answers
265 views

Why would a bot submit a sign up form with fake info? [duplicate]

I have a sign up form where people can enter their name and address, then click on a submit button. However I am getting BOTs entering sign up details and submitting the form - or at least I presume ...
1
vote
1answer
40 views

How do we know, what is the source from which BING search indexed my webpage? - A Blog or website where my website link is placed

One of my web page with sensitive information was indexed by BING search and which in turn showing in the search results. Now that i solved by Robots meta tag in the header. However i would like to ...
3
votes
1answer
580 views

Where can I redirect exploit scanning bots?

I get a lot of those exploit scanning bots like the ones looking for a WordPress login (which I don't have) or guessing other exploitable URLs. Where is a sensible place to redirect them to via ....
1
vote
1answer
389 views

Facebot sometimes fails to parse open graph meta data, causing share failure

We maintain a WordPress-based website where Yoast plugin takes care of Open Graph meta tags generation. Recently we had a problem with likes disappearing after domain / protocol migration, but we have ...
2
votes
1answer
73 views

Why wouldn't a website apear in Google search results after robots.txt update to clean up hacked site?

I have a WordPress website which was hacked couple of days ago. I have tried to add up some security. By following couple of tutorials, I have updated my robots.txt file. But now my website is not ...
1
vote
1answer
33 views

Re-Indexing Home Page - HTML/DOM Change

(Site on WordPress) Currently, our company homepage is using images instead of divs to display our main products/solutions. As the SEO, I wanted to remove these images (which consist of mostly text) ...
4
votes
1answer
692 views

Are user agent names case-sensitive in robots.txt?

I'm blocking various bots in robots.txt and I was wondering if their names are case-sensitive. For example: User-agent: grapeshot Disallow: / Would it matter if I use grapeshot or GrapeShot?
2
votes
2answers
660 views

Could a custom crawler find unlisted web pages?

Example: A website has no sitemap.xml, no robots.txt, no index of those pages. Pages are not blocked, bots and humans have access, but they would need the URLs URL format: example.com/ofehdjtd/some-...
3
votes
1answer
432 views

Rule out third party scraping, but allow Google crawling

How to make scraping of own content through wget, httrack etc. impossible, but allow crawling through googlebot? This should be done without showing to googlebot other content, as to other user ...
0
votes
3answers
131 views

A top directory in a URL path returns a 404 error, will its subpages still be crawled?

I found a site in which one of the directories in the URL path returns 404 error, but the subpages are still live. Example URL: https://shop.edurite.com/blogs/news/summer-activities In this URL ...
1
vote
0answers
35 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...
1
vote
1answer
58 views

How should I protect “secret” links I send in Emails from being indexed by search engines?

I have noticed that bing/msnbot tries to index pages that were only ever linked to in one single email. My guess is, that some kind of browser toolbar is responsible for this. Apart from robots.txt, ...
2
votes
1answer
26 views

Should article preview pages be crawled and indexed by search engines?

I have a page called "all articles" that loads previews of articles using AJAX. Because of that, the content won't be visible to search engines, so the page would be considered "poor content". Since ...
0
votes
1answer
303 views

Does adding CDN stops Google from crawling?

I want to add Cloudflare free CDN to my site but worried about whether Google bots will be able to crawl my site later on or not.
0
votes
1answer
7k views

How block only Yandex bot

Can you show me how will look robots.txt when I block only yandex bot, allow Google bot and block Yandex bot.
1
vote
2answers
434 views

How does robots.txt work with sites in subfolders?

I have a single web host with a number of other parked domains/sites in sub-directories, like this: example.com is the primary site and root directory of the web hosting. example.com/www.example2....
2
votes
2answers
246 views

Ajax Content from Blocked Resource

I have a site built in AngularJS. Most of the dynamically-loaded content comes from a Wordpress back-end that is separate from the AngularJS site. In fact, it's a completely different domain and we do ...
1
vote
1answer
28 views

How do you properly SEO-tag an app that is only a catalogue of images?

I am making a little helper app that mostly links a lot of graphs and images. There is also some text explaining a couple of things, but more importantly that text will probably never (or very rarely) ...
2
votes
3answers
98 views

Is it good to submit our site in website submission sites?

I was wondering why don't we submit our site in website submission sites and increase our site's rank. Is is good to do this? How is this working? Is this a spam activity? What are the advantages ...
2
votes
0answers
236 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
3
votes
1answer
899 views

Checking site for meta refresh redirects

How do you check or crawl site (couple of urls) to explore existing meta refresh redirects? Screaming Frog doesn't handle them - it indicates a page with meta refresh as 200 (it is partly true), even ...
3
votes
1answer
151 views

How to check whether the infinite page scroll data is crawled and indexed by google or not?

I am having a music platform website where at a time 20 songs are visible right now then after scroll the other 20 are visible. So how can I confirm whether the songs which are coming after scrolling ...
2
votes
3answers
159 views

How can search engines find my RSS feed?

I have a small blog. Now, I have created an rss.xml. I put that in the root of the site, on the server. Should I do anything to make search engines like Google, Technorati, Yahoo! be able to find it? ...
3
votes
2answers
434 views

Google Search Console: 404 errors on existing pages

There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
1
vote
0answers
10 views

Should i create seperate page for every ad for search engine indexing purposes in my website [closed]

I am going to create a simple ad marketing web site. so I should use databases as my storing method. Now my question is If I want to show my customers ads in Google search result page(just like ...
2
votes
0answers
81 views

Which one is more SEO friendly Dynamic pages or static pages for blogging website

I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
2
votes
1answer
644 views

“Is robots.txt blocking important pages?” in Search Console regarding static folder

This is the contents of my robots.txt file: User-agent: * Disallow: /static/* Disallow: /templates/* Disallow: /translations/* Sitemap: https://xx.xxx.com/sitemap.xml However I don't know why I am ...
2
votes
1answer
132 views

Does a link rel canonical tag pointing back to the page itself cause an infinite loop that wastes crawl budget?

I assume that search engine robots crawl the whole page and then the canonical tag because pages that are self canonicalized still get indexed and their contents are still parsed. However, does a ...
2
votes
1answer
125 views

robots.txt for website hosted in a subdirectory

I have 2 website which is hosted in a shared hosting. 1st website example.com hosted in a root directory as /public_html/ 2nd website example2.com hosted in a sub-directory directory as /public_html/...
1
vote
3answers
74 views

How is noFollow enforced on sites like Quora and Facebook?

I'm curious to know how search engines like Google enforce their noFollow policy on social sites. It seems like it would be largely out of their control, especially for webpages that cannot be crawled....
5
votes
1answer
999 views

Block Yandex crawler

Our site has been behaving very strangely for the last few days, lots of time outs etc. Finally think I found the cause, the Yandex bot is crawling around 10,000 pages an hour! I need to stop it ...
1
vote
1answer
495 views

Crawler says, page not found, but browser says otherwise

I'm stumped. Google has been reporting increasing numbers of 404 errors for my website. But my site is static. Ok. Maybe they just finding more as different chunks are scanned. At the same time,...
0
votes
2answers
194 views

Spider-Trap on a GitHub Site

I have a GitHub site and I hate web-crawlers that disobey or ignore robots.txt. How would I set up a Spider-Trap on a GitHub site that the robots.txt disallows the trap, so the good bots aren't ...
1
vote
0answers
33 views

How does a webmaster perform a proper redirect from one domain to another? [closed]

There are a few questions that are inherently present inside the main question. Those questions are: What is the best way to perform this type of redirect? (Presumably a 301 Perma for SEO Best ...
0
votes
2answers
974 views

Is it possible to block search engine indexing using DNS alone?

As per the title, is it possible to block all search engine indexing using DNS? Most guides point towards robots.txt or meta headers but I'd like to investigate the possibility of blocking all ...
4
votes
3answers
245 views

My personal website was cloned in its entirety, is this a security concern?

I was googling my (real) name yesterday (which is the name of my site) on a lark and discovered a website that had copied my personal website and its content. Like a carbon copy including links to ...
0
votes
1answer
879 views

Googlebot not respecting HTTP basic auth

I have basic auth set up and it has always worked. Suddenly Google started crawling my pages. The auth is still there (I have checked it using different browsers). I am at a loss how it's possible. ...
0
votes
2answers
22 views

Do I need an umbrella services page to match the /services/ part of our URLs?

From a usability perspective, this is not necessary. The mega menu dropdown on all pages makes it extremely clear how to navigate to the relevant services we offer. However, I am wondering whether a ...
0
votes
1answer
842 views

SEO issues with Elm?

Do search engines, particularly Google, render JavaScript created from transpiled Elm code when crawling? Can they follow links, even internal that modify the existing page?
2
votes
1answer
414 views

How does Google treat underscores in site map URLs?

Google is currently reporting that my URLs are invalid within my sitemap. Here's an example of a document that was considered erroneous by Google's validator: For each of the URLs in the map, Google ...
0
votes
1answer
537 views

Stopping Google from crawling my static domain

I use a cookieless sub domain static.example.com to serve all images, js, and css files. This static sub domain has as its root directory the same directory as the parent http://example.com. So static....

1
3 4
5
6 7
16