Questions tagged [web-crawlers]

A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.)

Filter by
Sorted by
Tagged with
1
vote
0answers
9 views

Wordpress Website not indexing in google, no error on google webmaster [duplicate]

I just recognized my SEO Score is 0 and that my wordpress website is not indexed on google search engine, no automatic filling as well. I am not an expert, I never used a sitemap, nor htaccess but it ...
2
votes
0answers
18 views

When does a search engine learn about a particular user inside a website and display the profile link and /or a Login link along with search results? [duplicate]

How does Google search engine display user profiles or Login links along with a search results of some websites? because obviously the username or profile name is coming from the server side, so does ...
2
votes
1answer
238 views

Is the User-Agent “gce-spider” a well known scammer, a bad bot?

My website has been scammed using some "scamming-web-site steals my content through a proxy and serves the stolen content from their own domain and replaces my site in the SERP"-technique. The Apache ...
1
vote
1answer
317 views

Prevent Googlebot from crawling “access denied” error (403) of private forum that are reported in Google Search Console?

I'm running a website base it by vbulletin. Recently I moved one of my forums to a private section of my website so users and googlebot cannot access it there. Now I'm getting lots of access denied ...
2
votes
2answers
59 views

Moving a site from one subdomain to another subdomain, what to do with pages which aren't “mappable”?

In the process of moving a site from one subdomain to another subdomain. For a lot of the subfolders a 301 redirect to their respective pages on the new site. However what should I do with subdomains ...
1
vote
2answers
511 views

How can I stop Google from indexing “pretty links” external redirects from my WordPress site?

When I search site:[example].com in Google for my blog, the majority of the pages that are being indexed are "pretty links" to external sites. I created these links using the Pretty Links plugin so ...
1
vote
1answer
52 views

Can a user or a crawler see the source of a page that has been redirected via a 301?

Is it possible for a user or a web-crawler to see the contents/source code of a webdocument that is automatically redirected to another page via a 301 redirect? If it is possible? How could a user ...
1
vote
1answer
82 views

Repeated hits on my site from different IP addresses trying to access .aspx files using all my bandwidth [closed]

I checked my raw access files after being notified that my site has been limited over the last 24 hours and see different IP addresses trying to access files with .aspx extensions, see attached log ...
2
votes
0answers
209 views

What % can be regarded as “normal” “not viewed traffic” reported in AWStats?

Recently statistics for my site is about 50% "not viewed traffic". This seems much as many robots are blocked. What % of "not viewed traffic" can be regarded as "normal"? Is my site under attack when ...
2
votes
2answers
81 views

How can I avoid site search page duplicate title tag error in pagination of site search?

Google Search Console shows me my site search page duplicate Title Tag,
10
votes
1answer
3k views

Googlebot submitting thousands of requests to our map locater and using up the API quota

We have a store locater page on our customer's site. The end user enters their postcode and a search radius and we display results on a Google Map. Recently we had begun to notice that the site was ...
2
votes
0answers
219 views

Google is adding “Archive” to the title of tag and category pages and not using the meta description

I am wondering why Google is indexing my tags and categories as archives. Is that OK to be like that or I should change them? How to change them? Actually it looks so awkward! I have configured my ...
2
votes
1answer
117 views

Should I prevent search engines from indexing empty user profile pages? If so, how much content is enough for indexing?

I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available: The main user page (about me, last activity, ...) Several book ...
1
vote
2answers
94 views

Would an online queueing system for a website have an effect on google's ability to crawl the site?

If I was to implement a visitor queuing system for my website, where if visitor count is higher than n, place them in a queue. Would this have an effect on google's ability to crawl the site? Thanks
3
votes
3answers
113 views

How long before Google indexes a new (to me) domain that previously had spam and virus problems?

I registered a new ".com" domain, but when I added it to the Google Search Console I saw that there were problems with viruses and spam problems. I submitted a request for reconsideration. Google ...
1
vote
0answers
20 views

Is it bad for SEO to have a URL with nothing on it? [duplicate]

I have a WordPress site and for a very complicated reason I have to set some post type's template to empty file. Nothing. Which means I have a URL on my site (let's say: example.com/postype/the-best-...
1
vote
2answers
613 views

How to Crawl a website requires cookies for audit?

Situation: My Client's website requires cookies to access it. Users should choose (Language and country) to access the website. The problem is: Whenever I try to crawl the website using any software (...
0
votes
1answer
92 views

Can we find all backlinks to a webpage? [closed]

In Page and Brin's paper on PageRank, they say that while you are guaranteed to be able to find all links that point away from a page, the reverse is not true. My impression from reading the paper was ...
1
vote
1answer
40 views

If .htaccess is used to block my bot from accessing a particular directory, will I know this?

I'm working on a research project and I have a question. Say I would like to crawl all pages of a given site. In the case that my bot is blocked from accessing a certain portion of the site, I would ...
1
vote
0answers
221 views

Why would a bot submit a sign up form with fake info? [duplicate]

I have a sign up form where people can enter their name and address, then click on a submit button. However I am getting BOTs entering sign up details and submitting the form - or at least I presume ...
1
vote
1answer
40 views

How do we know, what is the source from which BING search indexed my webpage? - A Blog or website where my website link is placed

One of my web page with sensitive information was indexed by BING search and which in turn showing in the search results. Now that i solved by Robots meta tag in the header. However i would like to ...
2
votes
1answer
459 views

Where can I redirect exploit scanning bots?

I get a lot of those exploit scanning bots like the ones looking for a WordPress login (which I don't have) or guessing other exploitable URLs. Where is a sensible place to redirect them to via ....
1
vote
1answer
363 views

Facebot sometimes fails to parse open graph meta data, causing share failure

We maintain a WordPress-based website where Yoast plugin takes care of Open Graph meta tags generation. Recently we had a problem with likes disappearing after domain / protocol migration, but we have ...
2
votes
1answer
71 views

Why wouldn't a website apear in Google search results after robots.txt update to clean up hacked site?

I have a WordPress website which was hacked couple of days ago. I have tried to add up some security. By following couple of tutorials, I have updated my robots.txt file. But now my website is not ...
1
vote
1answer
31 views

Re-Indexing Home Page - HTML/DOM Change

(Site on WordPress) Currently, our company homepage is using images instead of divs to display our main products/solutions. As the SEO, I wanted to remove these images (which consist of mostly text) ...
4
votes
1answer
559 views

Are user agent names case-sensitive in robots.txt?

I'm blocking various bots in robots.txt and I was wondering if their names are case-sensitive. For example: User-agent: grapeshot Disallow: / Would it matter if I use grapeshot or GrapeShot?
2
votes
2answers
359 views

Could a custom crawler find unlisted web pages?

Example: A website has no sitemap.xml, no robots.txt, no index of those pages. Pages are not blocked, bots and humans have access, but they would need the URLs URL format: example.com/ofehdjtd/some-...
3
votes
1answer
387 views

Rule out third party scraping, but allow Google crawling

How to make scraping of own content through wget, httrack etc. impossible, but allow crawling through googlebot? This should be done without showing to googlebot other content, as to other user ...
0
votes
3answers
104 views

A top directory in a URL path returns a 404 error, will its subpages still be crawled?

I found a site in which one of the directories in the URL path returns 404 error, but the subpages are still live. Example URL: https://shop.edurite.com/blogs/news/summer-activities In this URL ...
1
vote
0answers
35 views

How do spambots find and submit to email opt-in the end of a funnel?

I have an opt-in page that's being nuked right now with spambots. A simple webpage that I use to allow subscribers into my email list. The only way to get to that page was through one of my tracking ...
1
vote
1answer
57 views

How should I protect “secret” links I send in Emails from being indexed by search engines?

I have noticed that bing/msnbot tries to index pages that were only ever linked to in one single email. My guess is, that some kind of browser toolbar is responsible for this. Apart from robots.txt, ...
2
votes
1answer
25 views

Should article preview pages be crawled and indexed by search engines?

I have a page called "all articles" that loads previews of articles using AJAX. Because of that, the content won't be visible to search engines, so the page would be considered "poor content". Since ...
0
votes
1answer
262 views

Does adding CDN stops Google from crawling?

I want to add Cloudflare free CDN to my site but worried about whether Google bots will be able to crawl my site later on or not.
0
votes
1answer
6k views

How block only Yandex bot

Can you show me how will look robots.txt when I block only yandex bot, allow Google bot and block Yandex bot.
1
vote
2answers
375 views

How does robots.txt work with sites in subfolders?

I have a single web host with a number of other parked domains/sites in sub-directories, like this: example.com is the primary site and root directory of the web hosting. example.com/www.example2....
2
votes
2answers
230 views

Ajax Content from Blocked Resource

I have a site built in AngularJS. Most of the dynamically-loaded content comes from a Wordpress back-end that is separate from the AngularJS site. In fact, it's a completely different domain and we do ...
1
vote
1answer
28 views

How do you properly SEO-tag an app that is only a catalogue of images?

I am making a little helper app that mostly links a lot of graphs and images. There is also some text explaining a couple of things, but more importantly that text will probably never (or very rarely) ...
2
votes
3answers
81 views

Is it good to submit our site in website submission sites?

I was wondering why don't we submit our site in website submission sites and increase our site's rank. Is is good to do this? How is this working? Is this a spam activity? What are the advantages ...
2
votes
0answers
210 views

Set bot crawl delay 10 seconds EXCEPT Googlebot?

Is it possible set the crawl delay for all bots to 10 seconds except Googlebot and Bing/Yahoo which can proceed at any pace? I like being indexed by all these bots but with Yandex and Baidu it is ...
3
votes
1answer
735 views

Checking site for meta refresh redirects

How do you check or crawl site (couple of urls) to explore existing meta refresh redirects? Screaming Frog doesn't handle them - it indicates a page with meta refresh as 200 (it is partly true), even ...
3
votes
1answer
146 views

How to check whether the infinite page scroll data is crawled and indexed by google or not?

I am having a music platform website where at a time 20 songs are visible right now then after scroll the other 20 are visible. So how can I confirm whether the songs which are coming after scrolling ...
2
votes
3answers
119 views

How can search engines find my RSS feed?

I have a small blog. Now, I have created an rss.xml. I put that in the root of the site, on the server. Should I do anything to make search engines like Google, Technorati, Yahoo! be able to find it? ...
3
votes
2answers
398 views

Google Search Console: 404 errors on existing pages

There is a few years old small website with very few pages (~5), which were indexed and ranked by Google. A few days ago 4 of those pages disappeared from SERPs and Search Console gives me 404 Not ...
1
vote
0answers
10 views

Should i create seperate page for every ad for search engine indexing purposes in my website [closed]

I am going to create a simple ad marketing web site. so I should use databases as my storing method. Now my question is If I want to show my customers ads in Google search result page(just like ...
2
votes
0answers
73 views

Which one is more SEO friendly Dynamic pages or static pages for blogging website

I am new to web development and I am developing a Blogging website. While working on its architecture from the SEO point of view, I came across a problem that whether I should create a seperate page ...
2
votes
1answer
633 views

“Is robots.txt blocking important pages?” in Search Console regarding static folder

This is the contents of my robots.txt file: User-agent: * Disallow: /static/* Disallow: /templates/* Disallow: /translations/* Sitemap: https://xx.xxx.com/sitemap.xml However I don't know why I am ...
2
votes
1answer
119 views

Does a link rel canonical tag pointing back to the page itself cause an infinite loop that wastes crawl budget?

I assume that search engine robots crawl the whole page and then the canonical tag because pages that are self canonicalized still get indexed and their contents are still parsed. However, does a ...
2
votes
1answer
97 views

robots.txt for website hosted in a subdirectory

I have 2 website which is hosted in a shared hosting. 1st website example.com hosted in a root directory as /public_html/ 2nd website example2.com hosted in a sub-directory directory as /public_html/...
1
vote
3answers
70 views

How is noFollow enforced on sites like Quora and Facebook?

I'm curious to know how search engines like Google enforce their noFollow policy on social sites. It seems like it would be largely out of their control, especially for webpages that cannot be crawled....
5
votes
1answer
845 views

Block Yandex crawler

Our site has been behaving very strangely for the last few days, lots of time outs etc. Finally think I found the cause, the Yandex bot is crawling around 10,000 pages an hour! I need to stop it ...