New answers tagged

0

If you have to ask the question then you should not be using a 301 redirect. There is no way out once you add a 301 on your site. Actually http/2 does not require ssl, but there are no implementations I am aware of which allow for non-ssl requests. As to the questions you asked....compared to what? Leaving it as is? Using a 302? Yes Google say http2 will ...


1

The default directive for spiders is allow and when Google visits your robots.txt and sees the disallow command, it immediately knows that it isn't allowed to crawl the website anymore. The subsequent allows don't matter. Do note that when there are conflicting codes within the robots file or between the robots.txt and the Meta Robots tag of a page, spiders ...


0

411 Pixels Wide. This was changed as of April 16, 2016. It will likely change again.


0

Do not generate pages manually! You must use URL routing. So you can generate unlimited URLs with unique content for one view! Some reference about ASP.NET MVC Routing: http://www.asp.net/mvc/overview/older-versions-1/controllers-and-routing/asp-net-mvc-routing-overview-cs https://msdn.microsoft.com/en-us/library/cc668201.aspx#...


0

It does this by following : checking sitemap for new links on existing indexed pages, once they gets re-indexed, all links on that page are crawled/indexed. So you can put new links on existing pages for the visibility how fast that happen depends upon crawling rate of your site. It might take few hours to few days before Google start indexing these new ...


5

Google, conceptually, uses an HTML DOM parser. What this does is break any web page HTML down into its basic structure and each HTML tag is given an ID. This ID represents the order of the HTML tags from beginning to end, any dependency between HTML elements such as a li tag is dependent upon a ul tag, any parent-child relationship between HTML elements ...


1

If you have a new page, you probably have a link to it. Google visits your page often and will catch the link. If it's a matter of new content, that may take longer because I don't think Google examines content as often as it examines links. This is why they recommend a sitemap. When you have a significant change to a page, you can update your sitemap and, ...


1

From your comments on another answer, MJ12Bot is visiting your site less than once an hour (421 times in 25 days.) The best thing to do is to not worry about it. Crawl-Delay is useless for you because no crawler will obey a craw-delay so large.


0

I would suggest finding the plugin that will show users duplicate title when they are creating new thread on the forum. Also, the plugin will not allow users to submit the duplicate title post in the forum. Contact forum developer team and ask them about such plugin. Or you can build one which will scan the database and tell users about the similar title ...


0

The search console logs 404 pages as encountered by the bot The logs that you see on the server and on the console are different. A 404 on the server log won't necessarily reflect under Crawl Errors in the console.


2

There are really 2 issues here: Will the robots.txt on your site Disallow (block) Wayback from crawling your site. Will Wayback crawl your site. For point #1: As others have said, the correct entry for robots.txt is: User-agent: ia_archiver Disallow: Keep in mind that it might take a while (perhaps a good long while), for Wayback to notice any changes ...


0

Use Hreflang codes to indicate to Google which site it should index for which region. You'll find it then indexes all those links and presents them according to the engine the user uses to access the site. This largely replaces the need to identify a users location - even using a good service, it's going to be wrong if a user is behind a VPN or is part of ...


6

The Wayback Machine archive is a combination of data from a large number of different crawls: Alexa crawls, which appear after a 6 month delay Our own crawls, which are seeded from the Alexa top million list and others ArchiveTeam crawls, done by volunteers ArchiveIt crawls, done by our 400+ partners, mostly libraries, many of which allow their data to be ...


1

To answer your first point having reviewed your example site you do not have any indexable content or keywords. To index the content Google needs content to index. As your initial page doesn't have anything except for links Google will not continue down the path of indexing those links as it makes the website appear as not legitimate and a possible blackhat ...


3

You could always check what your site looks like to googlebot: https://support.google.com/webmasters/answer/6066468?rd=2 and to be sure look at the access logs of your server. There doesn't seem to be any information on the engine used w/i googlebot but if it's Chrome 36+ it will work. Alternatively, you could use webcompontent.js Polyfills at http://...


0

Using your example, www.example.com/dashboard versus www.example.com/dashboard?tab=something_public versus www.example.com/dashboard?tab=something_else_public, regardless of the resulting HTML, Google will see these as separate pages. The reason for this is simple. The URL is one of two major keys within the index. As long as each URL is unique, it is a ...


1

Use your chosen language to detect the user agent. Most legitimate search engine crawlers have their own user agent to identify it as a crawler. Then you can simply ignore those user agents.



Top 50 recent answers are included