253 reputation
28
bio website
location Bristol, United Kingdom
age
visits member for 4 years, 2 months
seen Feb 25 at 10:21

Dec
27
awarded  Tumbleweed
Dec
20
asked Insert file from the current directory from within a central ssi include?
Oct
25
awarded  Scholar
Oct
25
comment Google lists redirected URL instead of target URL
Checked again and the old URL has been replaced by the new one. Success :¬)
Oct
25
accepted Google lists redirected URL instead of target URL
Oct
10
comment Google lists redirected URL instead of target URL
Tried Disallow: /*/* to just disallow subdirectories, with a .htaccess 301 redirect on the root and old homepage. This has got rid of the 'robots.txt' message but Google is still displaying the old URL. Will wait a few days and see if it updates.
Oct
7
comment Google lists redirected URL instead of target URL
That makes sense. We need to have the old site in robots.txt though as it was messing up the search results. So I guess we need robots.txt to disallow the site but allow the root page?
Oct
6
awarded  Student
Oct
4
revised How do you format SEM URLS to properly track results using analytics engines like Google analytics?
Updated broken link
Oct
4
asked Google lists redirected URL instead of target URL
Jun
20
comment What's the proper way to handle Allow and Disallow in robots.txt?
Aha! According to Wikipedia, Google just looks at the number of characters to determine which directive to use and in the case of a 'draw' goes with 'Allow'.
Jun
20
comment What's the proper way to handle Allow and Disallow in robots.txt?
The order of Allow and Disallow statements makes no difference for Googlebot which looks at specificity instead. Bing also but don't know about other search engines so if in doubt put Allows first I guess.
Jun
20
comment What's the proper way to handle Allow and Disallow in robots.txt?
The above is how Googlebot handles your examples as can be tested on their robots.txt testing tool (Webmaster Tools > Blocked URLs). Google doesn't mind whether you put the Allows or Disallows first but has an algorithm that determines specificity, which leads to some not obvious results. For example, if you replace '/*.html$' with '/*myfile.html$' in the 3rd example, then 'myfile.html' is allowed rather than being blocked OR if you lose the trailing '/' from '/norobots/', .html files are also allowed.
Feb
2
comment How can I simulate a slow connection for page load?
Or try running the connection via the Tor Project proxies..
Jul
10
awarded  Yearling
Dec
9
awarded  Precognitive
Jul
31
comment Are there any downsides of including google-analytics on your website?
@Jamie. Interesting. Hadn't spotted that.
Jul
20
awarded  Beta
Jul
15
comment How can I simulate a slow connection for page load?
How about running BitTorrent at the same time :¬)
Jul
13
awarded  Commentator