Skip to main content
8 events
when toggle format what by license comment
Feb 7, 2013 at 14:58 vote accept taswyn
Feb 7, 2013 at 14:58 comment added taswyn Just a note, make sure to set the recursion flag for the diff. This MOSTLY seemed to work, although there are a few oddities I haven't had time to isolate (for some reason it completed far earlier on our test server)... and the wget needs to get tuned if possible to only pull html/text files, and not random 200MB mp4s someone left up =P about 10k files later, lol... More importantly it's managed to catch a number of minor issues I wouldn't have been able to find manually and might have missed the attention of the individual web admins for those departments, so thank you for the help Stephen!
Feb 5, 2013 at 11:58 comment added Stephen Ostermiller /etc/wgetrc can have a maximum recursion level in it (reclevel). It appears to me that the -l switch CAN override it, but the documentation isn't explicit. You could always edit the wgetrc file to change it.
Feb 4, 2013 at 23:34 comment added taswyn I'm fairly sure the maximum depth is 5, and -l can be used to restrict to lower (going by the documentation), but in either case thank you, I'm waiting for a couple fixes to go through first but will give it a try and see if it works tomorrow hopefully!
Feb 4, 2013 at 22:37 comment added Stephen Ostermiller Corrected the -R to -r. There is a -l for specifying the retrieval depth. The default depth is set to 5, but you can use -l to set it higher.
Feb 4, 2013 at 22:35 history edited Stephen Ostermiller CC BY-SA 3.0
edited body
Feb 4, 2013 at 20:02 comment added taswyn It's an enterprise (higher ed) level site with a number of "child"/department sub-sites, so it definitely needs to be recursive/spidering. For your second wget example, should that be a lower case r, and isn't wget's recursion limited to a depth of 5 layers (5 link jumps)? That would get everything fairly top level, hopefully, so I may pop Cygwin on here and try it a bit later today =)
Feb 4, 2013 at 19:00 history answered Stephen Ostermiller CC BY-SA 3.0