Could You Have Stopped This? Vegas Shooting Analysis with Attorneys on Retainer

[ARCHIVED THREAD] - Crawling a website

Posted: 3/13/2013 12:39:55 PM EDT

I'm wondering how the hive would do this before I reinvent the wheel.

I need to crawl a couple of websites to find the average size of the objects on the websites. Any thoughts on the best way to go about this?

Posted: 3/13/2013 12:42:10 PM EDT

[#1]

Wget, set the user agent and rate to be reasonable

Posted: 3/13/2013 12:50:13 PM EDT

[#2]

Quoted:
Wget, set the user agent and rate to be reasonable

So mirror with wget then look at the file sizes on my local system?

Posted: 3/13/2013 12:57:13 PM EDT

[#3]

Yes. You could grab the logs and post process them, too; pretty sure wget spits ou response sizes.

Posted: 3/14/2013 9:47:54 AM EDT

[#4]

Quoted:
Yes. You could grab the logs and post process them, too; pretty sure wget spits ou response sizes.

Thanks

[ARCHIVED THREAD] - Crawling a website

Warning

Confirm Action

[ARCHIVED THREAD] - Crawling a website

[ARCHIVED THREAD] - Crawling a website