Create Site Spider (basic)
I require a script that will visit my site, seasonmonkey.com and then spider the site and all sub-domains including their internal links.
The spider should pretend to be a standard web user so that the website will then refresh its cache whenever the spider views the page, just like if a normal user were viewing the site.
The script just needs to display a list of the pages it is crawling as it progresses so I can see its progress. Within the script there should be a variable that will allow me to specify the time between page loads so that it does not overload the server.
This is a simple script as I’ve made spiders before so please don’t try to make me think it’s very complex 
I need the script completed within the next couple of hours.
Thanks
