Parser
December 22nd, 2009
I need a parser or parsers that will crawl google for links and check if the files are alive from the following sites.
rapidshare.com
rapidshare.de
badongo.com
mediafire.com
sendspace.com
4shared.com
megaupload.com
depositfiles.com
uploading.com
filefactory.com
zshare.net
hotfile.com
netload.in
the crawler must be fast and all links should be saved into a database with the following information. Url, Caption, Source of link ie “from forum.com/thread”, Type of file ie “rar, zip, mp3, mpeg etc”, and file size. I must be able to run the crawler manually by using the command php parser.php or similar on linux and run as a cron job.
I need all this info to use in search results for a file searching site.
