Web Crawling
May 29th, 2009
Comments off
Am looking for a developer to create a custom crawler/spider capable of continuously crawling 1000-2000 sites per week.
1. Search 2000 sites.
2. To set frequency of crawl for each site
3. Option to search whole site or selected folders of a site
4. Option to add in a username and password for a site
Categories: MySQL, Programming crawling, crawling programming mysql, for each, for each site, Search, site, Web, web crawling, web crawling programming
