Hello Sir,
We've done a number of web scraping projects for our clients. We have scraped many directory websites including yellowpages, yelp and e-commerce websites including amazon, walmart etc and many more. We can deliver the data very quickly.
We use proxies with IP rotation to avoid being detected as bots. We use python with wget, scrapy, urllib and other tools to fetch webpages and parsers like HtmlXPathSelector, regular expressions etc to extract information from the html.
I have gone through your requirements. For your task, Selenium will be best framework to use. But I need to know how much speed of the software will matter over here?.
Looking forward to hear from you.
Thanks
Shiv Agrawal