data crawler to login & spider/crawl inventory data from distributor website to csv file

In Progress Posted Oct 19, 2005 Paid on delivery
In Progress Paid on delivery

We need to create a automated crawler that will log into a distributor warehouse website and download inventory data from tables to a delimtered file.

The website we will be crawling is the login/search catalog section of www.warrensworld.com.

I have saved copies of their site locally to demonstrate what needs to be done. After closing of project we will provide actual login details to the live site for the job to be completed.

Walk through process of what needs to be done:

Step 1: Login

[url removed, login to view]

Login using login name/password stored in config file, in upper right hand corner or site. Once logged in the page will refresh and product search menu become searchable with our pricing.

Step 2: Successfully Logged in, Search Form

[url removed, login to view]

Once logged in the search by bar in the header becomes active. WE need to be able to select defined CATEGORY names from the CATEGORY pull down menu to be searched (we dont want to use the ALL Categories option, this takes a long time on thier site).

Step 3: Result page to crawl/store product data from

[url removed, login to view]

This is the actual data page that we need to spider the info from. We need to store ProdCode, Manufactuer, Sku#, Description, Price, In Stock, CATEGORY fields as delimitered values.

Step 4: Result page 2

[url removed, login to view]

The search pages show only 25 products at a time, we need to paginate through all of the available result pages (2, 3, 4 etc) spidering/crawling/storing the results on these pages in the same manner..

* We need to be able to define the export text file delemeter

* Define step / crawl interval (freqency between page navigations) to prevent from being banned from site

* Provide a list of CATEGORIEs to be searched from the select search menu in a config file IE "LCD TV's" , "DLP TV's" , "Cables" etc.

Example CSV file export result from step3:

ProdCOde,Manufacturer,Sku#,Description,Price,In-Stock

JVC26575,JVC,LT-26X575,LT26X575 26"Flat Panel LCD,899.00,Limited QTY

JVC32575,JVC,LT-32X575,LT32X575 32"Wide I'Art LCD TV,1399.00,Very Limited QTY

JVC40776,JVC,LT40X776,LT40X776 40" HDTV LCD TV,2550.00,Very Limited QTY

.NET ASP Java Perl PHP

Project ID: #30159

About the project

3 proposals Remote project Active Oct 19, 2005

3 freelancers are bidding on average $83 for this job

mistersoft

Can be done on Perl. Thanks.

$100 USD in 2 days
(389 Reviews)
7.5
cliver

Hello, Please look at the PMB. Thanks, Sergey

$50 USD in 1 day
(36 Reviews)
7.0
softwareprabhu

Hello I am interested to do this job and looking forward for your reply.. Thanks.

$100 USD in 7 days
(15 Reviews)
4.5