Scraper / Crawler for a website. 400 000 + pages.

$30-250 USD

In Progress

Posted

over 5 years ago

$30-250 USD

Paid on delivery

Hello everyone, We have a website that we need to gather data from. What we need are user reviews, it is a local p2p shopping platform, something like amazon but much smaller. Link is included in attached file. Framework: Preferably javascript, but python is welcome as well. Data: We are thinking dynamodb local. Mysql is an options too. But we are open to suggestions. Other: We need some kind of proxy implementation, because this website blocks requests after some time. Software needs to run on windows machine. Big part of the project is that it needs to be able to revisit pages every day or so. And update data accordingly. Overall i would like to hear the project from start to finish how it will work. Any ideas out of the box are welcome. Usually don't respond to any automated proposals. Thank you for your time, Kristijan

Project ID: 18369198

About the project

27 proposals

Remote project

Active 5 yrs ago

Looking to make some money?

Email address

Benefits of bidding on Freelancer

Set your budget and timeframe

Get paid for your work

Outline your proposal

It's free to sign up and bid on jobs

Awarded to:

@ideepeners

Hi, I am a Professional Python Developer. I have scraped various websites. Let me describe what i understood about the job. 1. You want to scrape the link provided in the database to get reviews of customers. 2. The code should be automated, i.e. it should run automatically after each day. 3. The site stops bot to scrape so i need to bypass that. 4. Both Dynamodb and mysql are good choices. It operations that you will perform on database decides which one should be used. By the information given here, MySQL sound better. But we need to discuss about the operations on database before i can say MySQL. It won't cost you to just have a chat with me. I am very friendly and good person. I will be glad to help you with this. With Regards, Shubham Sharma

$200 USD in 3 days

4.7

(22 reviews)

4.8

27 freelancers are bidding on average $209 USD for this job

@mhmhz

Hi I can implement a desktop application i C# that can run in windows. The data can be exported to Excel or mysql Thanks

$300 USD in 2 days

5.0

(122 reviews)

7.7

@Nidhijain123

Hello, I have read and understood what has to be done in this task. I have sound experience in performing such kind of work as I have done lot of project of similar kind in the past. I think I can assist you in getting this task done with more focus on quality and getting it delivered on time. Regards, Nidhi

$155 USD in 3 days

5.0

(40 reviews)

6.4

@varunk28

Hello Kristijan, I can scrape the data, i have scrapped data from 5)+ wesbites, i have 9+ years of experience, please send me a message so we can discuss more. Thanks!

$155 USD in 3 days

4.9

(20 reviews)

6.5

@leemilun

Hi employer. i am a expert in web scrapping using python. so if you give me this project, i will satisfy you surely. and i can show previous projects to you. please ping me kindly. thanks.

$222 USD in 3 days

4.8

(67 reviews)

6.6

@kunitsynartem

Hello! I can propose you to make a Python + Scrapy script, that will crawl reviews and save data to csv, json, mysql (on your choice). The script can be put on the cloud and set up to start once a day... It will also use proxies to avoid getting blocked.

$250 USD in 5 days

5.0

(47 reviews)

6.4

@olasheni

I am an excellent web and software developer with over 7+ years experience. I want to use this skills to provide you good works. Please provide more information on your project and get it done. Thanks.

$250 USD in 3 days

4.8

(49 reviews)

5.8

@Dataworker44

hi there hope you are good we can scrape 200000 pages data we have team work and we can do ti so just ping me.......thanks

$222 USD in 3 days

5.0

(130 reviews)

5.7

@saranaone

I am a Professional Python Scraper I can do Scrapping of the Website with Proxy Enabled I am confident that i can do with high accuarcy Thanks in Advance

$200 USD in 3 days

5.0

(107 reviews)

5.7

@chembirline

hello there , i have just checked the website with several requests so i did not see any blocking but maybe after a while can be of course. i will write a script in python with requests and bs4 module. but after testinng the code if there is any blocking i can go for selenium module with proxies. that can take 2 days or maybe shorter i cann not predict at the moment. firstly i have to write the test code and runn it for a while. i can directly connect the output to your mysql databank if u have it. if u dont we can create a table inn local with mysql or sqllite table. we can discuss about it this is the easiest part. so please let me know and we can discuss it further and then i can start to code. cheers Amadeus

$105 USD in 2 days

5.0

(18 reviews)

5.8

@lightingdavid

Hello. I have checked your project description carefully. Web scrapping is my top skill and i can do it. I hope to work with you. Contact me please. Thank you.

$155 USD in 3 days

4.9

(63 reviews)

5.7

@imteyazkhan85

Hi there, You can choose 2 options, 1) chrome Extension 2) VB Script Advantage of VB Script is that it will run in background, but chrome Extension should have to be operated live. And daily 400,000 is not possible, as each page takes time to load, then only we can fetch data, if you want you can run the same software on different machine or same machine with different indexing. Lets chat to discuss this in details.

$444 USD in 3 days

5.0

(20 reviews)

5.4

@mladenbiz

Greetings Kristijan python developer from Serbia here. I can provide you with scraping bot that you need. Zdravo Kristijane Pythonista iz Srbije, mogu da Vam dostavim bota kog zelite. Mladen

$333 USD in 3 days

5.0

(30 reviews)

5.5

@akhileshgandhi

Hi Kristijan, We have extreme experience in Web Scraping. Have done a lot of work of Scraping using python. We use tools scrappy, selenium, scoop for scraping. We scraped data from many e-commerce websites and pushed it in excel and in a database. Also used selenium for creating Wassup bot. Ak

$333 USD in 3 days

4.8

(10 reviews)

5.3

@HKaraarslan

Hello, I can develop a Windows desktop application using C# for your project. It will be better and more stable for HTML changes or anti-bot measures. Thanks.

$200 USD in 5 days

5.0

(11 reviews)

4.7

@MEDIAPRONET

I can create a soft that will scrape info you need I can easily integrate it with sql, so results could be saved as csv files or imported direct to database I will create it with UBot Studio, you will get the SOFT only, EXE file To integrate proxy is not a problem but usual free proxy are not good, so you will need to buy proxy what I am offering you for $200: 1. Software, EXE file 2. 1 month free support/update 3. 1 month free us of my private proxies

$200 USD in 10 days

5.0

(35 reviews)

4.6

@dataexplorer

"Hello Dear, Yes, I can work for you as per your project description with amazing exploration and experimental research abilities of mine organization is perfect for that you may require. My work fulfills as well as keep you returning consistently. It is because of the importance behind this field that I am confident & serious when taking these types of projects. Let me introduce with professionalism and skill that exists to supply the best. Regards"

$250 USD in 5 days

4.5

(4 reviews)

1.9

@majidmarateb

Hello I am very professional in "Entering Types of Information - excel & Data Scraping". You can view employers' comments in this regard. I'm starting now and finishing it. Be sure you will be my regular customer. Send me a message. Maybe we agreed. Thank you for your attention.

$30 USD in 1 day

5.0

(1 review)

1.5

@shubhamgupta2612

I am doing web scrapping from past 6 months. I have scrapped many sites which includes IMDB websites and my college websites. I do this with beautiful soups.

$77 USD in 3 days