Download Millions of Individual Websites as JSON
$750-1500 USD
Paid on delivery
We need someone to download an entire Russian-language website consisting of millions of individual pages, preferably in JSON format:
[login to view URL]
This is not a straightforward job and will require advanced knowledge of HTML, Javascript, and AJAX. Simple website downloaders (such as SiteSucker) will not work. The website uses unique IDs to identify individual webpages, but the search function will not allow the complete archive to be shown at once.
Example: [login to view URL]
Russian speaker preferable (this site is in Russian and we require the JSONs to display the Russian text (not the translated English).
Deliverables:
1. JSONs for all files from [login to view URL]
a. We need the JSONS for all document IDs on the website. We expect there to be upwards of 100 million individual JSON entries
b. It would be ideal to have some sort of validation to ensure we are getting the entire site, and not some incomplete portion
2. Programming code used to make requests from the API
a. This can be delivered at the end of the project.
b. We may require some explanation of the code so we can use it later on to fill in missing entries or redownload corrupted files.
Method of Delivery:
- Shared cloud storage to be agreed upon. JSON files should be split into batches of between 500MB and 1GB for easier processing.
Project ID: #17792622
About the project
Awarded to:
OK, as we agreed on. I will make the script and will upload data into some file sharing system like dropbox. Once I have some chunk of file will upload it. Thx, let's do the great work!
13 freelancers are bidding on average $1073 for this job
Hi... How are you?. I saw your description carefully. And if you want to see my skills please go this link: https://www.freelancer.com/u/crocodile305 When you have enough time to discuss about your project wi More
Honorable Seniors. How are you? I have experience in web scraping and data mining using python, java, c# with selenium enough. I will try to deliver great result with your satisfy all my best. Hope to meet you soon More
Hi.. How are you? I saw your description carefully your project. Owing to my rich experience in python, scrapy, data entry , i can say i can do this perfectly. I have many top skills like python, scrapy,CSS,HTML More
i can download using python................................................................................................. Regards vasudha
Hi there This is web scrapping job I have ever done it before Please contact me and discuss more detail on chat Kind Regards