Python service that executes HTTP queries from a list in parallel.
€30-250 EUR
Completed
Posted over 3 years ago
€30-250 EUR
Paid on delivery
The system is composed of 3 milestones.
• System sends HTTP GET to remote URL endpoints in parallel.
• System records the HTTP response and response code and retries on failures
• The functionality of the system is available as a static web page and REST API
Milestone 1 - System sends HTTP GET to remote URL endpoints in parallel.
Description:
Write a python class called Executor that given a list of urls queries the URLs in parallel.
The maximum number of concurrent requests is capped.
The system uses aiohttp library: [login to view URL]
Every time the number of active requests falls below max_concurrent_requests, the system will immediately start the next request from url_list.
The system handles exceptions and failures gracefuly. In the even of error or failure only the single affected Postback URL is skipped and other postback URLs are still executed. A warning message is printed to the starnard python log ([login to view URL]()) in such case.
An example of how Executor API could look is:
url_list=[{"Postback URL":"[login to view URL]"+str(i)} for i in range(0,100)]
max_concurent_requests=10
executor=Executor(url_list,max_concurent_requests)
[login to view URL]()
An integration test must be provided written in python unittest framework that when run starts a local flask server and tests the Executor class against this local server.
The approach could be simillar to: [login to view URL]
Milestone 2 - System records the HTTP response and response code and retries on failures
Extend the executor class to record the HTTP response and the HTTP response code.
For every url in the list if the response is HTTP success (2xx) Executor records the response in the result as 'Response 1' key, response code under 'HTTP Code 1' key and sets 'Success' to 'YES'. Any other keys found in the url_list are copied to the result (e.g. aux_data).
url_list=[{"Postback URL":"http://www.google.com/q=xxx", "aux_data":"xxx"}]
max_concurent_requests=10
number_of_retries=3
miliseconds_between_retries=1000
executor=Executor(url_list,max_concurent_requests, max_retries=3, miliseconds_between_retries=1000)
result=[login to view URL]()
# The result should be something like this:
assertEqual(result['Postback URL'], "http://www.google.com/q=xxx")
assertEqual(result['aux_data'], "xxx")
assertEqual(result['HTTP Code 1'], 400)
assertEqual(result['Response 1'], "<!doctype html><html itemscope="" itemtype="[login to view URL]" lang="de"><head><meta...")
assertEqual(result['Success'], "YES")
If failure happens (HTTP Code other then 2xx is returned or other problem occurs), the system records the failure in 'HTTP Code 1' and 'Response 1', then waits miliseconds_between_retries and tries again. The result of the second try is recorded as 'HTTP Code 2' and 'Response 2'. And so on until the retry count is equal to number_of_retries.
The behaviour of the system is integration tested using the test infrastructure developed in Milstone 1. Given number_of_retries=3 the test checks at least the following scenarios:
• first request is a success (HTTP CODE 200)
• first request is a failure (HTTP CODE 404)
• first request is a failure (500), second request is also a failure (HTTP CODE 405), third request is success (HTTP CODE 200)
• first request is a failure (500), second request is also a failure (HTTP CODE 405), third request is a failure (HTTP CODE 405)
Milestone 3 - The functionality of the Executor is available as a staic web page and REST API
The system accepts an input CSV file from a static web page that has an html form.
The system first verifies that auth_token is "1234". Each row of the [login to view URL] is converted into a dictionary in the url_list. The url_list with additional parameters (miliseconds_between_retries,max_concurent_requests,max_retries) is then feed to the Executor. The result of Executor::run() is converted to the CSV and send as HTTP response.
Hello,
I've read through your job description carefully and I'm pretty sure that I can finish 3 milestones in 3 days.
The key aspect of this project is asyncio(async/await) in python and building RESTful api.
I'm pretty good at asyncio and building RESTful API with Flask is easy to do it.
As your job description is very clear, I can start working immediately with your current code base.
Looking forward to work on this job.
Best Regards,
Bruno
€250 EUR in 3 days
5.0 (3 reviews)
3.1
3.1
11 freelancers are bidding on average €231 EUR for this job
Hi, I've read your project requirements throughout and happy to say I'm able to work on this project will full functionality.
I can create Class "Executor" as per requirements and can write unit test for that as well. Please ping me for more details
Looking forward to your response!
Many thanks
Hi,
I am a senior sofware developer working on python and other technologies since last few years. Highly experienced with multithreaded applications.
I believe with my skill set, I can easily help you with your problem.
Please message me so we can discuss further.
Thanks
Hello sir.
Your project attracted my attention at first glance, because I've extensive experience in Python, Machine Learning Programming, NLP, AI.
I'm really confident about your project, and very eager to join your project.
If we have a chance to cooperate, I'll do my best to provide wonderful result.
Looking forward to your response.
Best Regards.