Requests.get download file in parallel

I found an issue with this script is that the combined download file doesn't match the byte size of the original. Sign up or log in Sign up using Google.

Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast what if you could invest in your favorite developer? Who owns this outage? Building intelligent escalation chains for modern SRE. Featured on Meta. Now live: A fully responsive profile. Reducing the weight of our footer.

You may need to prefix the above command with sudo if you get permission error in your linux system. The following python 3 program downloads a given url to a local file. The following example assumes that the url contains the name of the file at the end and uses it as the name for the locally saved file.

After running the above program, you will find a file named "posts" in the same folder where you have the script saved. The following python 3 program downloads a list of urls to a list of local files. The class and method are static and they are setting ServicePointManager class properties which is also static. That makes this class not thread safe. Now to give a test run to the code.

Although I did not do a test with a large file, you can see the difference in download speed. Complete code and ready to debug project you can find in the download section of this article page. Purpose of the code contained in snippets or available for download in this article is solely for learning and demo purposes.

Author will not be held responsible for any failure or damages caused due to any other usage. He is highly experienced in. He likes working on new technologies and exciting challenging projects. NET March Dark. NET Note: According to the documentation , Response. Not exactly what OP was asking, but Your chunk size could be too large, have you tried dropping that - maybe bytes at a time? It sounds as if python isn't flushing the data to file, from other SO questions you could try f.

Based on the Roman's most upvoted comment above, here is my implementation, Including "download as" and "retries" mechanism:. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams? Collectives on Stack Overflow. Learn more. Download large file in python with requests Ask Question.

Asked 8 years, 6 months ago. Active 1 month ago. Viewed k times. JoeyC 10 10 silver badges 19 19 bronze badges. Roman Podlinov Roman Podlinov Add a comment. Active Oldest Votes. Jenia 1 1 gold badge 3 3 silver badges 13 13 bronze badges.