![]() from threading import Thread,localĭownload 100 links in 1.1333789825439453 seconds To make it happen, The code will leverage the local object from threading package, so that 10 thread workers will share one Session object. ![]() To use the Session object, it is a waste to create 10 Session objects for 10 threads, I want one Session object and reuse it for all downloading work. I will use a queue to hold the 100 links and create 10 HTTP download worker threads to consume the 100 links asynchronously. here is the Youtube link.Īnyway, I am still going to use Python Thread to do the HTTP request job. Python threading is a dangerous topic to discuss, sometimes, multi-threading could be even slower! David Beazley brought with gut delivered a wonderful presentation to cover this dangerous topic. Looks like the performance is really boosted to 5.x seconds.ĭownload 100 links in 5.367443561553955 secondsīut this is still slow, let's try the multi-threading solution Solution #2: The Multi-Threading Way Let's see how many seconds are saved by changing to this: the withkeyword in Python is just a clean solution to replace try… finally…. To ensure the request object exit no matter success or not, I am going to use the with statement as a wrapper. So if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase - Session Objects The Session object will use urllib3's connection pooling, which means, for repeating requests to the same host, the Session object's underlying TCP connection will be re-used, hence gain a performance increase. We can leverage the Session object to further increase the speed. It takes about 10 seconds to finish downloading 100 links.ĭownload 100 links in 9.438416004180908 secondsĪs a synchronous solution, there are still rooms to improve. I forge 100 links for the test by this magic python list operator: url_list = *50 The most simple, easy-to-understand way, but also the slowest way. All code in gist is ready for copy and run. The code is running in a Linux(Ubuntu) VM host in the cloud with Python 3.7. What if I want to send hundreds or even millions of HTTP requests asynchronously? This article is an exploring note to find my fastest way to send HTTP requests. It is easy to send a single HTTP request by using the requests package. Measure execution time with timeit in PythonĪs mentioned at the beginning, the deque class from the collections module is imported as follows.Who dives faster? by Charles Zhu, my 6yo boy.To decide between list and deque based on your specific needs, consider benchmarking the processing time with the timeit module. In most cases, using list is sufficient unless you need to reduce processing time at the millisecond level. ![]() However, the difference in processing speed between list and deque is usually not noticeable when dealing with a few hundred or a few thousand elements, depending on the environment and conditions. It's recommended to use deque when you specifically want to handle data as a queue, stack, or deque.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |