Evgeniy Tatarkin avatar Evgeniy Tatarkin committed d5e6115

concurrenttools downloader generator behavior

Comments (0)

Files changed (2)

 
 Not released yet
 
+- concurrenttools change try/finally to generator behavior
 - processing requests through queue
 - better generator usage
 

pomp/contrib/concurrenttools.py

 `concurrent` package for python3
 """
 import logging
-import operator
 from concurrent.futures import ThreadPoolExecutor, as_completed
 
 import defer
 
     def get(self, requests):
         future_to_defer = {}
+
+        # configure futures and yield it
         for request in requests:
             d = defer.Deferred()
             future = self.executor.submit(self._fetch, request)
             future_to_defer[future] = (request, d)
-        try:
-            return map(operator.itemgetter(1), future_to_defer.values())
-        finally:
-            for future in as_completed(future_to_defer):
-                request, d = future_to_defer[future]
-                try:
-                    response = future.result()
-                except Exception as e:
-                    d.callback(BaseDownloadException(request, exception=e))
-                else:
-                    d.callback(response)
+            yield d
+
+        # get futures result
+        for future in as_completed(future_to_defer):
+            request, d = future_to_defer[future]
+            try:
+                response = future.result()
+            except Exception as e:
+                d.callback(BaseDownloadException(request, exception=e))
+            else:
+                d.callback(response)
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.