Commits

Evgeniy Tatarkin committed 639a24d

fix depth first method

  • Participants
  • Parent commits d5e6115

Comments (0)

Files changed (2)

 
 Not released yet
 
+- bugfix `depth first` method
 - concurrenttools change try/finally to generator behavior
 - processing requests through queue
 - better generator usage

File pomp/core/engine.py

         else:  # execute requests by `witdh first` or `depth first` methods
             if crawler.is_depth_first():
                 if next_requests:
-                    self.downloader.process(
+                    # next recursion step
+                    next_requests = self.downloader.process(
                         iterator(next_requests),
                         self.response_callback,
                         crawler
                     )
+                    self._call_next_requests(next_requests, crawler)
                 else:
                     if not self.stoped and not crawler.in_process():
                         self._stop(crawler)