Commits

Evgeniy Tatarkin committed 32edd26

docs

Comments (0)

Files changed (2)

- * lock in pomp.dive incrementing _in_process variable
  * write docs about BaseDownloader as recursive downloader
    implement new `queued downloader`, witch must get requests from external queue,
    and put next request to it (needed for distributed systems)

pomp/core/base.py

 
  .. note::
 
-    All this class must be overridden
+    All this class must be subclassed
 """
 import logging
 
         return self.CRAWL_METHOD == CRAWL_DEPTH_FIRST_METHOD
 
     def dive(self, value=1):
-        # TODO must be wrapped by `lock`
+        # this method called in one thread - MainThread
+        # dowloader fetch request sync or async
         self._in_process += value
 
     def _reset_state(self):