Commits

Show all
Author Commit Message Labels Comments Date
Gregory Petukhov
Some experiments with distributed mode
Branches
distributed
Gregory Petukhov
Version 0.4.0
Gregory Petukhov
Fix some tests
Gregory Petukhov
Add popagate_network_log option to grab.tools.logs.default_logging
Gregory Petukhov
Fix grab/grab_config handling in Task clone method
Gregory Petukhov
Add resolve_base option to process_links method
Gregory Petukhov
Fix mongo cache backend
Gregory Petukhov
Add new spider test
Gregory Petukhov
Refactor way of handling initial grab parameters during the processing of the request
Gregory Petukhov
Refactor response_headers into lazy property
Gregory Petukhov
Speed improvements
Gregory Petukhov
Add extensions test
Gregory Petukhov
Refactor default user agent generation: static file is not used anymore
Gregory Petukhov
Refactor extension sub-system: caching extension methods with help of metaclass magic
Gregory Petukhov
Add speed tests
Gregory Petukhov
Fix charset issues. Add new option: document_charset
Gregory Petukhov
Merge
al...@home-system
Fixed mongo queue backend: it clear on exit now.
al...@home-system
Deleted wrong Spider.initial_urls test
Gregory Petukhov
Fix charset issue
Gregory Petukhov
Fix mongo queue test
Gregory Petukhov
Refactor the way of storing grab settings inside Task object
Gregory Petukhov
Refactor config processing in Grab
Gregory Petukhov
Move code of copying the config into separate method: copy_config
Gregory Petukhov
Move `self.default_headers` into `self.config["common_haders"]`
Gregory Petukhov
Code cleanup
Gregory Petukhov
Fix mongo queue test
Gregory Petukhov
Merge with grom branch
al...@home-system
Inherited Spider.initial_urls test
al...@home-system
fixed UnpickbleError raised by MongoDB queue backend when self.curl initialized in Curl transport and Grab instance pushing to tasks queue.
  1. Prev
  2. Next