Show all
Author Commit Message Labels Comments Date
Grigoriy Petukhov avatarGrigoriy Petukhov
Some experiments with distributed mode
Branches
distributed
Grigoriy Petukhov avatarGrigoriy Petukhov
Version 0.4.0
Grigoriy Petukhov avatarGrigoriy Petukhov
Fix some tests
Grigoriy Petukhov avatarGrigoriy Petukhov
Add popagate_network_log option to grab.tools.logs.default_logging
Grigoriy Petukhov avatarGrigoriy Petukhov
Fix grab/grab_config handling in Task clone method
Grigoriy Petukhov avatarGrigoriy Petukhov
Add resolve_base option to process_links method
Grigoriy Petukhov avatarGrigoriy Petukhov
Fix mongo cache backend
Grigoriy Petukhov avatarGrigoriy Petukhov
Add new spider test
Grigoriy Petukhov avatarGrigoriy Petukhov
Refactor way of handling initial grab parameters during the processing of the request
Grigoriy Petukhov avatarGrigoriy Petukhov
Refactor response_headers into lazy property
Grigoriy Petukhov avatarGrigoriy Petukhov
Speed improvements
Grigoriy Petukhov avatarGrigoriy Petukhov
Add extensions test
Grigoriy Petukhov avatarGrigoriy Petukhov
Refactor default user agent generation: static file is not used anymore
Grigoriy Petukhov avatarGrigoriy Petukhov
Refactor extension sub-system: caching extension methods with help of metaclass magic
Grigoriy Petukhov avatarGrigoriy Petukhov
Add speed tests
Grigoriy Petukhov avatarGrigoriy Petukhov
Fix charset issues. Add new option: document_charset
Grigoriy Petukhov avatarGrigoriy Petukhov
Merge
Default avatar al...@home-system
Fixed mongo queue backend: it clear on exit now.
Default avatar al...@home-system
Deleted wrong Spider.initial_urls test
Grigoriy Petukhov avatarGrigoriy Petukhov
Fix charset issue
Grigoriy Petukhov avatarGrigoriy Petukhov
Fix mongo queue test
Grigoriy Petukhov avatarGrigoriy Petukhov
Refactor the way of storing grab settings inside Task object
Grigoriy Petukhov avatarGrigoriy Petukhov
Refactor config processing in Grab
Grigoriy Petukhov avatarGrigoriy Petukhov
Move code of copying the config into separate method: copy_config
Grigoriy Petukhov avatarGrigoriy Petukhov
Move `self.default_headers` into `self.config["common_haders"]`
Grigoriy Petukhov avatarGrigoriy Petukhov
Code cleanup
Grigoriy Petukhov avatarGrigoriy Petukhov
Fix mongo queue test
Grigoriy Petukhov avatarGrigoriy Petukhov
Merge with grom branch
Default avatar al...@home-system
Inherited Spider.initial_urls test
Default avatar al...@home-system
fixed UnpickbleError raised by MongoDB queue backend when self.curl initialized in Curl transport and Grab instance pushing to tasks queue.
  1. Prev
  2. Next
Tip: Filter by directory path e.g. /media app.js to search for public/media/app.js.
Tip: Use camelCasing e.g. ProjME to search for ProjectModifiedEvent.java.
Tip: Filter by extension type e.g. /repo .js to search for all .js files in the /repo directory.
Tip: Separate your search with spaces e.g. /ssh pom.xml to search for src/ssh/pom.xml.
Tip: Use ↑ and ↓ arrow keys to navigate and return to view the file.
Tip: You can also navigate files with Ctrl+j (next) and Ctrl+k (previous) and view the file with Ctrl+o.
Tip: You can also navigate files with Alt+j (next) and Alt+k (previous) and view the file with Alt+o.